66

You should write down your thoughts now, then get on and do the proper research, before trying to do any experiment, if you wish to be taken seriously. While there is a need for radically different ideas, most research succeeds by following what has been done before ('standing on the shoulders of giants'). It is very unlikely that you will come up with ...


50

This sort of thing happens in both the social sciences AND physical sciences. For instance, often a scientist will collect data to test a theory but will also collect lots of extraneous data. Analyses on these extraneous data often should be considered exploratory and labeled as such (because significant results could be due to the multiple tests) [As ...


47

I think the most important part of your post is in your comment Or for example grant agencies simply reject proposals that fail even slightly given formal criteria. If a student isn't corrected for sloppy work while in school, when will he or she improve? The question really boils down to How best can we as instructors give guidance to students so ...


43

Speaking as one outside of the social sciences whose work has been strongly influenced by readings from social science, I think it may be clearer if you tease apart three concepts that are often conflated: rigor, funding, and importance: The mathematical or analytical rigor of a subject makes it more difficult for outsiders to understand or hold an opinion ...


32

In many fields, finding good methodology was a long and tedious process. You gain nothing by repeating this process, if you can manage in one life time at all. However, if you do not use a reasonable methodology, your research will likely be without value. For example, consider that you have twenty drugs and try to investigate whether any of is good for the ...


31

The h-index is a measure of the impact of someone's publication list. An h-index of 10 for example means that the person has published 10 papers with at least 10 citations. The total number of papers published may be higher, but only 10 will have 10 or more citations. Critics argue that this measure disadvantages young researchers who did not have time to ...


30

This is an excellent question. I do think you (and others in similar situations) should speak up, but I realize this is very difficult to do. Two things I'd suggest: Try to figure out if the people you're dealing with understand that the methods they're proposing (p-hacking, etc.) are dodgy or not -- i.e. whether it's an issue of ethics or ignorance. This ...


24

The paper Adapting Scrum to Managing a Research Group by Michael Hicks and Jeffrey S. Foster, describes the experiences of adapting Scrum to running a research group. For those who don't know about Scrum, wikipedia has the following: Scrum is an iterative and incremental Agile software development framework for managing software projects and product ...


24

Background: I'm an academic in psychology that endeavours to apply innovative statistical techniques to answer psychological research questions. Over the years I have shared some of your concerns. While standard training in undergraduate and postgraduate psychology teaches many advanced statistical techniques, it is often taught at a high level that focuses ...


21

Kenji, For the last few years, I have given a continuing education course called Common Mistakes in Using Statistics: Spotting Them and Avoiding Them. I hope that some of the approaches I have taken might be helpful to you in convincing your colleagues that changes are needed. First, I don't start out saying that things are unethical (although I might get ...


16

While I do agree with Peter Jansson's answer that we should make it completely clear to students what they need to do to have their work understood, I feel differently about the impacting on assessment. In industry, the ability to write readable (which means maintainable) code is critical. The ability to write a really cool, super efficient block of logic ...


15

You wrote: I want to include it as a chapter in my report to show the marker that I have put over the recommended 300 hours work into my project Nooooooooo. Don't do that. This is a postgraduate degree, and you're expected to take the research seriously. It's not a 9-5 grunt job or prison sentence where you turn up, serve your time, and get the reward ...


15

While developing a method has merit within science, that method needs to be useful to answer a question which is not currently answerable. I too have worked primarily on methods development in graduate school, and got stung by this in some of my early work. So, while it's good to focus on the method, always have the question in front of you: "what do we want ...


15

TL;DR: if you don't know how to evaluate your work, you probably don't know exactly what research question you are working on. How would we perform evaluation for methods that can't be compared against others? I work in applied computer science, and this qualifies as the Number One statement my master students usually have when I ask them how they plan ...


15

Given that you don't have space enough to present all three sets of results, I would suggest that you choose the parameters that best illustrate the scientific point that you are aiming to make. When you state the parameters, however, you should add a parenthetical note or footnote that says something along the lines of: Note that we have chosen these ...


15

In principle, everything that has a research contribution can become a publication. Especially in computer science, a research contribution is very roughly defined as knowledge that is useful independently of the concrete implementation technology. That means that "look at my nice draft for an UI / syntax" isn't a research contribution, but showing (e.g., ...


14

Scientometrics is a science to the extent that it applies the scientific method to a field of inquiry. Researchers in this area formulate questions and conceptualize existing problems (e.g., “in these times of scarcity, the public wants research to be efficient: how can we measure this?”), they make hypotheses, make predictions, gather data to test them, ...


14

I often have competitions in class. I try to use them primarily for motivation, not assessment. Let me give an example: I was teaching an image processing class and we had some images of x-rays of "old master" canvas paintings. The goal was to create an algorithm that could count the density of the thread weave patterns. The quality of the answers could be ...


14

I think that an important criteria in deciding what action to take is how severe the effect on the research quality actually is. Practices that look horrifying to an expert in the practice may actually have only a minor effect on the actual validity of scientific experiments that do not depend closely on the details. I would recommend selecting a response ...


13

I think one thing you should bear in mind is that the superiority and appropriateness of a research method is determined by the nature of the research question. While quantitative methods have the advantage of, for example, being precise, qualitative research methods still have its place. Qualitative research captures aspects of the physical/social world ...


12

The Declaration of Helsinki was updated in 2013 to "mandate" that research involving human subjects must be pre-registered. While not perfect, pre-registration prevents many of these statistical manipulations. The idea of pre-registration is that publicly stating your hypotheses and the details of how they will be tested in advance reduces questionable ...


11

If you were to look up from your examination of how green the grass is on the quantitative side of the fence, you might notice someone peering in the opposite direction (Surprise! That's me!). Like you, I like learning theories on how the world works, but my early math background led me to a college major in Physics. Seeking more of a human connection in my ...


10

Proper presentation of results is a vitally important part of modern research. It's also a necessary skill for any field of work into which an academically trained individual will enter after graduation. If one does not learn how to exchange information with other people and organizations, the results can be catastrophic. Edward Tufte's Visual Information ...


10

I am not an expert in cognitive science or psychology but I have a few years of teaching assistant experience behind me so I will have a go at this one and I suspect that a few others might even agree with me. My general roles are in leading discussion sections and in lab sessions (information science: so combination of coding and data analysis in context ...


10

to show the marker that I have put over the recommended 300 hours work into my project So what? It's a thesis, not prison time. Your task in your thesis (or project) is to solve a given problem, not serve a given amount of time. To be brutally honest, the time that you spent on what was presumably a rather bad idea in the first place (re-implementing ...


10

Is the development of an original/novel method a contribution to knowledge worthy of a PhD? This depends largely on what you mean with "method". Coming up with a theoretically sound way to tackle a specific class of research problems better than currently possible is likely a good contribution (with potential for very high impact). If "method" means a ...


10

From the perspective of someone who has been on a search committee this would normally mean your teaching evaluations -- either as a TA or as the instructor on record. Any teaching awards you might have won in this regard would also count But we also recognize that not everyone has had TA/IOR experience. So it could also include: Evidence of having taken ...


10

Yes! Researchers in Human-Computer Interaction study the impacts of such things. While a syntax on its own isn't research worthy, it is publishable if you learn something generalizable. For example, some potential research questions might be: can programmers read/comprehend/edit code using your syntax faster or easier than some other syntax? There have ...


10

I get patronised by the guy whose randomised experiment allows him to get away with a t-test. How to react when that happens without sounding too defensive? I think this is the key aspect in your question. Once the "hardsplainer" starts patronizing you, you are on the defensive. And there is really no good way out of this situation once you react by ...


10

I'm hoping to be correct here, but this depends partially on personalities. Since you found the error, you should immediately inform everyone involved of the error and offer to supply a corrected version immediately. If people are reasonable they should act reasonably, of course. If you wait, and someone else notices it, you will have difficulties. Getting ...


Only top voted, non community-wiki answers of a minimum length are eligible