Is a hypothesis necessary for empirical research?

Question

Suppose I would like to do an experiment that compares two existing methods for solving a certain problem, and decides which method is better (in some predefined metric).

I can start with a hypothesis: "Method A performs at least as good as method B", and then do the experiment and refute the hypothesis by showing that method B performs better and the difference is statistically significant.

Alternatively, I can just do the experiment without any preliminary hypothesis, and find out - again - that method B performs better and the difference is statistically significant.

Is there any advantage to the first approach? Why do I need to start with a hypothesis if I end up doing the same experiment and getting to the same result?

Answer 1

You fail to recognize you already had a hypothesis. You have method A and method B, so clearly both are important And you knew that a priori.

That said, you can remain agnostic as to the direction or non-direction of the effect (better? Worse?) - and that is the difference between one-tailed and two-tailed tests.

Answer 2

It is not necessarily true that you will draw the same conclusions when doing experiments with or without a hypothesis. This is because a good hypothesis contains an expectation of the outcome which you can and usually will formulate - in text or in thoughts - due to your expertise. So you already have an idea what to look for. The conclusions you draw from the data is therefore biased by your previous knowledge. You might even overlook some other unexpected important result due to this bias.

The last sentence already tells you how you can describe results you obtain without a previous hypothesis: They are chance encounters you find when carefully analyzing your data you collected for another purpose.

You could in theory of course collect data without any intended purpose. However, people usually prefer to follow a systematic approach with hypotheses allowing them to plan their research. And this is the primary purpose of hypotheses: Planning the research. The necessary experiments are then deduced from the hypotheses. Therefore it is next to impossible that you would perform the same experiments with and without a hypothesis.

But results cannot be planned, and chance encounters are not so rare. Then often a new hypothesis is formulated afterwards which is used to generate some common thread e.g. in a publication.

Answer 3

Your question itself is very hypothetical, because as soon as you need funding/grants or conduct experiments with costly infrastructure you have to argue for others or your self, if it is reasonable what you are doing.

Just comparing two methods and looking what is better afterwards is not how planned research is working and experiments like hubble space telescope or CERN particle accelerator could have been never started.

If your "experiment" is computation time like pointed to in the comment of Siren, then it's up to you but maybe also up to computation infrastructure you need to start with a hypothesis. But I doubt as a physicist that CS is less hypothetical than natural sciences, because you can do very much cheap computation experiments and need a reason to start a specific one. So you also end/start with a hypothesis before doing the computation.

Research is planful thinking, what you describe would be random trial and error, this is also practiced for instance in superconductivity research, labs put together for decades random materials to test for high-temperature superconductivity because this was/is the only way without a theoretical base for long time. But you only get funding for such experiments, if at all, as long as the topic is trendy.

Answer 4

In some sense, there is always a hypothesis, but this is not always a useful way to look at the situation.

A key thing in experimental work in computer science is that the specific results rarely matter. We are measuring the performance of implemented methods, but the implementations are imperfect realizations of theoretical methods, which in turn are imperfect realizations of some key ideas. We are measuring the implementations, but we usually want to judge the ideas. If the ideas are worthy, somebody may eventually come up with a better realization of them.

There are often huge systematic biases in the measurements. The implementation of one method may be of higher quality than the other. Or maybe one method is a straightforward realization of the ideas, while another contains many tweaks and details that improve its performance. Or when we are interested in computational performance, the relative results may depend on hardware. Then we are measuring the performance on yesterday's hardware, but we are really trying to make predictions about the performance on unknown future hardware.

Because of the systematic biases, experimental computer science is often only interested in large differences in performance. If you have to think about statistical significance to see the difference, the difference is probably not significant. In such situations, the choice of the method often depends on other factors, such as the ease of implementing, using, and understanding the method or adapting it to solve a slightly different task.

While statistical significance has a role in experimental computer science, it is not as useful as in natural sciences. After all, statistical significance is a tool for dealing with random variation in measurements. But if we already know the specific causal mechanism behind the method, and if we treat the data as a fixed quantity that can be reduced to a set of combinatorial and statistical properties, there should not be much random variation left. If we see significant unexplained variation in measurements, it often indicates that we have not found the relevant properties of the data, or that there are some issues in the method, the implementation, or the experimental setup.

So what does this mean in practice? Often the first step is simply running the methods with various kinds of data to see what happens. This may already give decisive results. One method may be consistently better than the other, or maybe the performance of the methods is similar on all datasets. In such cases, we probably want to repeat the experiments in a more rigorous manner, but there is usually no need for formulating and testing hypotheses.

On the other hand, if the relative performance of the methods varies significantly from dataset to dataset, it probably depends on the properties of the data. Finding the relevant properties is clearly a place for hypothesis-driven work. Or maybe the results were not what we expected. Maybe we expected, based on our theoretical understanding of the method, that the method should perform well with certain kinds of data, but that did not happen. The issue may be with the implementation, the experimental setup, or the data – or even with the theoretical understanding. In such situations, it can be helpful to clearly formulate the hypotheses before testing them.

Answer 5

It depends on the type of research you’re doing.

There are three different general types of research: descriptive/qualitative, scientific/quantitative, and artefact-oriented. Only scientific/quantitative research uses hypothesis testing, and all three approaches are used in the IT space depending on what your research question is.

Artefact-oriented research involves the creation of an artefact and its evaluation against the state of the art, often using a methodology like the Design Science Research Methodology. Qualitative research involves gathering descriptive data that you then analyse to extract meaning from; in the IT space, it often involves asking users things like “what did you like/dislike about X” and then applying techniques like thematic analysis or coding to the results. Scientific/quantitative approaches involve creating a hypothesis, gathering numerical data, then performing statistical analysis on the data to prove or disprove your hypothesis.

Answer 6

Yes, for empirical research you should have a hypothesis. While review papers don't necessarily need one, you should still have one to outline the expectations going into it based on the literature. I would say "check the style guidelines" but I have never come across one that says a hypothesis is either optional or not required.

Some exceptions to this have been noted in other answers, but they are by-and-large just that, exceptions. I do touch on one potential exception later on.

If you are doing empirical research, a hypothesis is important for a number of reasons. If you absolutely have no preconceived idea regarding the outcome— which I find highly unlikely—a null hypothesis that there will be no observed difference is the go to here.

• Context: A hypothesis clearly defines the context of the experiment, "what it is" that we're looking to determine, it's why you're doing the experiment. Yes, you will have a chance to postulate on observed phenomenon and discuss the results and their implications later in the paper, but clearly defining the scope in relation to a hypothesis is crucial to a quality paper.
• Bias: When it comes time for peer review it is important to look for and consider biases in experimental design and outcomes. With no hypothesis—unless the abstract/introduction is so specific so that it might as well be one—there's no indication as to the perspective of the researcher. This is particularly important in clinical fields and other traditional sciences where there are real, sometimes large-scale human/environmental impact, but new(er) fields like data-science and A.I. are rapidly having as much if not more of an affect on society.

Even in validation studies where new equipment or methodology is being tested, you should state the expectations and whether they're based on similar models, testing, etc. This is really the only type of paper I would say could be an exception. Sometimes a papers only purpose is to answer the question: "Does this work?"

There is a peer responsibility to make sure conclusions and solutions resulting from empirical research is neutral in all regards except for the variable being tested. A strong hypothesis (or an introduction answering the "why?" and at least touching on expectations, if applicable) is where it starts.