9

A referee wrote:

The experimental results tried to confirm but did not attempt to reject the model.

Obviously my null hypothesis is that my model is wrong, and I rejected that hypothesis. What does the referee mean?

Details: the null hypothesis is that the model does not significantly explain the data. So I think if the null hypothesis is not rejected, then my model is not confirmed and not rejected. Confirm a model is much, much harder than not reject a model in statistics.

Did the referee mean that I have to set the null hypothesis as my model is correct and then try to reject that hypothesis?

3
  • 4
    It may be important that the referee's comment says 'the experimental results tried to confirm but did not attempt to reject', not 'the statistical analysis tried to confirm but did not attempt to reject'. Sep 27 '20 at 14:28
  • This might be a better fit for stats.stackexchange.com, possibly
    – Glen_b
    Sep 28 '20 at 2:27
  • Your referee's statement makes no sense: Experimental results don't do anything. Are the results not clear enough to make a judgment on your model? Did you not try to use the data to fit some other, incompatible model? Was the experiment badly chosen, or just not sensitive enough?
    – Karl
    Sep 28 '20 at 15:04
13

Of course, it’s hard to be any certain without intimately knowing your work and you have to judge whether the referee’s request is any reasonable, but:

This sounds like a request for Popperian falsifiability: The referee wants you to perform an experiment, analysis, or similar that – given your a priori knowledge – could reject the model.

Obviously my null hypothesis is that my model is wrong […]

I do not find this so obvious. In many cases, a model being wrong does not pose a feasible null hypothesis, because it would have to include every alternative model. There are exceptions to this such as when your claim is that some model has predictive power and you can investigate the null hypothesis that its predictions are as good as chance. But even then, your null hypothesis is something different from “my model is wrong”.

Details: the null hypothesis is that the model does not significantly explain the data.

That does not sound like a null hypothesis. First, the word significant (in the sense of statistical testing) does not make sense within the hypothesis as it is a property of the data with respect to the null hypothesis. Second, what does your null population look like? For any given model, there is an infinity of models that are an even worse description of the data. Are they present in your null population?

6
  • Very helpful! I updated the details. What do you mean specifically by "reject" on line 4? I think I do perform a analysis that could "reject" the model. I have a null hypothesis, and the null hypothesis is not . I think I might misinterpret your meaning of "reject" on line 4. Do you mean that I have to set the null hypothesis such that the model is correct and then try to reject it?
    – High GPA
    Sep 27 '20 at 11:50
  • 1
    @HighGPA : The only descriptions of your null hypothesis you have given are "that my model is wrong" and "that the model does not significantly explain the data". What experiment could reject either of these. Neither of these, as written, are "testable", that is, are susceptible to measurement or rejection. This Answer suggests a testable null hypothesis: "that [the model] predictions are as good as chance", which is testable -- there exists an experiment which produces at least one set of data that rejects this hypothesis and at least one set of data that does not reject this hypothesis. Sep 27 '20 at 20:01
  • 1
    @EricTowers I am not sure if I understand it correctly but do you mean that I have to prove that my experiment is able to produce at least one set of data the reject and one set of data that won't not reject the hypothesis? Any examples will help my understandings. Just one name of any good paper with a good design will suffice and I'll download the paper myself. Many thanks again!
    – High GPA
    Sep 27 '20 at 22:42
  • @HighGPA: I think I might misinterpret your meaning of "reject" on line 4. Do you mean that I have to set the null hypothesis such that the model is correct and then try to reject it? – No, and it’s hard to elaborate the alternative without writing an essay about falsifiability. What is usually done is comparing the model to some data that did not inform the model in any way, i.e., it neither inspired its general form nor contributed to the parameters. In this respect, your model should perform better than a reasonable alternative, usually the model you are proposing to replace.
    – Wrzlprmft
    Sep 28 '20 at 6:31
  • 1
    @AnonymousPhysicist: Well, the question was aiming at understanding the referee, which can be solved by rephrasing (in a very broad sense). The question does not ask for a detailed instruction of what to do – which we cannot give anyway not knowing the work in question and which would be off-topic here anyway.
    – Wrzlprmft
    Sep 29 '20 at 6:33
4

It's been many years since I've been in Academia, so I may be wide of the mark, but my interpretation of what the referee is saying – loosely speaking – is that your experimental data is "too easy" on your hypothesis. While your experiments tend to suggest the hypothesis might be true ("results tried to confirm [...] the model"), they do not (in the opinion of the referee) sufficiently "stress" the hypothesis by including "difficult" data.

Taking an analogy with software engineering, it is common practice to perform unit testing on any new block of code. While it is important to include tests that represent "normal" input data (that "[try] to confirm [...] the model"), it is also important to include "edge cases" and "difficult values" that are essentially chosen to try and "break" the new code ("attempt to reject the model").

For a simplistic example, consider a function add( param1, param2 ) designed to add two numbers together. A simple, if slightly naive, set of tests might check that add(1,1)==2, add(2,2)==4, add(5,5)==10 and add(10,10)==20. Passing these tests gives some evidence that the function is working correctly, but they don't really stress the code. Suppose that instead of writing result = param1 + param2 in the body of the function, you had result = param1 + param1 (either through "finger trouble" when first creating it, or a later search-and-replace that changed more than intended). Those tests will still pass, but they won't detect that the function is merely doubling the first parameter.

In conclusion, I believe the referee is saying that while the experimental data you've included does tend to support your hypothesis, you've not chosen a sufficiently wide range of experimental data, and that you cannot claim that the hypothesis appears to be true even in the face of experiments deliberately designed to disprove it.

0

Here is an example.

  • Hypothesis: Penguins only live in Antarctica.
  • Alternate hypothesis: Penguins only live outside Antarctica.

Experiments

  • Attempt to reject the alternate hypothesis: Look in Antarctica for penguins.
  • Attempt to reject the hypothesis: Look outside Antarctica for penguins.

To gain useful information, you need to do both types of experiments. The reviewer thinks you only did the first type of experiment, finding penguins in Antarctica and rejecting the alternate hypothesis. The reviewer wants both kinds of experiments because the reviewer correctly suspects that penguins live both inside and outside Antarctica.

Formally speaking, hypotheses should only be rejected based on evidence and not confirmed.

9
  • 2
    Wouldn't the null hypothesis in that case be that penguins live not only in Antarctica?
    – David Z
    Sep 28 '20 at 4:12
  • @DavidZ The null hypothesis is not necessarily unique. Sep 28 '20 at 4:34
  • 2
    In that case, maybe what I mean to ask is: haven't you incorrectly identified the null hypothesis?
    – David Z
    Sep 28 '20 at 4:37
  • 1
    There are considerable problems with this answer: 1) The null hypothesis and the hypothesis (a.k.a. alternative hypothesis) have to be complementary. As @David Z pointed out, they aren’t. 2) “Penguins only live in Antarctica.” is not a hypothesis that can be reasonably assessed within the framework of statistical testing. 3) You never attempt to reject the hypothesis in the framework of statistical testing. 4) This does not connect to the modelling context at all.
    – Wrzlprmft
    Sep 28 '20 at 6:41
  • 1
    @AnonymousPhysicist: The term null hypothesis is a technical term that is predominantly, if not exclusively used in the framework of statistical hypothesis testing. If you use this term, you have to adhere to its rules (or specify that you are outside this framework for whatever reason). And then Point 1 is not a matter of opinion anymore and Points 2 and 3 are very much relevant.
    – Wrzlprmft
    Sep 28 '20 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.