-4

Is it enough to discuss a higher mean in one set of results than another without statistical significance (as long as you are clear that significance wasn't tested)? Can this, in your experience, result in rejection by peer-reviewers?

2
  • This is something that I never understood. There is a huge number of papers in which errors treatment is simply neglected. Chemistry, material science, applied physics, etc. Including my papers except for some simple regressions or variability windows. Not that I feel we should do different. I guess it depends on the context. Or on the hypothesis to test / compare. – Alchimista Jan 14 at 9:58
  • 1
    You could go ahead and do a rough-and-ready significance test (compute "type A uncertainty estimate" standard error of the mean for each set of results; comment on whether the difference between the means is large or small compared with the larger of the two standard errors), and it would probably take you less time and effort than it did to type this SE question! – Daniel Hatton Jan 14 at 11:34
4

Of course I can't speak for EVERY journal in the world. Who knows?!? But in principle, a difference in means, without further statistical analysis is 'meaningless'. You are applying a transformation that hides the variance of the underlying data, so it is impossible to interpret if the differences observed are likely due to a different mean of the samples, or due to simply random variation.

In addition, I would argue that if the journal does allow you to publish a difference in means without further analysis, you still shouldn't do it--and probably find a different journal to publish in.

2

It is a good idea to avoid meaningless "Null Ritual" statistical tests that give the appearance of statistical rigor, but are often deeply misleading. For instance, there is no point in performing a test for zero-difference between things that you know a-priori cannot reasonably be expected to be the same. For example, a test for zero difference in the proportion of journal paper abstracts that express an acceptance of anthropogenic climate change, versus the proportion when judged on the basis of the whole paper, tells you nothing whatsoever. A non-significant result only tells us that the sample is too small to reveal what we already know. The space in the abstract is too small to include a statement of whether anthropogenic climate change is happening, unless that is the specific purpose of the paper. However there is more room in the body of the paper, so it is more likely to contain such a statement in providing context for the findings. So we would expect the proportion to be somewhat higher in the latter sample. It certainly does not cast any doubt on the usefulness of the survey of abstracts (in case you are wondering, yes, this is a real example).

If the difference is too small to be of any practical significance, I'd argue that there is no point in performing a test for statistical significance either.

If you are going to claim that X is different to Y based on some data, then you need to perform a test of statistical significance. However, I'd argue that it doesn't necessarily need to have a significant result for the paper to be worth publishing, providing the conclusions are suitably circumspect (the method/reasoning may be of sufficient interest, even if the results are inconclusive). The main purpose of statistical significance tests is to enforce a degree of self-skepticism on the part of the researcher, and not much more than that. It is most useful when it give a non-signifiant result.

-2

It depends on the situation.

If the mean of one result is obviously much higher than another one, then there is no need to provide statistical significance at least from my point of view.

If the mean is the same or very close to each other, then there is definitely a need to provide statistical significance.

4
  • I agree. I guess depending on the field this is not very welcomed. But I am speaking of facts, like saying the boiling point of X is Y. I am really puzzled myself although I proceed by feeling. – Alchimista Jan 14 at 10:01
  • If it is obvious though, why not perform the test anyway and say "obviously the difference is statistically significant at the X level of significance". Best not to give hostages to fortune with peer review ;o) – Dikran Marsupial Jan 14 at 11:11
  • 'obviously much higher' is a completely meaningless point if the underlying data is unknown. Maybe it's higher because there are a bunch of NA values? Maybe it's higher because there is a small number of samples with huge variance? There is no absolute need to run a statistical test, but at the very least there needs to be a representation of the underlying data/variance and not just two means with one being 'obviously much higher'. – Mario Niepel Jan 14 at 12:36
  • "if" is doing a lot of work there. If the underlying data is unknown, then nothing is obvious, but in that case @LGDGODV's reasonable suggestion doesn't apply anyway. If I give a comparison of the mean radius of stars and that of rocky planets, then it is obvious that the difference is going to be statistically significant, and it would be a waste of time performing a test of an entirely uncontensious statement. As I point out in my answer to the question, "null ritual" NHSTs can be misleading as well. – Dikran Marsupial Jan 14 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.