18

My regression tables have coefficient estimates, standard errors, and significance stars indicating the range of p-values (e.g. two stars means p < 0.05).

A reviewer wants the exact p-values. I personally think it would make the table more cluttered than it already is, and I have 12 regression tables.

Should I adopt the reviewer's suggestion, simply because they are the ones who will play a big role in determining whether my paper is accepted? Or should I attempt to explain, saying that the p-value can be calculated from the coefficient estimates and standard errors (besides, the significance stars already give a good guide to the p-value?)

=====

Edit: I decided to get rid of the significance stars and insert the p-values. Thanks for the feedback! (Interestingly, the paper was accepted, and the reviewer recommended the paper for acceptance, but they suggested I have both the significance stars and the p-value. But I didn't take their comment)

  • 4
    Are you dead-set on your opinion and want to know what a good strategy is to get your paper accepted without taking this reviewer comment into account? Or are you asking about other opinions regarding the pros and cons of providing exact p-values? – henning -- reinstate Monica Jun 21 '17 at 15:16
  • 4
    (Hint: one of these two alternatives is off-topic here.) – Federico Poloni Jun 21 '17 at 15:17
  • 27
    Are these exact p-values written elsewhere in the paper? Otherwise you are potentially hiding part of your data, which seems like a poor idea. – Tobias Kildetoft Jun 21 '17 at 15:22
  • 12
    @TobiasKildetoft if I understand correctly, the p value can be calculated from the provided values so all the needed info is there. – StrongBad Jun 21 '17 at 15:25
  • 6
    @StrongBad Ahh, right. I suppose this reveals me as someone who never works with any of these things :) – Tobias Kildetoft Jun 21 '17 at 15:28
50

In general you are not obliged to comply with all reviewer wishes in your revision, unless maybe the editor has mentioned such a point explicitly in the decision letter. If you don't want to follow a reviewer's suggestion for revision, just explain in your response letter in a factual way why you choose to do so, and see what the editor and reviewer think about it. You have good points for such a response in your question already.

I would nevertheless suggest to make the reviewer's life as easy as possible. Even if you don't want to have them in the paper, you could maybe add a list of p-values to the response letter for exactly that comment, stating that for the convenience of the reviewer you calculated the p-values from the data given in the paper (instead of expecting the reviewer to do this). You could also offer to put such a list in any supplementary material if the journal has such and the reviewer thinks it's important that readers see that.

  • 28
    +1 for "put such a list in any supplementary material". Everybody, who wants to see the p-values, can read them there but they do not disturb the flow of the relevant information in tables in the article. – daniel.neumann Jun 21 '17 at 17:57
  • 8
    Supplementary materials was my first thought as well. If the reviewer really wants those p-values easily accessible, it may just be best to incorporate his suggestion. Adding it to the supplementary materials won't change your paper at all, but will arguably make it more complete and will satisfy the reviewer. Everyone wins! – Nuclear Wang Jun 21 '17 at 19:24
21

Something like this should not influence the final decision of the editor. Further, this is not a fight worth having with the editor/reviewer so you should be flexible and accommodating.

You should check past issues and see what the format is. If p-values, regression coefficients, and standard errors are typically provided, you should probably do that. If only 2 are provided, go with those. Assuming regression coefficients and standard errors are typically provided, in your rebuttal letter a statement along the lines of

Following what we think is the standard format for Journal Title, we have only included the regression coefficients and standard errors. As the p-value can be estimated from these values, there is no loss of information. We would be happy to make a change if our understanding of the standard format is incorrect or if an exception should be made in this case. For your convenience we have attached a copy of the tables with the p-values included to this letter.

  • 2
    should not influence the final decision, but it may still. So why putting up a fight for such a trivial matter? – Walter Jun 22 '17 at 10:25
16

Subject Matter Answer: You should do as the reviewer asks. Regression stars are vague and non-specific, and amplify differences that are actually trivial. They are a coarser version of the same information, and considerably less useful.

I'm also skeptical that the difference between "*" and "0.46" is what's going to make your table too cluttered.

If anything, the most defensible answer based on "the p-value can be calculated from the coefficient estimates and standard errors" is to drop both.

Academia Answer The answer to what you should do will depend very much on the reviewers, and how big a deal they want to make of this - as well as the journal's stance on it. There are journals I review for where this objection (and indeed, the p-values generally) are likely enough to derail the entire paper. There are others where this is of less concern.

The question then becomes "Is another round of review worth potentially not having to reformat my tables?"

I can't imagine a circumstance where I would answer "Yes" to this, but YMMV.

  • 4
    best answer so far (an by far) – Walter Jun 22 '17 at 10:24
  • 1
    For a dissenting opinion on your first paragraph, take a look at Boos & Stefanski (2011, TAS). Basically, $p$ values are statistics just like coefficient estimates and others, and they exhibit sampling variability. Boos & Stefanski argue that in the light of this sampling variability, the precision of "significance stars" may be just what is warranted, whereas giving the exact $p$ values implies a precision that simply isn't there. More on this here. – S. Kolassa - Reinstate Monica Jun 23 '17 at 9:19
  • +1 Sometimes you might run out of "rounds" and the paper just gets straight rejected by the editor. Is it really worth the risk? – Sparhawk Jun 23 '17 at 11:21
  • @StephanKolassa good points, and the paper The Meta-Distribution of Standard P-Values by Taleb claims to show that a p value of 0.05 makes no sense anyway. – crobar Jun 23 '17 at 12:58
  • @StephanKolassa Personally, I've simply stopped reporting p-values, but that seemed too far for the ops question. I'm mostly opposed to stars at the boundaries. p = 0.055 gets ' ' and p = 0.05 gets '*' – Fomite Jun 23 '17 at 15:20
13

In addition to the suggestions to include the exact p-values to make the referee happy, I also want to add that including the p-values explicitly (rather than making your readers calculate it themselves if they want to know) will make your readers more happy as well.

When I read a paper, I hope the author has included as much information as I need to understand what I need to from the paper without having to do a lot of footwork myself.

  • Comments are not for extended discussion; this conversation has been moved to chat. – eykanal Jun 23 '17 at 19:50
4

Explain that the table would be too cluttered and include a full table with the p values in the response.

Sometimes requests like these are there to determine if the values (p values in this case) are acceptable in all cases. "I want to see the values" does not always mean that they must be in the paper. He/she might literally mean that he/she wants to personally see the values for review purpose.

As a reviewer, I have asked extra info just to see if everything is alright and as an author I have provided reviewers with extra information on the reply to assure that everything is alright and missing things are really not there only because it really does not fit into the page limits.

  • I don't think having info in the response to the reviewer only is sufficient, if it is necessary, it should be available to the reader as well. – Davidmh Jun 21 '17 at 19:05
  • @Davidmh Certainly, but sometimes there is no way around, some tables are too big to fit into any page. Readers are allowed to assume everything is in order. – Cem Kalyoncu Jun 21 '17 at 19:43
  • "Readers are allowed to assume everything is in order." I strongly disagree. Peer review is not a guarantee of quality, and readers are strongly encouraged to treat every paper critically. And without open reviews, I have no way to know what did the reviewers think of, or what is their expertise (for example, they may be very knowledgeable on the domain at hand, but weak in statistics). – Davidmh Jun 22 '17 at 7:26
  • First of all, let me defend my answer by saying that this is (reviewer asking info for him/herself) sometimes the case. Then let me continue with saying you are right on everything you have said. Peer review should be a guarantee on the quality, but, I have witnessed enough to know that's not the case. – Cem Kalyoncu Jun 22 '17 at 10:18
3

I understand your objection to putting in redundant information, but are your tables as-is really all that cluttered? With just coefficients, standard error, and some coded asterisks, along with a label for each row, that sounds like just 3 columns (no need for the asterisks to be their own column).

Unless I am missing something, I'd recommend just adding the p-values to appease the reviewer in your revision. If it causes a problem with typesetting and organization, you could ask the editor for their preference, which might overrule the reviewer or result in a movement to supplementary material.

Just as a personal preference, I'd usually rather see coefficients and confidence intervals rather than standard errors, p-values, and significance stars, if we are limiting to just a couple items. Confidence intervals are much more flexible.

  • It is a correlation table which usually is the upper/lower triangle of a symmetric NxN matrix so could have a very large number of rows and columns. Each entry of the table probably looks like x.xx (y.yy)* – StrongBad Jun 22 '17 at 17:59
  • @StrongBad That makes sense; I've rarely if ever seen those tables included in the body of a paper, anyways. OP referred to "regression tables" which I assumed was a table of factor coefficients, though I suppose those could also be arranged horizontally in levels and vertically in factors, I have seen both. – Bryan Krause Jun 22 '17 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.