How can I avoid being "the negative one" when giving feedback on statistics?

Question

Results get sent around a group of biological collaborators for feedback. Comments come back from the senior members of the group about the implications of the results, possible extensions, etc. I look at the results and I tend not to be as good at the "big picture" stuff (I'm a relatively junior member of the team), but I'm reasonably good with statistics (and that's my main role), so I look at the details.

Sometimes I think to myself "I don't think those conclusions are remotely justified by the data". How can I give honest feedback in a way that doesn't come across as overly negative? I can suggest alternative, more valid approaches, but if those give the answer "it's just noise", I'm pouring cold water over the whole thing.

Answer 1

I would suggest approaching your colleague in a humble and inquisitive way (especially since you're a junior member of the team). If you start the conversation with "your conclusions are wrong and here's why" you're going to set a combative tone for the rest of the meeting. There may be reasons that they interpreted the data the way they did that you're not aware of.

Instead try approaching the situation with something similar to "I looked at the data and came to this interpretation, can you explain your interpretation to me?" You're a researcher in your own right, so junior or not your opinion should be valued. But at least with this approach you indicate that you are open to the idea of being wrong, and hopefully that will start a constructive conversation where you can debate the merits of analysis type A over type B, etc.

Answer 2

General feedback rules apply.

Here are some of them:

• There is no need to criticize any person.
• Stick to facts.
• Describe things, especially describe what you think about things, e.g. don't write "the assertion is not justified by the data" but "I can't see how this is explained by the data" (and probably give an example of some claim which you find equally "unexplained" by the data).
• Don't be negative. Instead, make a suggestion for something better.
• If you don't know an improvement, ask a question (e.g. "I could not figure out, how this conclusion was drawn, could you please clarify/provide further explanation").

Last tip: Sandwich your criticism, i.e. start with something good, end with something good and put the meat in between.

(You may also want to google "feedback rules" or "how to give feedback", but there are some rules/tips which may not apply…)

Answer 3

It would be nice if statistics was always about the truth, and there was a right answer or method to every question. That simply isn't the case though, and many elements have room for debate. I'm an economist, and I've seen this first-hand in three different areas.

First, I did some interdisciplinary empirical research where I worked with a sociologist for a while. I was struck by how our assumptions ran contrary to each other; a few times I suggested things that were completely standard in the field of economics, only to discover that he couldn't fathom why we would do it that way. Then at least three times it was him proposing standard methods in sociology that I couldn't fathom.

Second, I moved into the policy research field. Holy cow, the policy field does things with statistics and econometrics that would make an econometrician roll over in his or her grave, while they were still alive.

And then it just got worse, because I started working alongside some data scientists. I won't even get started, except to say that they accepted as completely normal things that made me roll over in my grave.

My point in sharing my anecdotes is to suggest a more humble approach than "I don't think those conclusions are remotely justified by the data." Offer criticism, certainly, but don't be the person in the department who is a complete pedant about every statistical detail. My points were mainly about what I ran into when venturing outside of my field, but I think the lesson is still relevant within your primary field. Be particularly cautious about things you find objections to that are, regardless, widely used in your field.

None of this is to suggest that you should just ignore things you find incorrect. Rather, try statements like:

• "I'm familiar with using method A as you did here. The issues raised in (somepaper, someyear) look possibly relevant, so you might want to address their points here too."
• "What made you decide to use method A over method B? Maybe method B would be a good robustness check?"
• "A line or two about how you verified the data fits with assumption X might be good here."

In short, approach it as if you're assuming they know what they're doing, then asking helpful questions to lead them to your desired points.

Answer 4

"If I were a reviewer..."

The easiest way to accomplish this is to frame the critique as something a reviewer would want to know. In this way, you are positioning yourself as a very valuable team player, who is protecting the team from an imaginary adversarial reviewer. I frequently give critiques like this:

"if I were a reviewer, I would want evidence that this couldn't be the result of [null model X/violated model assumption Y]."

or

"I can imagine a reviewer wanting [some robustness check] -- let's just run it"

or

"if we get a reviewer in [academic subcommunity], they would probably want [some more rigorous technique]"

Everybody loves to hate reviewers, and anticipating all of the horrible things reviewers complain about is usually seen as a valuable and helpful contribution. (Incidentally, it's a good exercise to ask these questions of oneself as well. It's easy to get locked in to one perspective on the research and it can be a good skill to empathize with a skeptical reader.)

Answer 5

Science is about truth, not hurt feelings. If the data don't support the conclusion, say so. You are doing nobody any favors by dancing around the subject.

I don't know if there's office politics or something else involved here, or if this is your first real job and you're having trouble finding your place in the pecking order. Is that part of the issue? Try to speak to one of the collaborators, face-to-face, and ask about how to bring up your concerns in an appropriate manner.

Answer 6

Help them prove the truth

You are on their team, so to not appear "too negative", act like you're on their team - help them.

For work in progress, a preliminary conclusion will often be a hypothesis that is not yet fully covered by analysis or even the available data. "I don't think those conclusions are remotely justified by the data" would be something that a reviewer can reasonably say about a finished paper. However, you're not a reviewer for this, and the paper is not finished - what you should do instead is use your knowledge of statistics and describe in very specific terms what analysis and metrics would be required to properly support or deny that hypothesis which they're targeting.

It may be that this requires additional data. It may be that this requires simply a more thorough analysis. It may be that this particular hypothesis cannot ever be verified with the process and type of data that you're gathering, so there need to be significant changes - in any case, it's better to start handling this early, instead of waiting for a formal review.

Answer 7

If your main role is to provide a statistics perspective to the review process, then you have to weigh in. Imagine what would happen if an outside review for publishing came back with adverse conclusions that you might have been expected to identify in an internal review.

Politics aside, I assume the collaborators want their names on a well received paper, and they will appreciate that you are making an effort to get into the groove of working with them. If you start off without assuming someone has made any fundamental or conceptual mistakes, you will have them listening to you more easily.

Keep in mind that, in general, when you don't come to the same conclusion that your colleague came to, it may be just as likely due to a poor choice in how the material was presented rather than a fundamental error in thinking. This is especially true in an internal review prior to publication. I would assume that you provide a set of eyes that are unbiased from having been buried in the production of the paper and, as a reviewer, you can find areas where the communication of the concepts has broken down, if that happens to be the case.

If at all possible, you should arrange to discuss it with the collaborator who is closest to the issue that you are concerned with. That person should be already aware that you are experienced in statistics. (If not, you may have to introduce that point to them.) You might suggest that this may be presented in a way that you are not accustomed to (that may be true or not), and that you have consequently been unable to draw the same conclusions. Let them have a chance to present their concept while you ask questions related to your expertise.

If that person is unavailable to discuss it, at least make sure you have communicated to them that you have questions specifically about the statistics and how they are presented. Then try to discuss it with one of the other collaborators In the same way. (Do it in this order to avoid any objections that you bypassed the key collaborator on the matter.)

Answer 8

How can I give honest feedback in a way that doesn't come across as overly negative

Don't be overly negative. For an internal review, you should focus on both the strengths and weaknesses of the manuscript, not just the weaknesses. Obviously, you want your colleagues to know what needs to be improved, but it is also helpful to know what others think are the strengths.

You also want to be careful about how you say things. Instead of saying something like

I don't think those conclusions are remotely justified by the data

which I view as very negative, you could let the author know that they have not convinced you of the conclusions for reasons x, y, and z.

Answer 9

tl;dr: Do not aim to prove their way wrong; aim to find the right way instead.

Do not make yourself a remote verifier or something like that.

If you write I don't think those conclusions are remotely justified by the data. You are stating that their approach is wrong. Period. You don't need to wrap it in nice words either.

I can read from your question that You were hired to improve usage of statistical tools in your team. Do your job.

1. Do not be rude.
   State your concerns and points without mocking them or using rude words tec. During chit-chat you can mention where you were wrong etc. Some people are offended when you use overpolite and defensive language - they want hard facts, not digging out them from 2hour monolgue. Some people are crushed by emotionless facts - they took their work personally. Allways try to find something right and mention it.
2. Try to understand.
   Ask them to explain their work to you. You all will find why they come to that conclusion, that the conclusion is wrong, why it is wrong and you can suggest different approach. Ask appropriate questions: "What does that mean..." "Why was this parameter neglected?"
3. Help them.
   Suggest different approaches and methods. Offer them your assistance with such tools.
4. Be open.
   Be ready to say that You did't get their picture or made wrong conclusion. Sometimes the deeper discussion reveals all the hidden points that were missing in the text, because they though "it is easy to see", but it isn't.

---

On the other hand, if you want to help them to prepare for a conference, the colder and nastier questions and notes you have he better they will be prepared for the actual questions from the audience.

Answer 10

Hi Senior Member Bob,

I have been working to improve my skills at reconciling the data to draw accurate conclusions but I keep on getting stuck at conclusion XYZ for this set of data. This troubles me because I see that you've concluded QRS. Would you have a few minutes to show me how to arrive at QRS?

Now, let Bob speak and try not to interject. If he is professional then he will want to hear you out on XYZ after you hear him out.

This is a classic example of "seek first to understand then to be understood"

Answer 11

I have - unfortunately - found that in many cases people tend to be very resistant of conclusions that you have made and are offering them, almost universally, while being much more likely to accept the exact same conclusions if you give them the chance to make the last rhetorical steps. In other words:

Is B true? Yes, it is. This is due to the fact that A->B and A.

is likely to get you responses like:

• "How can you say something like that?"
• "You're being overly critical"
• "There goes OP again with his/her outlandish claims."
• "You're just saying that because C"
• "You are a terrorist/communist/chauvinist, we should expel/fire you"

while if you write, or say, something like:

So, I've been thinking about the question of B. I think the fact that A is something we need to take into account.
...
I was talking to X about the B question, and she reminded me that A->B holds. I think she makes a good point. But where does that lead us?

and then you get the:

• "oh, oh, wait! I've got it! B! B! I knew it all along!"
• "You know, since we're stuck, let's just try assuming B and seeing what happens."
• "Hmm, I suppose B could be an option"
• "I have always known that B, and I've said it all the time."

kind of responses. It sounds dumb but it just happens a lot when B is something that's not easy to stomach - socially, psychologically, politically etc.

Now apply this to your specific case :-)

Answer 12

I have been in a similar boat (doctoral student working with faculty), on a project I was working on. It was working with a colleague with a different disciplinary background, with different norms about what was considered convincing evidence. Based on the standards I was used to, I wasn't convinced of the results, and didn't think reviewers with backgrounds like mine would be either. I did use a bit of the future reviewer argument, but also had to say that I wasn't yet comfortable putting my name on the paper - that I would need to see X, Y, and Z. I had to do a lot of that myself. One thing I wanted to know was whether the selection of controls had driven the results (of an archival study), since there were many researcher degrees of freedom there. I had to write a program to run the models with every possible combination of control variables to convince myself that we hadn't just subconsciously played with combinations until we got something we liked (I was actually surprised at how robust the results were - though of course that isn't proof, but it does rule out one concern). My co-author actually proposed adding an experiment to the study, which made the results even more convincing. I don't think anything less than expressing my discomfort with putting my name on the earlier version would have yielded the result I'm now comfortable with.

Other thing I did, to give myself legitimacy: I had to learn a lot more about the methods he had applied - which was not easy. Some of the commands were part of a widely used (in his field) macro incorporating algorithms based on some older methods papers. I had to go back and read through several of those papers until I actually understood what the algorithms were doing (and it did seem like something that could be useful in my own field, with more transparency). A detailed understanding was required to express why I wasn't yet convinced. It's easy to say, "I'm not convinced by this." It's more convincing to others if you can explain what explicit sources of bias exist, or which specific assumptions are violated.