92

Results get sent around a group of biological collaborators for feedback. Comments come back from the senior members of the group about the implications of the results, possible extensions, etc. I look at the results and I tend not to be as good at the "big picture" stuff (I'm a relatively junior member of the team), but I'm reasonably good with statistics (and that's my main role), so I look at the details.

Sometimes I think to myself "I don't think those conclusions are remotely justified by the data". How can I give honest feedback in a way that doesn't come across as overly negative? I can suggest alternative, more valid approaches, but if those give the answer "it's just noise", I'm pouring cold water over the whole thing.

  • 11
    Perhaps consider that it will hurt a lot less coming from a colleague rather than a reviewer. Sure, there's no guarantee that a statistically-minded reviewer will be assigned the paper, but still, it's better to improve the paper before trying to publish. If the results aren't supported by statistics, they are likely to fall apart under further scrutiny, and you wouldn't want to build a research program on a false finding. – Bryan Krause Jan 16 '17 at 18:45
  • 9
    Also applies when replacing "statistics" to "choice of colormaps" ;-) – gerrit Jan 16 '17 at 18:46
  • 70
    The "magic words" for this are not the "please" and "thank you" you learned as a little kid, but "I don't understand why blah blah blah". If you like the explanation you get, the person giving it will feel good about being smarter than you, and if you don't like it, the other guy started the technical debate not you, and you never accused him/her of being "wrong". – alephzero Jan 17 '17 at 6:24
  • 2
    @alephzero When my supervisor says I don't understand why you do X, I tend to interpret it as X is wrong, but there is probably a degree of imposter effect in that interpretation. – gerrit Jan 17 '17 at 14:05
  • 7
    In support of @alephzero's comment, in pedagogy / communication courses I came across the term I-message, which I really like. It's all about: "I don't understand why..." vs. "You must be wrong...". See here for further information: en.wikipedia.org/wiki/I-message – Buochserhorn Jan 18 '17 at 11:36

12 Answers 12

62

I would suggest approaching your colleague in a humble and inquisitive way (especially since you're a junior member of the team). If you start the conversation with "your conclusions are wrong and here's why" you're going to set a combative tone for the rest of the meeting. There may be reasons that they interpreted the data the way they did that you're not aware of.

Instead try approaching the situation with something similar to "I looked at the data and came to this interpretation, can you explain your interpretation to me?" You're a researcher in your own right, so junior or not your opinion should be valued. But at least with this approach you indicate that you are open to the idea of being wrong, and hopefully that will start a constructive conversation where you can debate the merits of analysis type A over type B, etc.

  • 5
    I came here for interest in this question, and this answer just seems wrong to me. I have tried the equivalent of paragraph 2 and gotten as answer to the initial question something obviously unsubstantiated, basically a repeat of the same error. Yet I seek how to avoid exactly this. – Joshua Jan 17 '17 at 17:54
  • 7
    @Joshua the idea is to start the conversation as I suggested, then take it from there. For example, if they say "I used Foo's method to solve the widget" and you think that Foo's method doesn't apply to this situation, tell them that and see what their justification is. I can't predict everything that is going to happen in the conversation, but if you approach it with the right mentality you can discuss your differences of opinion until the truth emerges – BarbalatsDilemma Jan 17 '17 at 18:01
  • 2
    If they don't provide any justification for the analysis methods they used and refuse to discuss it with you, then there's not much more that can be done. It's their research after all (unless you're coauthoring a paper with them, in which case your name is on it so you can push a lot harder for answers) – BarbalatsDilemma Jan 17 '17 at 18:17
40

General feedback rules apply.

Here are some of them:

  • There is no need to criticize any person.
  • Stick to facts.
  • Describe things, especially describe what you think about things, e.g. don't write "the assertion is not justified by the data" but "I can't see how this is explained by the data" (and probably give an example of some claim which you find equally "unexplained" by the data).
  • Don't be negative. Instead, make a suggestion for something better.
  • If you don't know an improvement, ask a question (e.g. "I could not figure out, how this conclusion was drawn, could you please clarify/provide further explanation").

Last tip: Sandwich your criticism, i.e. start with something good, end with something good and put the meat in between.

(You may also want to google "feedback rules" or "how to give feedback", but there are some rules/tips which may not apply…)

  • 4
    I would also add: be open to other people's opinion, and realize it is possible that you are wrong - regardless of background and qualifications. – Bitwise Jan 17 '17 at 12:48
  • 6
    I have to dispute the next-to-last point ("Don't be negative. Instead, make a suggestion for something better.") Saying "Why don't you try X" when you mean "Y doesn't work" is very confusing (I've been on both ends of this exchange). – Charles Staats Jan 18 '17 at 7:09
  • Isn't "I can't see how this is explained by the data" the same as "the assertion doesn't seem to be justified by the data"? The use of could, might, seems, maybe, ... should be enough to indicate you might be wrong as well. Of course it never hurts to be as polite as possible. – Trilarion Jan 18 '17 at 14:28
  • 2
    As a former biostatistician, I regret to say this is not the way to address this problem. Bio projects have lots of money behind them, creating extremely intensive pressure for positive results. Bad statistics is often the result of trying to "will" a positive result. Not being negative and sandwiching criticisms is a sure fire way to have your insights ignored in these situations. – Cliff AB Jan 18 '17 at 21:09
  • 1
    @Dirk: being a biostatistician can be very difficult. Doing things correctly doesn't always make you a lot of friends, although that's very lab dependent. – Cliff AB Jan 18 '17 at 23:50
26

It would be nice if statistics was always about the truth, and there was a right answer or method to every question. That simply isn't the case though, and many elements have room for debate. I'm an economist, and I've seen this first-hand in three different areas.

First, I did some interdisciplinary empirical research where I worked with a sociologist for a while. I was struck by how our assumptions ran contrary to each other; a few times I suggested things that were completely standard in the field of economics, only to discover that he couldn't fathom why we would do it that way. Then at least three times it was him proposing standard methods in sociology that I couldn't fathom.

Second, I moved into the policy research field. Holy cow, the policy field does things with statistics and econometrics that would make an econometrician roll over in his or her grave, while they were still alive.

And then it just got worse, because I started working alongside some data scientists. I won't even get started, except to say that they accepted as completely normal things that made me roll over in my grave.

My point in sharing my anecdotes is to suggest a more humble approach than "I don't think those conclusions are remotely justified by the data." Offer criticism, certainly, but don't be the person in the department who is a complete pedant about every statistical detail. My points were mainly about what I ran into when venturing outside of my field, but I think the lesson is still relevant within your primary field. Be particularly cautious about things you find objections to that are, regardless, widely used in your field.

None of this is to suggest that you should just ignore things you find incorrect. Rather, try statements like:

  • "I'm familiar with using method A as you did here. The issues raised in (somepaper, someyear) look possibly relevant, so you might want to address their points here too."
  • "What made you decide to use method A over method B? Maybe method B would be a good robustness check?"
  • "A line or two about how you verified the data fits with assumption X might be good here."

In short, approach it as if you're assuming they know what they're doing, then asking helpful questions to lead them to your desired points.

  • 18
    Being a complete pedant is actually the right thing to do, esp. in statistics. That does not mean being a complete jerk. It means that any additional assumptions should be stated and only claims that really follow from data and used theorems are valid. Even that one slight generalization, which sounds really reasonable can be only a hypothesis or potential explanation (which is still worth including in a paper), but not a result. – dtldarek Jan 16 '17 at 21:39
  • 5
    I think this is a good answer: in practice, whether the data justifies the conclusion has a strong academic-cultural component. This means that you can come off as an ignorant jerk if you just walk in the room and assume you're right and everyone else is wrong. Of course, whether the data justifies the conclusions is also a statistical issue. So I think the OP's goal should be to try to contribute to the discussion of the validity of the results in a useful way. (It seems unlikely that the role will be to convince the rest of the group to go back to the drawing board...) – Pete L. Clark Jan 16 '17 at 22:13
  • @dtldarek There's a difference between being pedantic and being thorough and correct in your work. And since the OP's question was about how to best engage with his colleagues, it's an important difference. – Jeff Jan 17 '17 at 9:05
  • 1
    @dtldarek Pedantic is a pejorative term. It implies, for example, arrogance or superiority in your approach. – Jeff Jan 17 '17 at 14:57
  • 1
    As a "data scientist", this answer did make me chuckle. – zelanix Jan 18 '17 at 15:25
21

"If I were a reviewer..."

The easiest way to accomplish this is to frame the critique as something a reviewer would want to know. In this way, you are positioning yourself as a very valuable team player, who is protecting the team from an imaginary adversarial reviewer. I frequently give critiques like this:

"if I were a reviewer, I would want evidence that this couldn't be the result of [null model X/violated model assumption Y]."

or

"I can imagine a reviewer wanting [some robustness check] -- let's just run it"

or

"if we get a reviewer in [academic subcommunity], they would probably want [some more rigorous technique]"

Everybody loves to hate reviewers, and anticipating all of the horrible things reviewers complain about is usually seen as a valuable and helpful contribution. (Incidentally, it's a good exercise to ask these questions of oneself as well. It's easy to get locked in to one perspective on the research and it can be a good skill to empathize with a skeptical reader.)

14

Science is about truth, not hurt feelings. If the data don't support the conclusion, say so. You are doing nobody any favors by dancing around the subject.

I don't know if there's office politics or something else involved here, or if this is your first real job and you're having trouble finding your place in the pecking order. Is that part of the issue? Try to speak to one of the collaborators, face-to-face, and ask about how to bring up your concerns in an appropriate manner.

  • 29
    Just because "science is about truth, not hurt feelings" doesn't mean that you should ignore other people's feelings just because you think you're right. Science is also about collaboration and cooperation; try to work with people, not against them. Plus, the OP clearly wants to tell them their interpretation is wrong, they just want to present it in as constructive a light as possible. If you can critique their research and avoid anyone getting upset then it's a win-win – BarbalatsDilemma Jan 16 '17 at 19:39
  • 1
    One way might be to suggest that the section on the statistics might be misconstrued by a statistically-minded reader. And, offer to teach a short seminar on statistical techniques, using the various papers sent your way as examples of how to describe the stats or do the stats in a clear and defensible way. Don't be the naysayer, teach them the right way. – Jon Custer Jan 16 '17 at 20:25
  • 5
    @Jon I think, given the context that the OP is a junior and the people they disagree with are senior members of the group, that offering to run a seminar to teach them stats might be taken as being quite presumptuous. It should be enough to explain your reasoning to them, you should definitely not imply that they need to relearn statistics – BarbalatsDilemma Jan 16 '17 at 20:40
  • Science is about truth, yet speech is about impressions and feelings. – einpoklum - reinstate Monica Jan 19 '17 at 21:58
9

Help them prove the truth

You are on their team, so to not appear "too negative", act like you're on their team - help them.

For work in progress, a preliminary conclusion will often be a hypothesis that is not yet fully covered by analysis or even the available data. "I don't think those conclusions are remotely justified by the data" would be something that a reviewer can reasonably say about a finished paper. However, you're not a reviewer for this, and the paper is not finished - what you should do instead is use your knowledge of statistics and describe in very specific terms what analysis and metrics would be required to properly support or deny that hypothesis which they're targeting.

It may be that this requires additional data. It may be that this requires simply a more thorough analysis. It may be that this particular hypothesis cannot ever be verified with the process and type of data that you're gathering, so there need to be significant changes - in any case, it's better to start handling this early, instead of waiting for a formal review.

  • 5
    I think this is a good answer. I will add that I have had good luck describing my own objections as the potential objections of a reviewer. Something like, "As it currently stands, I think a reviewer might say X about Y . I think we could overcome this objection by doing Z." – Dawn Jan 17 '17 at 1:07
  • "so to not appear "too negative"" They way not to appear too negative without ignoring the mistakes would be: "Hey, I have discovered shortcomings in the draft, but I hope we can fix them." – Trilarion Jan 18 '17 at 14:34
5

If your main role is to provide a statistics perspective to the review process, then you have to weigh in. Imagine what would happen if an outside review for publishing came back with adverse conclusions that you might have been expected to identify in an internal review.

Politics aside, I assume the collaborators want their names on a well received paper, and they will appreciate that you are making an effort to get into the groove of working with them. If you start off without assuming someone has made any fundamental or conceptual mistakes, you will have them listening to you more easily.

Keep in mind that, in general, when you don't come to the same conclusion that your colleague came to, it may be just as likely due to a poor choice in how the material was presented rather than a fundamental error in thinking. This is especially true in an internal review prior to publication. I would assume that you provide a set of eyes that are unbiased from having been buried in the production of the paper and, as a reviewer, you can find areas where the communication of the concepts has broken down, if that happens to be the case.

If at all possible, you should arrange to discuss it with the collaborator who is closest to the issue that you are concerned with. That person should be already aware that you are experienced in statistics. (If not, you may have to introduce that point to them.) You might suggest that this may be presented in a way that you are not accustomed to (that may be true or not), and that you have consequently been unable to draw the same conclusions. Let them have a chance to present their concept while you ask questions related to your expertise.

If that person is unavailable to discuss it, at least make sure you have communicated to them that you have questions specifically about the statistics and how they are presented. Then try to discuss it with one of the other collaborators In the same way. (Do it in this order to avoid any objections that you bypassed the key collaborator on the matter.)

4

How can I give honest feedback in a way that doesn't come across as overly negative

Don't be overly negative. For an internal review, you should focus on both the strengths and weaknesses of the manuscript, not just the weaknesses. Obviously, you want your colleagues to know what needs to be improved, but it is also helpful to know what others think are the strengths.

You also want to be careful about how you say things. Instead of saying something like

I don't think those conclusions are remotely justified by the data

which I view as very negative, you could let the author know that they have not convinced you of the conclusions for reasons x, y, and z.

2

tl;dr: Do not aim to prove their way wrong; aim to find the right way instead.

Do not make yourself a remote verifier or something like that.

If you write I don't think those conclusions are remotely justified by the data. You are stating that their approach is wrong. Period. You don't need to wrap it in nice words either.

I can read from your question that You were hired to improve usage of statistical tools in your team. Do your job.

  1. Do not be rude.
    State your concerns and points without mocking them or using rude words tec. During chit-chat you can mention where you were wrong etc. Some people are offended when you use overpolite and defensive language - they want hard facts, not digging out them from 2hour monolgue. Some people are crushed by emotionless facts - they took their work personally. Allways try to find something right and mention it.
  2. Try to understand.
    Ask them to explain their work to you. You all will find why they come to that conclusion, that the conclusion is wrong, why it is wrong and you can suggest different approach. Ask appropriate questions: "What does that mean..." "Why was this parameter neglected?"
  3. Help them.
    Suggest different approaches and methods. Offer them your assistance with such tools.
  4. Be open.
    Be ready to say that You did't get their picture or made wrong conclusion. Sometimes the deeper discussion reveals all the hidden points that were missing in the text, because they though "it is easy to see", but it isn't.

On the other hand, if you want to help them to prepare for a conference, the colder and nastier questions and notes you have he better they will be prepared for the actual questions from the audience.

  • "Do your job." Sometimes this means being the bearer of bad news, if for example you tried to understand and finally arrive at the conclusion that there is indeed something wrong which cannot be made right. Communicating this would probably be a very unpleasant duty. – Trilarion Jan 18 '17 at 15:02
  • @Trilarion If you prove why they are wrong it is less unpleasant than stating they are wrong. And offering help and assistance will improve the workspace as well. – Crowley Jan 18 '17 at 15:11
2

Hi Senior Member Bob,

I have been working to improve my skills at reconciling the data to draw accurate conclusions but I keep on getting stuck at conclusion XYZ for this set of data. This troubles me because I see that you've concluded QRS. Would you have a few minutes to show me how to arrive at QRS?

Now, let Bob speak and try not to interject. If he is professional then he will want to hear you out on XYZ after you hear him out.

This is a classic example of "seek first to understand then to be understood"

  • +1, but I think that the first clause sounds a little forced. OP should not pretend to be completely clueless. How about "trying to derive conclusion XYZ from the data like you have, but I keep getting stuck." ? – einpoklum - reinstate Monica Jan 19 '17 at 22:06
  • @einpoklum Agreed, OP should definitely modify it to suit their situation and personalities. I mainly wanted to show a non-combative way to get someone to elaborate. Working in IT, I find that merely trying to get someone to reproduce their "issue" in front of me solves their issue because they missed a step so IF OP is correct and conclusion QRS is wrong then hopefully that gets uncovered while the senior member expresses their conclusion process. – MonkeyZeus Jan 20 '17 at 13:44
1

I have - unfortunately - found that in many cases people tend to be very resistant of conclusions that you have made and are offering them, almost universally, while being much more likely to accept the exact same conclusions if you give them the chance to make the last rhetorical steps. In other words:

Is B true? Yes, it is. This is due to the fact that A->B and A.

is likely to get you responses like:

  • "How can you say something like that?"
  • "You're being overly critical"
  • "There goes OP again with his/her outlandish claims."
  • "You're just saying that because C"
  • "You are a terrorist/communist/chauvinist, we should expel/fire you"

while if you write, or say, something like:

So, I've been thinking about the question of B. I think the fact that A is something we need to take into account.
...
I was talking to X about the B question, and she reminded me that A->B holds. I think she makes a good point. But where does that lead us?

and then you get the:

  • "oh, oh, wait! I've got it! B! B! I knew it all along!"
  • "You know, since we're stuck, let's just try assuming B and seeing what happens."
  • "Hmm, I suppose B could be an option"
  • "I have always known that B, and I've said it all the time."

kind of responses. It sounds dumb but it just happens a lot when B is something that's not easy to stomach - socially, psychologically, politically etc.

Now apply this to your specific case :-)

1

I have been in a similar boat (doctoral student working with faculty), on a project I was working on. It was working with a colleague with a different disciplinary background, with different norms about what was considered convincing evidence. Based on the standards I was used to, I wasn't convinced of the results, and didn't think reviewers with backgrounds like mine would be either. I did use a bit of the future reviewer argument, but also had to say that I wasn't yet comfortable putting my name on the paper - that I would need to see X, Y, and Z. I had to do a lot of that myself. One thing I wanted to know was whether the selection of controls had driven the results (of an archival study), since there were many researcher degrees of freedom there. I had to write a program to run the models with every possible combination of control variables to convince myself that we hadn't just subconsciously played with combinations until we got something we liked (I was actually surprised at how robust the results were - though of course that isn't proof, but it does rule out one concern). My co-author actually proposed adding an experiment to the study, which made the results even more convincing. I don't think anything less than expressing my discomfort with putting my name on the earlier version would have yielded the result I'm now comfortable with.

Other thing I did, to give myself legitimacy: I had to learn a lot more about the methods he had applied - which was not easy. Some of the commands were part of a widely used (in his field) macro incorporating algorithms based on some older methods papers. I had to go back and read through several of those papers until I actually understood what the algorithms were doing (and it did seem like something that could be useful in my own field, with more transparency). A detailed understanding was required to express why I wasn't yet convinced. It's easy to say, "I'm not convinced by this." It's more convincing to others if you can explain what explicit sources of bias exist, or which specific assumptions are violated.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.