5

I'm going to a conference in a few days. Among the presentations I'd like to listen to, there's one paper that is nicely written but is extremely flawed methodologically. In short, the authors violate at least three assumptions of causal inference like it's no big deal - and they don't even discuss these assumptions. Due to that, any attempt to interpret the results is laughable. (I'll edit in more details, if needed).

The point is, if it were a good paper, I'd love to keep contact with the authors. I'm familiar with their previous work, they are usually good in doing what they are doing. We work in neighboring sub-fields. It's unlikely we'd work together - we just happen to use similar methods for different things. But it is likely we'll keep bumping into each other at other conferences or will review each other's papers.

Right now I see three options:

  • go to their panel, raise my hand during the Q&A time, list everything that's wrong with their paper. The presenter is really not going to like it (I know I wouldn't).
  • talk to the presenter before or after the panel over coffee, list everything that's wrong with the paper, but save him public humiliation. Problems: we are not presented to each other, so he might go on the defensive and stop listening to my comments right after the first "Man, you can't do it this way." Obviously, if he reacts like this, any "let's stay in touch" would be impossible, too.
  • say nothing, hoping that reviewers would say it instead. Problems: I don't establish any contact with the authors; it might be quite some time before the paper goes through the review process; the reviewers might not be that rigorous and familiar with the methodology (the latter depends on the submission venue, I guess).

If I decide to give feedback, how to use these options to make this feedback useful without sounding like I'm trying to trash these authors' work?

Additional details that might affect the responses:

  • Quick googling shows that the paper now has a "working paper" status and is under submission somewhere. And the results have been already featured in The Financial Times. Oh, well...
  • Seniority: both the presenter and I are at the post-doc/assistant professor level. Two other coauthors of the paper are senior professors.
  • "Are you sure you got the flaws right"? I'm sure.
  • 1
    In general, a diplomatically phrased email can be a better way to raise a contentious issue, because the other party doesn't have to respond immediately and can take some time to think instead of immediately going on the defensive. – ff524 Mar 6 '16 at 10:52
  • @ff524 That's right, thanks. Maybe, I should write a short mail in these few days before the conference... – I.M. Mar 6 '16 at 10:58
  • 1
    I would emphasize diplomatic. Calling their work laughable would not be a good way to start... Remember that assumptions are always broken in models: models are simplifications of reality, and simplification is just another word for "wrong in some meaningful way". So if someone writes me an email just listing what assumptions are broken I would most likely shrug and move on. You would have to come up with an argument (phrased politely) why those assumptions are especially relevant for this paper. – Maarten Buis Mar 6 '16 at 12:28
  • Thanks, @MaartenBuis. Yes, it goes without saying that it's not about picking out some minutiae and these assumptions are relevant for the paper. For one, they estimate the effect of absence of a treatment by using treated "units" as a control group for a non-treated "unit". The very idea of causal inference tells you that it should be the other way around: non-treated patients used as a control group for the treated. In any case, I'll do my best to be as polite and diplomatic as possible. – I.M. Mar 6 '16 at 12:48
  • 1
    In general the safest way to phrase your comment would be something like "I do not understand your strategy of using treated units as a control group" rather than "you cannot use treated units as a control group". That way if they happen to use a strategy you hadn't thought of (that can happen) your email is still perfectly appropriate. If they made an error (that can happen to), you pharased it to cause the least offence. – Maarten Buis Mar 6 '16 at 13:05
2

The key idea in a conference is to publish your research findings and get feedback from the panel and those participating in the conference who are related to your research area.

If you do believe that there are certain assumptions that are made with fundamentals blatantly violated, then you do have the right to clarify the basis on which such assumptions are made during the query session. What if they do have justifying reasons for their simplification I their assumptions? Due to their limited time and space, it might not be possible to explain all the details.

Being diplomatic is the key here (as pointed in the comments). State your points not as judgements of facts but rather as doubts for clarification. The panel may take the line and ask for further clarification as necessary.

  • 1
    Very close to my own thinking. Thanks for pointing out that the panel might take the line as well and that it could still be possible to proceed if they would just discuss further the reasons behind their assumptions. – I.M. Mar 6 '16 at 13:32
  • 1
    Ask questions. Even if you think you are right, ask questions. Listen. Then, if the opportunity opens, discuss. It gives both sides opportunity to gracefully save face if one of them is wrong. – Captain Emacs Mar 6 '16 at 13:44
  • Absolutely @CaptainEmacs, that's the rationale behind a research conference. That's the main difference that sets conferences apart from other modes of publications. – Ébe Isaac Mar 6 '16 at 14:20
6

First: Cromwell's Rule applies. You're absolutely sure that you're right, and they're wrong. That sureness should be a huge warning sign for you. So, accept the possibility that you are wrong.

Second: ask, don't tell. So not this:

You say C causes D, but actually D causes C!

But rather, something along the lines of:

Do I understand correctly that you've assumed A and B and that you've then shown that C is correlated with D, and conclude that C causes D? How did you exclude the possibilities that D causes C, or that C and D have a common cause E?

  • First: thanks for the answer. Second: yes, there's always a possibility that a reader gets it wrong, while the authors are actually right. But not this time - or, rather, I wouldn't have even opened my mouth if I didn't think there was something to talk about. I was right and I asked questions to make the authors to come to the right conclusion themselves. Made sure that it was a discussion, not a trash-talk, and cleared up some additional questions they had. Haven't seen their new version yet, but they thanked me for a "detailed and fruitful chat." – I.M. May 2 '16 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.