15

I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method. This particular method was originally created in 1980 but then improved upon in 1998 by other researchers, after they had discovered certain issues with it.

This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.

This is concerning because it shows that published researchers make mistakes and don't review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also, which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while.

Is this acceptable? If so, why?

Improve this question
4
  • 1
    It's not clear to me why the method is "outdated" in this specific case. Could you elaborate on this? So far you only mention that it is from 1998, that means it is 21 years old, but that does not mean that it is outdated.
    – PoorYorick
    Commented Feb 28, 2019 at 19:10
  • @Spectrosaurus: I don't think the OP claims that every 21 year old method is outdated. That it is outdated and known to be problematic is the premise of the question.
    – user105041
    Commented Feb 28, 2019 at 20:40
  • @holla, the question does not clearly state that the method is outdated. It is implied in the title, but this could be simply a personal opinion. The relevant part in the question itself does not give a reason why the method is outdated, which is why I have asked for clarification. It's really easy to clarify this by adding either "I think the method is outdated because it's so old" or "The method is definitely outdated because there has been a new method that leads to much better results".
    – PoorYorick
    Commented Feb 28, 2019 at 22:02
  • Discussion on the general quality of research and researchers moved to chat. Please read this FAQ before posting another comment.
    – Wrzlprmft ♦
    Commented Mar 2, 2019 at 19:36

5 Answers 5

Reset to default
26

What you call "an outdated method" another may call "the well-understood method".

In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.

I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.

1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)

Improve this answer
6
  • This would probably be better as a comment, as it doesn't answer the question, but challenges its premise.
    – user105041
    Commented Feb 28, 2019 at 17:38
  • 6
    @holla: The question asks why it would be acceptable to use older methods. This answer says why it is acceptable to use older methods. How does it not answer the question?
    – Nick Matteo
    Commented Feb 28, 2019 at 19:42
  • @kundor: Because the question states that it is about a dated method with known issues. This answer reads like "there are also unproblematic/more suitable old methods and sometimes the newer methods are worse". Which is, of course, true -- but the question is about old methods which are for given problems proven to be worse than new methods.
    – user105041
    Commented Feb 28, 2019 at 20:02
  • @holla, see my other comment, but the question does not state what you are saying it does. It does not describe a "dated method with known issues", it describes a "dated method with known issues that were then fixed in 1998". That's a huge difference and makes it very ambiguous what the actual problem is here.
    – PoorYorick
    Commented Feb 28, 2019 at 22:04
  • @Spectrosaurus: I read that in 1998, researchers created a new method because of the issues, but the researchers in questions still use the bad version (and provide no justification). Otherwise, the question would be trivial: what should be the problem with an old method without issues?
    – user105041
    Commented Feb 28, 2019 at 22:22
20

What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.

Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.

Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.

Improve this answer
4
  • 7
    This reminds me of an occasion where a university Vice Chancellor told a department head that they need to update their programme and not continue teaching concepts that are hundreds of years old... such as the methods developed by Newton, Fourier, Leibniz, etc.
    – Mick
    Commented Feb 28, 2019 at 8:21
  • 1
    Using 50-year-old methods that work well is not so bad as publishing papers that claim to have rediscovered them. I've seen that more than once in my own field. It's easy to "forgive" students for not accessing literature that was only ever published on physical paper, but referees should know better IMO.
    – alephzero
    Commented Feb 28, 2019 at 9:54
  • 7
    @alephzero You mean like this paper which invented integration in 1994?
    – David Richerby
    Commented Feb 28, 2019 at 10:26
  • 1
    @Mick Please tell me you're joking. OT: The validity of research is not necessarily dictated by the up-to-dateness of the methods. A lot of current research in natural sciences is based on methods from the 60's which were deemed unfeasible but are now used because raw comp. power is available which makes them feasible (and the methods are easy to comprehend).
    – Nox
    Commented Feb 28, 2019 at 13:36
12

I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?

You're the referee, so you tell us!

As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identified that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).

Improve this answer
4

This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.

This is not at all uncommon. It happens to many well-known techniques too.

Symbolic execution was invented in 1976. But it had been dead for decades until being resurrected around 2005 (thanks to significant advances in constraint solving). Now, it is popular, used in Google, Microsoft, NASA etc. All winning teams in DARPA Cyber Grand Challenge used it, the top team was bought by the Pentagon. What a comeback.

Similar story about neural network, it was crashed to dead by SVM (with kernel methods) years ago. It is now resurrected with a new fancy name: deep learning.

Improve this answer
3
  • I am not sure if I understand this correctly. Your examples read more like "somebody rediscovered old methods and explained why they are better than the new ones", while the question reads to me as "somebody used an old methods while the researchers in the field know the new method is better and provided no justification, as if they did not hear about the new methods". Am I wrong?
    – user105041
    Commented Feb 28, 2019 at 20:15
  • @holla, I don't think that's fair, I have never read a paper that doesn't endorse the method they use. The question is whether it really is good. OP says the trend has moved on and the method in question is accepted as not interesting anymore. As qsp points out if this was cause to prevent publication we wouldn't get revived methods, but we do!
    – ANone
    Commented Mar 1, 2019 at 10:16
  • @ANone: I work in maths. An analogy in my field would be "somebody proves something which is an easy corollary of the well-known result X without using X, but in a complicated way". Of course, this can be reasonable, but I as a reader expect to see a justification why they did not use result X (=the new method). I don't think the question is about trendiness (it's clear and trivial that older methods can be useful), but rather about not justifying the reason to use the old method. As I read it (I may be mistaken), qsp's examples are different.
    – user105041
    Commented Mar 1, 2019 at 12:27
1

I would say that in it's own right, that a better method exists does not, and should not, invalidate research.

It might be worth noting that if: a better method exists, has been used, and provides strictly better results that an outdated method does little or nothing to improve upon, then that's a different story.

To reiterate, I would be very uncomfortable citing "could have done better", on it's own, as a rebuttal.

For what its worth, my field mostly involves computational modelling and new methods are a frequent occurrence. The entire field only ever publishing with the latest and greatest methods would be almost inconceivable, and perhaps that effects my opinion more than it should in other fields.

Improve this answer
3
  • In this generality, I disagree. While using the old, problematic method is still "research", it doesn't have to be "interesting research", i.e. publishable.
    – user105041
    Commented Feb 28, 2019 at 17:41
  • @holla I didn't say it shouldn't be a factor. It may well not be interesting, I think I made that caveat clear, but it doesn't make it not interesting either. It's necessary to exercise judgement.
    – ANone
    Commented Mar 1, 2019 at 10:10
  • 1
    @holla The research can well be interesting, even ground-breaking. The methods are just a tool. If you are using the tool to find out interesting stuff in your subject, the tool can well use older methods. In urban climate and wind engeneering many use computational methods for solving fluid flow developed in the 1970s to 1990s. There is huge iterature threafter, but you don't need the most recent methods of computational fluid dynamics to find out interesting results. The older ones still work and often converge to the correct result.
    – Vladimir F Героям слава
    Commented Mar 1, 2019 at 14:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .