26

I recently came across several papers in the area of Information Extraction and Data Mining which seem to have passed the review process even if they (1) use some new terminology without describing it, (2) replicate some old and popular technique and just describe it in a new way, (3) use new datasets without releasing them publicly or describing them in detail, (4) confusing language that doesn't seem to describe the methods... and much more.

Why challenge them? Such papers give these people a kind of leverage in situations where number of publications count rather than quality of genuine work. They also encourage, let's say, 'pseudoscience' within the scientific community - or fraud, as those more experienced among you say.

I've faced situations like:-

  • Someone hired me from a freelance site to implement a previously published paper. The paper doesn't adequately describe the novel methods they claim. I spend a week on it but don't get paid.
  • Similar situation but the implementation's accuracy doesn't match the claim.
  • In a semi-academic gathering (tea time in a summer school), a discussion turned into a debate. Though I wasn't involved in it, one of the others said something which I pointed out as being logically wrong. It turned personal and one of them claimed that they knew better because of having twice the number of papers than me. Later when I read their papers I found that they didn't contain any ideas that would count as a genuine research.

I guess some of you might have been in similar situations before. Yes it doesn't matter in the longer term; but it is hard to avoid some situations like this and I think it might be hampering the research community as a whole.

Here are the exact points of my question:

  1. Should we confront such publications? Would we get anything out of it?
  2. Is it even feasible?
  3. How can I do so? What actions are possible in such cases? What is the best course of action?

p.s. Yes citations matter more than just throwing papers - but such publications also get citations from other equally content-less papers published in similar conferences.

pps. Most of these papers are from less-than-tier-3 level conferences but still published by top publishers like IEEE and Springer.

  • 2
    What action are you hoping would come of your "challenge"? Retraction? – Nate Eldredge Jun 14 '15 at 23:05
  • @NateEldredge I guess I must add one more line in the question. What all actions could be there? – Aditya Jun 14 '15 at 23:06
  • Your best bet is to replicate and publish the replication with a sound explanation of why the original research was dodgy. This, of course, is expensive. I can't go around replicating and disproving a larger, better-funded, more experienced team's lazy work. If there is a significant amount of bad work in a specialised field, explain the general issues with it, propose a new methodology, and apply it in a case study to show its benefits. – Steve DL Jun 15 '15 at 13:59
  • 5
    I spend a week on it but don't get paid. - You should always get paid for your time. You may want to revise whatever contractual agreement you send them in advance for your work, as well as be sure you know what you're making, (and they know what they're getting), before you start working. – DoubleDouble Jun 15 '15 at 17:43
  • 3
    Academia is a game. Some people who aren't good at research are good at playing the game. Many areas could improve their credibility, for example, by not publishing an article without the raw data, or not publishing an algorithm without an implementation. But there is massive resistance to this, because it would expose too many academics as frauds. – Flounderer Jun 15 '15 at 22:23
43

I think you have to further examine your motivation for wanting to “challenge” these substandard papers. All of the following are reasons which are suggested in your question.

  1. You feel that contentless or decisively problematic publications bring down the quality of the academic community as a whole.
  2. You feel that the people who are publishing these works get too much credit.
  3. In particular, they are holding themselves up as superior to you.
  4. Specific people – e.g. you – suffer when you try to rely on publications which do not do what they promise.

This is a lot to answer, but I’ll take a stab at it:

  1. This is a disturbing trend in the world of academic publication. I now get more “journal spam” emails than all other spam put together (well, counting only what makes it into my inbox). One can no longer assume that an academic publication is reputable, legitimate, or sincere in any way without having prior specific knowledge of the people and institutions involved. I think this is a terrible problem, but it's well beyond my pay grade. I do not have a solution in mind.

  2. The best revenge is living well. In line with point 1, there is an increasing segment of the worldwide academic community in which people, in order to maintain their very low-paying academic jobs, simply must meet a certain quota of authorship. No one cares about the content or quality; in many cases there is no infrastructure there for them to evaluate the content if they did care. This is very sad. At an individual level, what seems best to me is just to get out of this part of the academic community. In many cases this is best accomplished by relocating to a different part of the world where the academic infrastructure exists. You are not going to get a job at a research university in the United States by having 50 bullshit papers: that's just not the way things work here. However, if you are a scholar in India, you can get offered a job in the United States (or Europe or Japan or...) on the basis of your work: it just has to actually be very good.

  3. The person who tells you that he’s right and you’re wrong because he has twice as many papers as you is on the one hand an asshole. On the other hand, he’s splashing around in a pond that you want to get out of: I can assure you that there are many academic environments in which who's right is not decided by counting publications. You sound like a serious person: get yourself into one of these environments.

  4. This seems to open up another can of worms, but relative to point 1 above: you simply cannot uncritically take publications at face value. To be honest about it, you can do this to a greater or lesser degree according to the level of trust and esteem you place in the author, the journal, the institution...but doing this at all is a compromise we make only because we are finite beings living in an imperfect world. Learn to develop a “nose” for what is correct and what is problematic. Definitely do not accept a work-for-pay situation which is predicated on the assumption that an unvetted academic work does everything that it promises!

    In general, I would say that point 4 is a good reason to challenge papers: if you sstumbled by relying on a paper or can imagine another serious, reputable person doing the same, then you owe it to the community to try to get the word out about the defects of the work. But as above you should also understand that part of being a good academic is knowing what looks good and what looks bad. There is an ever-expanding sewage sea of worse-than-useless academic papers. You can't take responsibility for getting rid of all of them. But you can teach yourself to avoid the unpleasant experience of being submerged in this sewage sea, and you can teach your colleagues – and, as you get older, your mentees – how to do the same.

Good luck.

  • 1
    I appreciate your analysis of the situation I presented. Especially when you say "there is no infrastructure there for them to evaluate" - I believe you have a good experience of seeing the world and knowing what's right and what's wrong. Yes ALL of the publications I've pointed to are from people/universities which are below par than those of the papers I usually read. And last paragraph in the answer was, in a way, the most feasible solution to the problem. – Aditya Jun 15 '15 at 0:45
  • 4
    Whilst I generally agree with you, I find that it is specifically in the US that having 50 bullshit publications will land you a job, as opposed to some European countries where there's a lesser pressure to publish and a community that is more demanding on scientific rigour. Our mileage may vary across research fields of course. – Steve DL Jun 15 '15 at 14:01
  • 50 bullshit publications can land you a job at a research university in the US, just not a very good one. Usually. 50 bullshit publications can also land you a job at a research university in Europe, just not a very good one. Usually. – JeffE Jun 15 '15 at 15:39
  • 3
    I actually chose the word "bullshit" deliberately: it is not so polite, but I thought it conveyed my meaning so precisely as to be worth it. Now I think I should find a different way to express myself. (Profanity didn't elevate my discourse like I thought it would: damn!) – Pete L. Clark Jun 15 '15 at 16:45
  • 2
    50 bullshit publications may land someone a bullshit academic job at a not-so-good research university ("big fish in small pond" syndrome), but once you get out of the small fishpond, the rest of the world respects real-world achievements and not that sort of nonsense. The sort of personality who is attracted to academic echo-chambers is intelligent but vain, deeply insecure and manifests as arrogant. That's why they're so insecure about anything perceived as a challenge. The way to handle them is praise and stroking their ego, while politely pointing out that their work duplicates prior work. – smci Jun 15 '15 at 21:02
16

I will add my different perspective on the OP's question. @PeteL.Clark answer pretty much covers completely the one side of the story: The OP has discovered some legitimate inconsistencies / mistakes on published papers and wants to make thing rights for the community.

But as we know there is always a second version to each story. Let's discuss some of those aspects as well.

Someone hired me from a freelance site to implement a previously published paper. The paper doesn't adequately describe the novel methods they claim. I spend a week on it but don't get paid.

This does not prove that the paper does something wrong. The assumption that one programmer can implement a method described on one paper that may build on several other papers within a week, is unrealistic / naive. CS scientists may spend 3-12 months working on a paper that may build on previous works that required several other months to implement. A CS paper is not a standalone piece of engineering that needs to describe everything. If it is based on previous works, it only needs to say what it does in addition to previous works. Therefore, in order to fully implement a method described in a paper, you also need to implement all previous methods this paper builds upon. And this takes months to years for newcomers to the suggested area, even if they are good programmers. In other words: Being a good programmer does not guarantee that you can implement every CS scientific paper there is.

Similar situation but the implementation's accuracy doesn't match the claim.

Again that does not guarantee that the paper has done something wrong. Perhaps the OP has done a wrong implementation of the created method. I find very strange the fact that the OP, assumes first that everyone else is wrong and has total confidence in his work, abilities and implementation, even if they contradict published work by many other authors. Assuming everyone else is wrong and my code / implementation is always correct, is not only a characteristic of cranks but contradicts the basic principle of software engineering: All codebases have bugs and first you should look for bugs in your codebase and then look for bugs elsewhere.

...which I pointed out as being logically wrong. The guy turned personal and claimed that he knew better because he had twice the number of papers than me. And later when I read his papers I found that the papers din't contain any ideas that would count as a genuine research.

What does logically wrong means? Did you manage to do experiments and prove things over a tea table? Good for you. And what is this? "The papers didn't contain any ideas that would count as a genuine research". What is genuine research? Is this another version of the No true Scotsman argument? If I do not like someone else's research, I can belittle it by saying that it is no genuine research?

Use new datasets without releasing them publicly or describing them in detail

Again, this is the standard argument of newcomers. "Freely release your code and your datasets so I can use them but I will keep my code by myself so nobody steals it". The main argument is "You have already published your work and I have not, so you must help me". Again, this line of argumentation is wrong. An author has the exclusive right to share or not share his code or datasets. And has the OP asked the authors to give him access to their code or datasets? The fact that someone does not release the code publicly does not mean that if contacted, he will not share.

Conclusively, in most CS PHD students meetings I have encountered, there is always the "purist" guy who insists that he is only interested in A* conference publications, he only wants to do top publications, he is working 2 or more years in his "seminal" paper, most works of his peers is mediocre (as the OP described "not genuine research") and he will never work on incremental papers, since publishing something less than perfect is not suited for him. Unless this guy is exceptionally talented (which is easy to tell by his previous publications) this is also the guy that needs 7 years to get his PHD (if he actually gets one) and has an average h-index of 2, when finishing. Please, do not be that guy (unless you are that talented). Focus on publishing your original work, instead of trying to find errors on everyone else.

  • 5
    @AdityaJoshi There is no need to give actual details. As a rule, I would only care to implement methods in papers from a) reputable authors I know b) from reputable conferences (not necessarily top but good ones) and never really cared for obscure workshop papers. These criteria will filter 99% of most junk papers. – Alexandros Jun 15 '15 at 10:08
  • 1
    yup! I think that's my takeaway from this question. – Aditya Jun 15 '15 at 10:10
  • 3
    The assumption that one programmer can implement a method described on one paper that may build on several other papers within a week, is unrealistic / naive. – As far as I understood it, the OP spent one week on the paper and then decided that it was not implementable given the lack of information. While this might still be due to mistakes from the OP’s side, it does not sound too unrealistic to me. – Wrzlprmft Jun 15 '15 at 11:14
  • 5
    An author has the exclusive right to share or not share his code or datasets. — Many computer scientists (like me) believe that researchers have a responsibility to publicly share their code and data. Unfortunately, your attitude is more common. — You have already published your work and I have not, so you must help me — Indeed, that is precisely what research is for: helping other people. – JeffE Jun 15 '15 at 15:42
  • 3
    @JeffE While that's an ideal to strive for, reality isn't always so nice. Sometimes publishing data would violate privacy expectations or even laws. On the code side you might use some proprietary libraries or tools you don't have the right to publish. Still I'd like scientists to publish as much code and data as possible. – CodesInChaos Jun 15 '15 at 19:48
8

A somewhat "indirect" challenge is to is to publish your comments on a website such as PubPeer. It has been historically used to comment mostly on papers in the area of medical research, but it also hosts discussions regarding papers in computer science. A full list of the journals that have comments regarding specific papers is available on that site as well.

In case these discussions point to substantial flaws in a paper, they can be forwarded to the journal editor/publisher.

  • 1
    Good link. I encourage people to up-vote the Pubpeer add, since there are a number of questions about pointing out bad research. – Cape Code Jun 15 '15 at 7:46
5

To answer the question from a more technical point of view: Some journals allow you to publish comments on papers (though I learnt from the comments that the prevalence of this practice varies strongly depending on the field). They usually are like regular publications, having citations, figures, tables and whatever you need and being subject to peer review. Depending on the size and relevance of the criticised paper, such a publication might also be a regular paper.

In your example where a method’s accuracy didn’t match the claim, you can probably publish a comment on this, substantiating your opposing claim with data or similar. Something similar goes for methods that are identical or very similar to existing ones without crediting them¹. However, I consider it unlikely that a journal will publish a comment on the inadequateness of a method’s description. In all cases, it might be sending an outline of your comment to the journal and ask if they would publish it.

Now, given the huge amount of bad papers out there, you could spend all your life writing comments exposing fundamental flaws in papers in your field. Unfortunately, there is no financial and little academic credit for doing this, so it is rarely done. However, if you already spent some time on trying to make use of a paper, you may already have most of what you need to publish such a comment and you may consider it worth your time.

Some modern journals (e.g., Plos One) also have online interfaces for instant comments that allow for criticising a paper with much less effort.


¹ Consider the possibility of an independent discovery though.

  • 6
    "Most, if not all, journals allow you to publish comments on papers (I do not know how this works for conference contributions)." Definitely not all. Whether it's "most" depends on your scope. In my field (mathematics), it is very rare for reputable journals to publish comments on papers written by anyone other than the original authors. (I am not defending the practice, but just pointing it out.) – Pete L. Clark Jun 15 '15 at 12:36
  • 4
    I have never seen a computer science conference accept or publish "comments", except for a few extremely rare errata submitted by authors. – JeffE Jun 15 '15 at 15:41
  • I second JeffE's comment. 95% of the journal papers that I see in CS are either published by Springer, by IEEE, or by Elsevier. None of them allow comments. This should be the same in many subfields of CS. Actually, this was already discussed here (comments of the original posts): academia.stackexchange.com/questions/32406/… – DCTLib Jun 15 '15 at 19:13
  • @DCTLib: That question was more about instant comments as mentioned in the last paragraph (at least that is how I understood it). Anyway, you seem to be right about journals accepting comments not being as prevalent as I estimated. I amended my answer. – Wrzlprmft Jun 15 '15 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.