8

For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy.

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen.

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are:

  1. Is this common for groups (esp in Bioinformatics) to select a biased dataset as input and claim it to be random?

  2. Does reviewers really run the tools (or read the source code) that are accompanied by the paper in order to test it and see whether it really does what it claims or they only read the paper?

  3. Can an end user later write to the editor and let him know that the tool published in their journal isn't what it was suppose to be?

migrated from academia.meta.stackexchange.com Mar 21 '14 at 12:51

This question came from our discussion, support, and feature requests site for academics and those enrolled in higher education.

  • 1
    Please note that on StackExchange sites it's common to not "chain" multiple questions into one super-question, but ask them separately (see: meta.stackexchange.com/questions/39223/…) – posdef Mar 21 '14 at 13:13
  • 1
    Can an end user later write to the editor and let him know that the tool published in their journal isn't what it was suppose to be? yes, and you should if you suspect foul play (cherry picking). Talk this over with your advisor, first. – Marc Claesen Mar 21 '14 at 13:27
  • 2
    Have you asked the authors of original paper for more details (regarding how their dataset was chosen, how to apply their tool, and what its limitations are)? It sounds like you suspect them of something between incompetence and fraud, and it would be best to investigate their side before accusing them of anything. – Anonymous Mathematician Mar 21 '14 at 13:54
  • The tool may have worked well on that dataset because it only works well on datasets with certain characteristics. Did the paper in question make no reference to what qualities a dataset would have to possess for the tool to perform well? Unfortunately, such tools can have a lot of specificity - so one has to know when it is appropriate to use a certain tool. – Faheem Mitha Mar 21 '14 at 17:44
  • 1
    You do not mention the tool or anything about your data, but in case the tool was built to work on prokaryotic data and you have eukaryotic or the other way round this might be a problem. – skymningen Nov 30 '16 at 13:19
9

What you are describing here is an in silico form of Cherry Picking Fallacy. Before answering your questions specifically, I would like to point out that before assuming bad intent or foul play, consider the biological variability in the datasets, and difficulties writing generalised solutions to biological problems. Take it from a 3rd year grad student and a bioinformatician in training, it's not straight-forward to develop tools for biological datasets. There are both biological and technical challenges in writing "foolproof" software in bioinformatics.

Consider for instance, that there are multiple ways to estimate error and false discovery rate (FDR). There are also varying levels of thresholds used in different labs. Any heuristic value, if hardcoded, might alter the results.

Similarly the biological diversity, as well as technical variability introduced by wetlab benchwork will likely have serious influence on the datasets. Once everything boils down to a large table of numbers, all of that variability is implicit and thus often forgotten.

That said, my answers to your question(s):

  1. No it's not common to pick datasets just to show that your model/software works, at least not publicly. That would go against scientific rigour and ethics. At the same time, you start from what you have available. I typically start with datasets we have in the lab, when I start developing a new tool. But in order to publish your method, you should typically show that it works on data from independent sources, or alternative data types etc. All in all, your results might not be representative of all datasets out there.

  2. It is impossible to answer this question, factually. I am sure there are reviewers that take a look at raw data and/or source code, but it's unlikely that any reviewer actually digs into the source code, line by line, to see if everything checks out. But I am also certain that there are some who just see if you have actually made the source code or test datasets available or not, without actually look at them.

  3. You can always do that, end-user or not. But beware, before you go blowing the whistle on someone else's handiwork (which probably took them a significant amount of time and money) you better get your facts straight. Claiming that they are selling false results or simply misinterpreted their results is a very serious accusation, especially if the authors are renown and respected in the field.

Prior to writing a letter to the editor (who AFAIK have a duty of publishing scrutinising critique on something that has been published in that journal) you should check, and double check, your results. Make sure you are not misunderstanding the statements made in the original work. Make sure you are not misunderstanding your own results. Make sure what you have observed is not a side-effect of your datasets or your own use of the software, try to talk to other people who have used the software. And make sure, your supervisor (and others in the group) are behind you, get your own results investigated by your superiors and colleagues.

If everything checks out you can, and should, inform the journal that the results in the original article might not reflect a general reality, and that you could not replicate the results on independent datasets, then show your results.

  • 5
    As well, I would suggest it's probably worth while to get in touch with the original authors if you believe their system is faulty after more investigation. It could simply be a misunderstanding of the properties the data needs for the tool to be applicable -- bad writing/editing can really destroy the meaning of technical terms sometimes! – Matthew G. Mar 21 '14 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.