Tweeted twitter.com/#!/StackAcademia/status/447326661756063744
2 formatting
source | link

For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy.

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen.

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are:

a.) Is this common for groups (esp in Bioinformatics) to select 
    a biased dataset as input and claim it to be random?

b.) Does reviewers really run the tools (or read the source code)  that are 
    accompanied by the paper in order to test it and see whether it really does what it 
    claims or they only read the paper?

c.) Can an end user later write to the editor and let him know that the tool 
     published in their journal isn't what it was suppose to be?
  1. Is this common for groups (esp in Bioinformatics) to select a biased dataset as input and claim it to be random?

  2. Does reviewers really run the tools (or read the source code) that are accompanied by the paper in order to test it and see whether it really does what it claims or they only read the paper?

  3. Can an end user later write to the editor and let him know that the tool published in their journal isn't what it was suppose to be?

For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy.

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen.

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are

a.) Is this common for groups (esp in Bioinformatics) to select 
    a biased dataset as input and claim it to be random?

b.) Does reviewers really run the tools (or read the source code)  that are 
    accompanied by the paper in order to test it and see whether it really does what it 
    claims or they only read the paper?

c.) Can an end user later write to the editor and let him know that the tool 
     published in their journal isn't what it was suppose to be?

For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy.

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen.

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are:

  1. Is this common for groups (esp in Bioinformatics) to select a biased dataset as input and claim it to be random?

  2. Does reviewers really run the tools (or read the source code) that are accompanied by the paper in order to test it and see whether it really does what it claims or they only read the paper?

  3. Can an end user later write to the editor and let him know that the tool published in their journal isn't what it was suppose to be?

    Post Migrated Here from academia.meta.stackexchange.com
1
source | link

Are datasets really random for Bioinformatics tools

For my masters thesis, I needed a to run a tool to filter out RNA sequences and then research on the filtered data. I researched online reading papers to find an appropriate tool and I found one that really fitted my needs. The tool was published by a renowned group in a good journal (IF around 7). The tool claimed to have been run on a 30 random datasets and claimed to have around 90% specificity and sensitivity. I was very happy.

When I ran it on my dataset, the tool produced so many false positives and false negetives. Its accuracy was less than 10%. I ran on other datasets too and never got an accuracy of more than 20% in any one of them. To my surprise, the tool only worked well on the dataset that was published with the tool, with a good accuracy and no other input. I cannot believe that the dataset was randomly chosen.

I spoke to my PI about this and he said that he understands. He allowed me to choose another topic for my thesis and helped me a lot to complete. In the end I could successfully defend my thesis.

My questions are

a.) Is this common for groups (esp in Bioinformatics) to select 
    a biased dataset as input and claim it to be random?

b.) Does reviewers really run the tools (or read the source code)  that are 
    accompanied by the paper in order to test it and see whether it really does what it 
    claims or they only read the paper?

c.) Can an end user later write to the editor and let him know that the tool 
     published in their journal isn't what it was suppose to be?