Institute II is setting up a benchmark project under the lead of main-author MM. 20 other developers participate, run their methods without having access to ground truth data and also MM contributes with his methods results.

A paper is written by MM. Is it the right of one or all of the other 20 developers (results have been submitted to MM already, so there is no reason that tuning/overfitting can happen) that he gets access to GT data to check if the analyses of the results are reasonable?

The best source for publication ethics known to me is Cope, the Committee on publication Ethics. They have a document for author guidlines (PDF), where it is recommended that an author submits "A declaration that that person takes responsibility for the integrity of the paper".

This is like always in Ethics only a recommendation, the journal obviously has to do here the final decision.

In my opinion I can try to get access to the GT by writing MM a mail, nevertheless if not granted, I cannot personally assure the integrity of the paper and have to retract my authorship. Or I get in contact with the editor of the journal if MM is not cooperative and let him decide.

Am i right about my conclusions? Are there other strong widely followed ethical guidelines about this issue?

Thanks for further recommondations.


In order for the benchmarking exercise to be convincing to readers, it will ultimately be desirable to release all of the data (and presumably code) so that the comparison can be replicated.

There's no reason that the ground truth data couldn't be revealed to all participating researchers after they've done their analysis and the results have been frozen but before the submission of the paper.

  • I absolutely agree here, it will addtionally make future methodological advances or completely new developed methods impossible to validate against those published results. Sep 19 '17 at 11:50

The co-author certainly does not have a legal claim over the data.

With regards to ethics, it strongly depends on the communication beforehand. Was it made clear that the data was not accessible to co-authors? Or could it have been accidentally implied they will get the data? One way or another, this is something that the authors have to discuss. The owner of the data should give a good reason why the co-author cannot get the data.

It is likely, though, that the co-author will be disappointed or upset that they do not receive the data.

  • 1
    Even though a co-author has no legal claim over the data, I find it strange to not have shared access in the first place. Depending on the field that might be normal, though. Sep 19 '17 at 11:19
  • @AttilaKinali I think it is okay to prevent access before participants submit their results. On one out of 24 data sets GT access was granted to check for example if results can be formatted in the requested way. The main author though is as well a participant (having full access to everything), and his method is winning in nearly every aspect. My method shows weaker results partially than on other validation sources. That makes me curious... Sep 19 '17 at 11:48
  • 1
    It's generally best to have a neutral third party organize this kind of exercise. The challenge in doing this is that you need someone who is quite familiar with the subject but doesn't have their own method to test. Being the neutral third party is a somewhat thankless job... Sep 19 '17 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.