22

I am trying to reproduce some of the results in a paper from 2018 where the results were included in a graph but no exact values were given. There is no results table to go with the graph. I can't exactly compare what I produce with my guesses as to what the values in the graph are. They also did not specify one of the parameters used to produce the results.

I have emailed the authors multiple times and they keep ignoring my question about sharing these values with me and apparently do not have the parameter values as they don't have the logs any longer. I am unsure how to proceed with this. It's a paper published in a high-impact well respected journal.

Any help or guidance would be appreciated. I am not being rude or brash about the way I've asked for these results. I just explained I want to compare to them and can't use the graph because I need some level of accuracy.

Improve this question
  • 10
    Its unclear how you know that they don't have the lgos any longer if they are ignoring you? – Ian Sudbery Nov 1 '20 at 21:27
  • 12
    "they don't have the logs" - does anybody honestly believe this? It's excusing malice by pleading incompetence. If the journal truly is respectable they'd have open data rules, can you force these people to properly show the work they're happy to reap publication prestige from? Also be aware of course, that by asking uncomfortable questions about peoples' data you burn bridges and make enemies. Science talks a game of openness and scepticism; the reality is more like @Buffy's first couple of paragraphs. Get used to that kind of post-hoc excuse – benxyzzy Nov 2 '20 at 6:42
  • 10
    What is your question? This is a question-and-answer site, so we require you to articulate a specific answerable question. I don't see a question in your post. "Any help or guidance would be appreciated" are too open-ended to be a good fit here. See our [help/how-to-ask]. – D.W. Nov 2 '20 at 8:13
  • 14
    @benxyzzy “does anybody honestly believe this?” — Yes. Anybody who has ever worked in science should believe this, because it’s depressingly common. It’s not malice, it’s just standard incompetence, and — at a close approximation — every starting scientist has done this at least once. I know I have, and I know many of those I did my PhD with have as well. I’m not excusing this practice but I can confidently state that it doesn’t require malice or even negligence other than what’s explained by lack of experience. – Konrad Rudolph Nov 3 '20 at 14:21
  • 5
    Just to second Konrad's comment - it's stupendously easy to lose data. Just because the paper was published in 2018 doesn't mean the data was that recent. It could have closed in 2016 and taken ages to review. Could have been based on a student's project who didn't organize very well. I've seen far more poorly organized labs than well-organized ones. – Azor Ahai -him- Nov 3 '20 at 15:44
41

There is no results table to go with the graph.

Use Datathief: https://datathief.org/

Despite the name, in most countries using Datathief is explicity allowed by copyright law. Data cannot be copyrighted.

They also did not specify one of the parameters used to produce the results.

Two possibilities: it does not actually matter or peer review has failed. When it is your turn to peer review, check for this type of mistake. Sorry I cannot not help in this situation.

Paper authors can choose not to answer their email if they wish.

Improve this answer
  • 4
    academia.stackexchange.com/questions/7671/… – Anonymous Physicist Nov 1 '20 at 23:07
  • 1
    Datathief works well; in the old days we would have manually read the data off the graph at key points for comparison. It may be that the authors aren't used to open data or sharing their underlying data - and if the journal has a page limit there's no room for a table, unless supplemental information is an option – Chris H Nov 2 '20 at 8:50
  • 9
    A good opensource alternative is WebPlotDigitizer. – Anyon Nov 2 '20 at 14:08
  • 10
    'Despite the name, in most countries using Datathief is explicity allowed by copyright law. Data cannot be copyrighted.' Although for the avoidance of doubt, this doesn't remove the ethical requirement to cite the data source when you use the data. – Daniel Hatton Nov 2 '20 at 16:40
19

Maybe not very nice, but hardly "misconduct". You might need to reproduce more of their "experiment" to get new data. Their data is their own, I think.

It may be that the reviewers saw more than the authors are willing to share publicly. And it is possible that they have future plans for the data that would make release at this time unwise.

If you want to reproduce the research you will need to do more than re-use their data. In fact, the results you would gain by looking at fresh data would likely be more valid than reusing theirs. You might find, in fact, that their conclusions aren't supported, assuming it is a statistical argument. Or, you might give it additional credence from using different data.

Improve this answer
  • 1
    I don't think it is misconduct in any way - I didn't say that, or I hope it didn't come across that way. Is it not the same as the results they've already shared in the graph though? But in numerical form? My thoughts were that it's purely just that they've already 'shared' those results just in such a form that they're not useful for numerical comparison. I'm not after any of the actual data, just the numbers to match the graph and if possible the parameters to reproduce those numbers. – Espore Nov 1 '20 at 21:53
  • 10
    You had it tagged misconduct originally. I was responding to that. – Buffy Nov 1 '20 at 22:28
  • 7
    Not sharing the raw data associated to a plot in a published paper maybe isn't misconduct, but I find it definitively shady. I think that journals ought to require that the raw data used for a plot, and all the analysis pipeline and generation code used to create it, should be submitted to the journal and made public along with the paper, in the name of reproducibility. Sadly, most journals don't currently bother with this. – a3nm Nov 2 '20 at 10:48
  • 7
    I'm not sure its even "shady", its probably just lazy and is definitely poor practice. Requiring all raw data behind every figure is becoming more common. For example, Nature journals require this now. – Ian Sudbery Nov 2 '20 at 14:52
  • @a3nm The journal is NeurIPS – Espore Nov 4 '20 at 0:14
13

Authors are not in any way obligated to comply with random requests for additional information about their paper, nor do they even need to respond at all.

If they provide what you ask for, it would be them doing you a favour.

Their obligation is to the journal and the reviewers. This obligation mostly ends after they've made the reviewers happy and their paper is approved for publishing.

Some egregious problems might cause a paper to get retracted, but some missing data probably doesn't meet the threshold for that.

If contacting them has not worked for you, then you're probably out of luck. Contacting them multiple times is likely to only annoy them.

This applies in general. Specific journals or jurisdictions may have different policies regarding the above.

Improve this answer
  • 6
    In some circumstances, in some jurisdictions, authors/institutions are obliged to respond to Freedom Of Information requests. – Bob says reinstate Monica Nov 2 '20 at 17:04
  • 2
    @BobsaysreinstateMonica indeed, some funders make data sharing a condition of funding. At the research funder I work for we will if necessary take an appropriately anonymised copy of study data and hand it over to other reputable researchers if the original data owner won’t do it. Luckily we’ve never had to - once the original researcher realises what’s in the contract they produce the data, on those rare occasions they didn’t want to share. – rhialto Nov 2 '20 at 19:58
  • 5
    “If they provide what you ask for, it would be them doing you a favour.” — Nah, that’s nonsense. It’s not a favour, it’s good scientific practice. Even if they’re not legally obliged, they’re ethically and professionally obliged to respond to earnest requests by peers. They only don’t need to respond to crackpots. – Konrad Rudolph Nov 3 '20 at 14:24
  • 3
    @KonradRudolph It just seems frustrating when im trying to spend my time reproducing their work - something which should be helpful to the research community. I'm only asking for the numbers for the graph which was already presented. It's almost just like asking for a higher resolution graph in a sense. But also the parameters which they describe, provide a formula to calculate, but don't say what values they used to produce these results - which seems odd to me. – Espore Nov 4 '20 at 0:17
7

Check the journal's guidelines and/or contact the editor. Sometimes journals require data and code to be made available (which is obviously the right thing to do) and authors that refuse to do so are breaching the agreement under which they published.

If the journal does not, then you are out of luck. Sadly there are many researchers out there who think science is a race and will do everything in their power to be the ones ahead. This is reprehensible but unavoidable.

I'd refrain from using their results and actively call them out if you publish anything similar, i.e.: attempts where made to reproduce the results from XXX et al. but the authors refused to share their data/code/methodology.

Improve this answer
  • 1
    The journal is NeurIPS. I can't see anything on sharing data from my checks. If i intended to publish something similar how is it best to approach this? I wouldn't put something so straight forward, but perhaps "we were unable to obtain results from XXX and so we reproduced them.." – Espore Nov 4 '20 at 0:20
  • 1
    You could contact the editor and ask directly. What you propose looks good to me. – Gabriel Nov 4 '20 at 12:38
  • 2
    "call them out" still gives them cite points and hence maybe a wage increase etc. – Ian Nov 4 '20 at 16:32
  • 1
    Ian that's a valid point, but there's no way to get around it unfortunately. Not mentioning the article (the authors really) is the same as giving them a free pass at doing it again. – Gabriel Nov 4 '20 at 17:41
1

Usually research communities are small and you will see each other at the next conference or the conference after that. Than you can ask them in person. Either informally between the sessions or when they present new research based on their paper. It is way more difficult to brush someone away in a room packed with respected peers who will be the referees for their next paper.
But act carefully, you don't want to affront them and turn them into an enemy!

Improve this answer
-4

So do your own experiment and make your own graph. Compare the graphs. Why do you need the specific data anyway? There will be noise aka experimental errors in all the data so they wont be exactly the same every time which makes needing the original data irrelevant.

Improve this answer
  • Compare how? I can approximate their numbers but that just seems so messy – Espore Nov 6 '20 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.