Take the 2-minute tour ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

I was recently going through an article (which I was told is fundamental in the area of my new project) published in a respectable medical journal. In short, the paper is about the similarities between a particular sub-type of tumor cells versus a particular type of stem cells in the body, then they go on to investigate what that similarity might indicate.

Barely two pages in and I realize that the authors omit what I consider to be critical data when motivating the use of two particular proteins as markers in establishing their fundamental assumption, that their immunohistochemistry findings are representative of the reality and that their model is valid:

Accordingly, we screened all known ..... markers against our ..... data to determine which, if any, decrease with differentiation (data not shown). Of all the potential ... markers, we found that X and Y are the best candidates, and they were therefore used in this study.

Now it might not be a big deal to some but I find it important to see that omitted data, since:

  1. I am not that informed in that particular type of biology

  2. Seeing that their "real" findings are built on the results of the aforementioned screening, the validity of their research is practically depending on the decision of using those two markers.

Which brings me to my question: Are there any cases where not showing "critical" data in such a manner is acceptable or common? I know that some journals do not accept "data not shown", but obviously it does happen in better journals as well...

share|improve this question
    
I am not in any way familiar with biology, but it seems odd to refer to "all known...markers" when, apparently, there are only two markers of interest. Are you sure what you quoted from the paper is accurately reflected by what you understand is the authors' methodology? –  Nicholas Feb 26 '13 at 16:07
    
I think I got the methodology right, I had skipped the next line which might have caused the misunderstanding. Please see the quote now. –  posdef Feb 26 '13 at 16:09

5 Answers 5

up vote 11 down vote accepted

I think it is perfectly reasonable to not "show" data. For example, consider a multivariate randomized control trial with two groups. If there are a large number of partially correlated dependent variables, the data are not suitable for graphical or tabular presentation. What we are potentially interested in is if there is a difference between the two groups. We can "see" that from a single sentence about a statistical test and a comment that the data are not shown (so people do not go looking to find the figure).

In a biology example, maybe you are counting the number of intact cells after two different treatments. There may be hundreds of slices that result in two numbers. What exactly do you want to see?

share|improve this answer
    
Interesting, I had not thought of the data presentation problem, even though I doubt that is the case. This field of biomedical research is riddled with HUGE pdfs with cluster graphs and heatmaps, you name it... It has not hindered people from having 900+ pages of supplementary (yes, I did come across such a paper). Long story short, yes I guess it might be an issue in some cases. –  posdef Feb 26 '13 at 16:29

Critical data? Probably not. But, I have seen many papers with data left out simply because of paper length restrictions, and in all cases other graphs and data appropriately contributed to the papers' fundamental arguments.

In the cases where I wanted to see the data that was omitted, I've written one of the authors and they have almost always forwarded it along.

share|improve this answer
    
how about supplementary data then? I mean there usually isn't a limit on supplementary data, right? –  posdef Feb 26 '13 at 16:07
    
@posdef I think it goes back to the question of whether it is critical or not. If the authors didn't think it was critical, and the reviewers/editors accepted the paper, then the peer-reviewed answer is that it wasn't critical. –  Chris Gregg Feb 26 '13 at 16:09

What constitutes "data" is a highly subjective issue. If "data" means "all scientific output," in many cases this exceeds what can reasonably be shown in a paper.

In molecular simulations (my particular field), we may generate gigabytes or even terabytes of data for individual papers. This data needs to be crunched down and represented in figures that process the data and make sense of it to the reader, as it is plainly impossible to show the reader the same data over and over again. Thus, we choose to show only the most essential information, rather than deluge the reader with more information than can be handled either visually or in tabular form.

share|improve this answer
    
I sometimes start with gigabytes of measured raw data, even before any bootstrapping of models or whatever takes place... Clearly outside usual sizes for supplementary material. –  cbeleites Mar 4 '13 at 18:51

It is of course difficult to judge this particular paper without having seen it so I will stick to general terms.

I do not see any reason for not representing all data in some form. When I say represent I mean that the complete data set can be given in terms of, for example, statistical measures which would at least provide some way to see or judge whether the subset shown is truly representative. But, I still have a hard time seeing a good reason for omitting it without clearly explaining on what grounds.

Journals allow supplementary information and so it seems reasonable to use that option if it exists in th eparticular journal. In very old papers where plots were mae by hand, there may be some excuse if not everything is included.

As I see it one of the fundamental principles of scientific publication is reproducibility. That requires access to all data. There are of course instances where this i snot possible such as when patient journals are involved.

share|improve this answer

From the way you've described it, it sounds like the choice of markers was somewhat arbitrary. Presumably they had figures that demonstrate that the markers they did choose are good at marking whatever-it-is that needs to be marked. Thus, the fact that they looked at howevermany others is immaterial to the scientific content of the paper and is rightly omitted.

Now, that data could well be useful to someone else for whom markers X and Y are inadequate (maybe because of expression problems or tissue type or whatever). It would be useful to the scientific community to know what that data is (if indeed it was done carefully enough to be worth anything--maybe it wasn't, but it didn't matter because X and Y checked out). But there's no requirement to be especially helpful to the rest of the community (or to avoid being sloppy in some areas as long as you go back and do it carefully/right once you know what you're doing). So it seems reasonable to me.

(Again, inferring from what you've said. If that screen was what told them that X and Y had the properties that they thought, and there is no other confirmation, then they'd better show that data in convincing detail!)

share|improve this answer
    
"Thus, the fact that they looked at howevermany others is immaterial to the scientific content" not necessarily: looking at howevermany may constitute a multiple testing situation. And then it does make a difference whether 10 markers or 100s or 1000s were screened. See e.g. nature.com/news/2011/110323/full/471428a.html However, the fact that they are not reported points to rather larger numbers I guess. The question is whether later on comes a really independent confiramation of the required properties or whether this is not independent of the screening data. –  cbeleites Mar 4 '13 at 18:48
    
@cbeleites - It's irrelevant how many you look at if your screening criteria are different from your criteria used to confirm. Otherwise, yes, the statistics are rubbish. I said as much in my last paragraph. –  Rex Kerr Mar 4 '13 at 20:44
    
I read that last paragraph. But I think data dredging is so widespread a problem that it needs to be emphasized far more than by a paragraph in parentheses. I'm biased, for the moment I tend to believe that if such a paper doesn't have an explicit sentence stating that there is no problem because the final testing was independent of the screening, I tend to assume that it probably wasn't. This may be wrong for the biological field in discussion, though (I'm not a biologist, but my field interacts with biological/medical questions). –  cbeleites Mar 4 '13 at 22:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.