Bad data - how to move on successfully

Question

I work in an interdisciplinary field. My input is not generated by myself, but by talented people that I trust and that trust me to analyse their data and generate fascinating insights.

But here I am, once more stuck with a project where the input is bad. There is no use for blame and finding a scape goat, we are in this together. And people learn. But I have been stuck in projects with big promises and bad/insufficient input since the start of my PhD. I moved to another place for a PostDoc now, but this situation seemingly haunts me wherever I go.

There is not much I can achieve when most of my projects stop after the input quality control. But if I want to stay in academia, I desperately need to step up my game in actual output, not just in my ability to troubleshoot, right?

How do I move on from this? What chances do I have if I can just never land a "prestigious project" that results in a valuable publication? Is there a chance to still build up a good scientific reputation without those? Should I try myself at writing a review? Take on projects until one finally works out? (But how long will I succeed in getting another job if they don't?) I could work with published data for a bit, but it is often not comparable between studies and severely lacking metadata.

The question is, do I have to accept that success (as in, basically being able to stay) in academia is to a large degree based on luck and I am not one of the lucky ones or is there anything major I can do? I really love the work I do, I would like to keep going.

So far I have taken on additional projects, tried to do my own 'side projects' on at least roughly usable parts of the data in the hope I will find a better dataset eventually where this could come in handy and kept in touch with collaborators in an effort to troubleshoot and eventually produce better input.

EDIT: To address some questions: I produce analysis pipelines, partially based on my own methods. Without "real data" application it's hard to publish those in my field. Yes, it is "real world data". I do not expect perfect data at all. But I do expect technically correct, usable data. If the input is random/to few features to be statistically relevant there is nothing I can do, though. Imagine trying to do a statistical test on the similarity of blog posts based on word usage written by different groups of people but many "groups" only are represented by two authors, the text is sometimes just one sentence long and quite a few of the posts looking like they are produced by a random letter generator, not having any actual words in them. While I was promised at least 5 authors per group, minimum 5000 words per text and of course the post actually written by the author assigned to it.

Answer 1

Summary:

• Real research life with real world data is messy*, and there will hardly ever be sufficient samples (my very personal prediction).
• There are huge opportunities (and needs) in working about small and messy data. Maybe that could become your area of research?

• Good data analysis work requires close collaboration. Actually already when planning the experiments, but for sure during data analysis.
  Close collaboration will allow you to make them aware of your needs and that data analysis cannot work miracles. It is also necessary for your because otherwise you may be employing inadequate analysis methods.

  * When I say messy I don't mean bad curation (although I do see opportunities here as well - though maybe more business than research) but reality sneaking in with lots of influencing factors creating structure in your data where many (most?) data analysis approaches assume nicely independent data. I think this is a field that not only deserves more research but also that this has high practical importance.

---

I feel your pain. Been (almost) there as well. Actually, I still am (just my PhD is long done): so far with ≈ 15 years of professional experience in chemometrics, all the real world data I've encountered so far has one thing in common: too small sample size (even if it may look nice at first glance).

• One consequence I drew for me is that I started to do research on such less-than-ideal situations I encounter in practice, e.g.
  • small sample size situations: knowing that I have far too few cases (some orders of magnitude below rule of thumb recommendations), how to diagnose when things break down, how to stabilize models, what will break down, are there hard limits, etc.
  • in terms of messy data in the sense above (with lots of influencing factors): I had situations where it turned out that biology doesn't really obey the disease classification medical doctors use (that was developed for totally different purposes, as I learnt later on) how to adapt data analysis methodology to these situations (that were somewhere in between classification and regression)
  • how to adapt validation/verification procedures in such situations
    (I work very much along the lines that you can to whatever you think may work for modeling as long as you do a honest verification and validation of that model)
  • I see tons of similarly important questions that are unanswered.
    To the extent that if you need such research ideas, I'd happily supply you with questions ;-)

---

• In my field, I think, it is going to stay like this: well-characterized samples are expensive.
  In some respect you may even say that basic restearch isn't meant to have comfortable sample sizes. It's meant to find basic knowledge and point out promising possibility but the leg work of obtaining (and paying for) large sample sizes to make a method robust for routine use is something applied research/industry is supposed to do (and pay for). That point of view would say that taxpayer money should not wasted on work that industry could and should do.

• On the other hand, I often see unnecessarily too small sample sizes in academic research: too small here meaning that given the sample size even without any experimental data it is (or would have been if one had bothered to check) clear that no knowledge is gained because the study is too underpowered. This is clearly just bad science, and a total waste of experimental and data analysis effort.
  If that's what you refer to in your question, it's going to be hard work to improve this - but don't give up! Science needs people like you pointing this out.

  My experience with that is that as PhD student or fresh postdoc how much you can actually do to improve the data may depend very much on how much weight what you say has with your supervisor (or even top level director).
  What you can (and should) always do is to clearly discuss the limitations in terms of possible interpretation of the results of your study - including in the manuscripts you write.

• To be fair, there are practical limitations. If we study a rare disease where the big university hospital gets maybe one sample per year, I tend to think that working with very few cases is necessary (but again: spell out the limitations). After all, one has to start somewhere.
  Whereas, if we're talking easily accessible measurements of no particular ethical concern of a disease where the hospital sees 10s of cases per week then of course a thesis on 5 cases looks somehow lazy (not necessarily on the PhD student's side, though: The PhD student may not have been able to change pre-existing sample plans)

• One consequence for my PhD thesis was: as I did not only data analysis but also sample preparation and measurements for my thesis, I put in substantial effort to have more samples (fortunately I had access to a comparatively large data bank, but in the end also that approach was limited by availability of the more rare conditions).
  I'd recommend to at least take a decided interest how the data are generated (get a lab tour, get the collaboration partners explain how things work and what the data mean).

---

 If the input is random/to few features to be statistically relevant there is nothing I can do, though.

Yes. Again, this needs to be clearly communicated: I do have the experience that applied groups may expect miracles from data analysis (and you may even have a particularly uphill fight here if this group in the past got data analyses that were badly overfit and thus looking overoptimistic and noone realized this).
In addition, you'll have to document that it isn't your "fault" that no nice results come out of this data. It's doable though (and again in my experience something that is needed in everyday data analysis work as well: I'm just having such a situation on my desk right now again).

[...] "groups" only are represented by two authors, the text is sometimes just one sentence long and quite a few of the posts looking like they are produced by a random letter generator, not having any actual words in them. While I was promised at least 5 authors per group, minimum 5000 words per text and of course the post actually written by the author assigned to it.

A few thoughts about this. I "smell" some communication/collaboration issues here. Again, they are typical everyday issues in my research and data analysis work:

• I've met similar things due to fundamental communication issues (e.g. between statisticians talking about necessary sample sizes in the 1000s to answer a particular question and experimental scientists "translating" this to "many" ≈ 7.

• "quite a few of the posts looking like they are produced by a random letter generator"
  Your experimental collaboration partners may have no clue what you don't know about their techiques (again communication issue): unless you have a background in these techniques you have no chance to recognize what is going in those noisy measurements and how to deal with them.
  They could be anything from artifacts that should be deleted because the underlying mechanism that causes them is well known and can be disregarded over "the signal is hidden under this noise and you data analyst surely have some magic to get it of there" (not going to work, but typical expectation) to "your outlier is my most interesting case" - without the help of the experimental/data supplying folks you won't be able to adequately deal with this.

• Seeing all this makes me think whether you do have sufficient information about the background of the study to even decide which data analysis approach is suitable?

Answer 2

If you are the data analyst/scientist/statistician, you need to be aware of the limitations of your approach given the data you are provided. If you do not have sufficient data, you should not even run the analysis - if you do so, you are more likely to accept the results if they are what you "expect" and discard them otherwise.

This is dangerous.

A big part of data analysis is knowing your data and knowing their limitations. If you are given data that is insufficient to make the conclusions you are asked to make, you must say so and refuse to do the analysis. Especially in the ridiculous case you used as an example, where you are expected to make generalizations about groups from two authors. This has nothing to do with luck.

You would never conclude in a scientific setting that Group A is taller than Group B based on n=1 in each group. Don't let yourself get into a trap where you attempt to make the same level of conclusion in another context.

I think you already know most of this, because you talk about stopping at the quality control stage, but if this is how all of your projects are ending then you are putting in way too much time towards a project without having access to the data that shows you the project is feasible. As soon as you are given data that is unsuitable, you need to tell your collaborators that it is not sufficient, explain why, and move on. This step should take 15 minutes if the data are truly as poor as you are describing.

Answer 3

In science we investigate the unknown. As such, it is impossible to guarantee that your project will produce positive results. If you know that your hypothesis is true beforehand, there would be no point in doing the project. However, if all your projects are turning out bad, that also sounds unusual. The best place to look would be peers and colleagues who work with similar things - is it really the case that they are just luckier than you? Or maybe they are doing something differently?

Almost any scientific project will have strengths and weaknesses. It is not necessarily your job to provide an exhaustive account of the weaknesses. Starting with reviewers and even before, there will be no shortage of detractors who will point these out. Selling the strengths, on the other hand, is something only you can do. If there are 99 features that have no statistical significance, it is not productive to get hung up on those. Obviously don't deny them when presenting your results. But the 100th feature that does have significance is the most interesting and bears mention, in addition to features whose insignificance by itself is striking. From there, more significant features can be uncovered.

As you get experienced with analysis you should cultivate a feel for good projects and bad ones. Poorly conceived experiments, lack of controls, experimenters known for carelessness, crazy hypotheses that have no foundation in literature, are all examples of giveaways for a project to stay away from. If you are backed into a corner and all of your prospective projects have crap data, then you can look outside your immediate collaborators. Successful analyses get made all the time so surely there is data out there that is not useless. As you mention, reviews are a good way to at least publish something, they can also attract new collaborators and help you understand the field better and get more capable at detecting bad projects. You can also try to re-analyze data from other researchers or papers.

Another option is to improve matters with your collaborators. Even though a negative result is not helpful for publishing, it is still useful information. It keeps them from wasting time on a red herring. If the data you get is bad after all, you should try to show this convincingly ASAP, so you can quickly get back to your collaborators and start coming up with a solution. In fact, you can use your experience of previous failures to guide them on study design and point out errors they might be making that gave you problems before. If too little is known about what it takes to produce good data, then projects should be small and quick, so you can iterate many times as you find the correct parameters. Big projects should be avoided until you're confident you've taken care of all the basic mistakes.

Answer 4

This is a common issue and it's why I recommend to get more into the weeds and to be more choosey about who you work with. (I basically always took leadership of any collaboration...then again this is easier as the synthesist...people like you are used to being support...then again, I checked their work and sometimes found issues...none of them ever bothered to ask me questions or learn/criticize my methods!)

Similar issues happen when people ask for PDE mathematical models and analysis, but the engineering assumptions are incorrect. You actually add way more intellectual value by "asking five times" and checking input quality, assumptions, etc. than by just cranking the stats machine or the diffyQ solver. Ideally you should try to be involved even with the study design.

One serious area you might look into is US oil/gas. There's a lot of interest in optimization, neural nets, big data, etc. Also, they have a lot of money. (Even when they say they don't, they do. Are used to paying a lot for services, travel, tools, etc.) The data is not always perfect, but they have experience with making do and dealing with missing points also. Of course you need to become more involved yourself in scrubbing, inspecting, correcting, etc. the inputs. But I don't think they will be put off by a questioning approach, only if you tell them to reshoot the seismic or get in a time machine and drill better test verticals in 1950. But I suspect your tools can still often add value even with imperfect data AS LONG AS the imperfections are known before the analysis starts.

P.s. Even the questions on SE often suffer from this. People ask for help with outcome X bounded by conditions 1, 2, 3. But they would really be better served by questioning of what their real output aim should be and of the restrictions.