I work in an interdisciplinary field. My input is not generated by myself, but by talented people that I trust and that trust me to analyse their data and generate fascinating insights.
But here I am, once more stuck with a project where the input is bad. There is no use for blame and finding a scape goat, we are in this together. And people learn. But I have been stuck in projects with big promises and bad/insufficient input since the start of my PhD. I moved to another place for a PostDoc now, but this situation seemingly haunts me wherever I go.
There is not much I can achieve when most of my projects stop after the input quality control. But if I want to stay in academia, I desperately need to step up my game in actual output, not just in my ability to troubleshoot, right?
How do I move on from this? What chances do I have if I can just never land a "prestigious project" that results in a valuable publication? Is there a chance to still build up a good scientific reputation without those? Should I try myself at writing a review? Take on projects until one finally works out? (But how long will I succeed in getting another job if they don't?) I could work with published data for a bit, but it is often not comparable between studies and severely lacking metadata.
The question is, do I have to accept that success (as in, basically being able to stay) in academia is to a large degree based on luck and I am not one of the lucky ones or is there anything major I can do? I really love the work I do, I would like to keep going.
So far I have taken on additional projects, tried to do my own 'side projects' on at least roughly usable parts of the data in the hope I will find a better dataset eventually where this could come in handy and kept in touch with collaborators in an effort to troubleshoot and eventually produce better input.
EDIT: To address some questions: I produce analysis pipelines, partially based on my own methods. Without "real data" application it's hard to publish those in my field. Yes, it is "real world data". I do not expect perfect data at all. But I do expect technically correct, usable data. If the input is random/to few features to be statistically relevant there is nothing I can do, though. Imagine trying to do a statistical test on the similarity of blog posts based on word usage written by different groups of people but many "groups" only are represented by two authors, the text is sometimes just one sentence long and quite a few of the posts looking like they are produced by a random letter generator, not having any actual words in them. While I was promised at least 5 authors per group, minimum 5000 words per text and of course the post actually written by the author assigned to it.