56

I am about to finish my PhD in bioinformatics. I am trying not to give out too many details, so I won't be very specific.

In the field I am working in, there are two extremely important databases being used. During my MSc internship, I chose one of the two databases, decided to stick with it and never looked back. However, as my PhD approaches its term, I find more and more errors in this database.

I only contacted the database team once to tell them I thought I had spotted an error and to ask them whether my reasoning was correct. They took more than 2 weeks to reply and only said they had fixed the error without providing other explanations. Intrigued, I looked more closely at the data and started "reverse engineering" the database: I don't know how everything is internally articulated but I start getting a pretty good picture of how they do it. It's a very old and big database that keeps getting bigger and richer. The scientists behind the database are constantly trying out many new things to enrich the data. They keep some changes and discard others.

Now here's the problem: these attempts at enriching the data are badly integrated. The database is never checked for internal consistency. It is highly inconsistent, and I can prove it with a systematic approach. I can point out actual problems and potential problems, with suggestions to discard dangling references.

They also license their data. Some of it is freely available and some of it can be obtained with an academic or a commercial subscription. If they sell their data they might as well sell clean and coherent data!...

Many database maintenance operations seem to be handled manually and with no regard to existing data. Their enrichment methods seem to be based on in-house scripts that the team praises in different publications and that can be used in web forms, but for which the source code is never available. The whole database seems to use a very old and messy design and I have the feeling they never had engineers in their team. The team itself and their research... well, they are not really transparent. The website only has a feedback form. One can never know what problems have already been reported and fixed. There is no changelog. They can change what they please, when they please and in any way they please.

My PhD adviser encouraged me to pursue this investigation and warned me not to contact the database team any more because we will need to figure out what to do. According to my adviser, even though I'm right, this investigation alone does not warrant a publication. In any case, it will at least make a chapter in my PhD thesis.

What I want to obtain is to get these problems fixed. What I would like to obtain, but I don't think it's likely to happen because of the aforementioned lack of transparency, is to get a paper co-published with the authors behind the database, or at least a collaboration for my lab.

Side note: my PhD research is utterly uninteresting and advances a boring method of doing something nobody wants to do and that will therefore never be used. If I manage to link my name to updating and correcting this database, then my PhD will have served a purpose.

Do you have any experience and advice on how to best handle this delicate kind of issue?

EDIT: I want to thank everybody for their contribution. I will select the most helpful answer. The answers and comments I received pointed me to the right direction. I want to make the issues known once I finish writing up my tool on consistency checking. How exactly I will go about making the errors known is something I still need to decide on. A few final notes:

  • As one of the comments pointed out, it is only fair for the bioinformatics community depending on this database that such errors be known. Maybe making them public will lead some research groups to reevaluate the degree of this dependence.
  • I did not mean to start a stir on computer science vs engineering.
  • My PhD is valid and defend-able as it is; it's just not innovative, but that is another story. Pointing out inconsistencies in a public database is not what my PhD is about and will not be used to "save" my PhD as it doesn't need to be saved. :-)
Improve this question
2
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – aeismail
    Jan 14, 2018 at 22:25
  • @user3638629, Re " If they sell their data they might as well sell clean and coherent data"; Hmm, often once you have the word "sell", to them it's all about the "maximize revenue, minimize cost" chant.
    – Pacerier
    Mar 2, 2018 at 21:13

4 Answers 4

Reset to default
50

There are many critical scientific resources out there that have massive known flaws, but are still useful because the flaws don't prevent people from getting high value from the resources. GenBank, for example, is the predominant source of genetic information in the world and is also known to have many mislabelled sequences.

From what you have written, it is not clear whether something like this is also the case with the resource that you are dealing with. The course I would recommend you take depends on 1) the degree to which the flaws are known and 2) their degree of impact on scientific conclusions derived from the database. The key cases that I see are:

  • The flaws are well-known and researchers are able to work around them: Here, there's nothing you really have to do, and the maintainers are unlikely to be particularly responsive to your complaints, since their system is "good enough" and they likely have other priorities.

  • The flaws are well-known and difficult to work around: This seems the least likely case, as why would people use this database and not the alternative that you mention? If it is so, however, you should probably just finish your thesis and move on: a paper on the flaws isn't interesting if they're already well-known, and while you should report your problems to the maintainers, you'll just be one more instance of the issues they already know about.

  • The flaws are not well-known and likely to cause serious problems in most research: In this case, a publication about the flaws is likely to be of interest and worth doing. It might or might not cause the database to be fixed, but it is likely to be important to alert researchers using the database to the problems in their work, creating pressure for the database to be fixed or people to migrate elsewhere.

  • The flaws are not well-known, but not likely to have a serious impact: This is likely to be the case if you are using the database in a very different way than most others, such that your research is more strongly impacted. In this case, talking about it in your thesis seems sufficient. You should document the problems you had and the flaws you discovered, but you are unlikely to get them corrected because you are not their target market.

Notice that in all of these cases, I assume that the database is unlikely to get fixed. That is because the persistence of the problems over time and the non-professional curation that you report indicate an organization that is probably missing either the resources or the incentives to make the fixes you would like, even in collaboration---although you might turn out to be pleasantly surprised.

Improve this answer
1
  • 3
    Very insightful, thanks for this classification. To my knowledge and surprise, there aren't any reports for the problems I identified. So I would say the flaws are not well known. As to impact... I need to reflect on this. The database lends itself to both purely visual and programmatic exploitation. Visual users won't see any problems. Programmatic ones will.
    – user3638629
    Jan 8, 2018 at 12:35
7

You should contact the person above the database team.

The database that you describe seems to be huge and complicated. As such any modification on it can be relatively big and complicated.

I've already worked in a place with an atrocious database. The relying architecture was simply badly designed. Everyone was aware of it, however we decided to leave it that way.

Why? because improving it meant recreating and redesigning the whole thing and then migrating all of the data from the old one to the new one. We estimated that it would take us months, if not years, of work to end up with a nice database.

At the end of the day, it was decided it was both less expensive and less of a hassle to correct by hand all of the errors created by the database than to improve the underlying system.

Which is to say, that there is a possibility that they are aware of a certain number of these problems but choose not to solve them and not to inform the users as it could leave a bad image. Imagine them answering you :

"We know our database has a lot of problems, but we're fine with it. Too bad for your errors."

Improve this answer
6
  • Interesting. I can understand the decision to keep an old and bad design if the effort required to rebuild everything is deemed too important. Even if it involves correcting problems manually, as you say. What strikes me, however, is the team in my example not learning from reported errors and not attempting to run checkups to make sure the data is left in a consistent state once modifications are operated.
    – user3638629
    Jan 8, 2018 at 12:10
  • 6
    I’m not sure your advice applies in our field. “The person above the database team” is likely an institute director since the relevant PI will be involved in the database’s creation (if not directly then by association: the database likely doesn’t have a “database team” per se; instead, it will be managed by the researchers themselves), and their purpose is purely administrative.
    – Konrad Rudolph
    Jan 8, 2018 at 13:45
  • 7
    @KonradRudolph And for smaller or more specific databases, the original "person above the team" might not even be in the team anymore and the whole database might be floating in a futureless "we do what we can" state.
    – skymningen
    Jan 8, 2018 at 14:51
  • 2
    @KonradRudolph The "person above the database team" is whoever controls the purse strings. ;) The point this answer is making is that whoever maintains the DB needs to understand that the DB has flaws that lessen its value to end users so that there is some possibility they might consider addressing it a priority. Whether that means allocating funding, instructing someone to spend the time, or whatever details are involved really don't matter; the point is that someone who can allocate resources has to care before something will be done. That applies in all walks of life.
    – jpmc26
    Jan 9, 2018 at 6:06
  • 1
    @jpmc26, thats exactly the idea I meant to get across and probably better explained than what I did.
    – everyone
    Jan 9, 2018 at 8:47
2

It is injudicious to describe your own PhD project as boring. It sounds like you are not able to go beyond what is currently known. Well, it takes all sorts. Maybe you can teach what is known to those who don't know later.

Anyway I see no gain in getting into conflict with external groups.
You need to quantify what impact the database errors have on the results of your work and ideally show they are minimal or that you can detect them when they occur. And that should be the most minimal part of your presentation. Using that issue to fill an empty void isn't going to pass muster.

Improve this answer
0

Aha ok, I read your question; and here my thoughts. You may like it or not but here they are:

Pure Engineering Is not Research: Ok this might be debatable to some, but if you go to your PhD viva and say well "I fixed the database", you will have a hard time explaining your role as a researcher.

Supervisor Issue: If what you said is the truth, it is the supervisor and his/her to keep the integrity and semantical correctness of the DB. Your supervisor must manage this situation and it is not your job to do so.

What if you cannot do your research with corrupted DB: This is where the head of research group should weigh in and suggest perhaps an extended deadline where you can do your research and finish your thesis.

Improve this answer
4
  • 3
    Thanks for your thoughts. Clarification: my supervisor is not related to the DB team and does not maintain the DB. Pure engineering is not research, true, but too often engineering is brushed aside when research is involved as if it were some kind of unwanted wart. But I digress. My PhD contribution is mainly research but it might contain some engineering aspects such as "I fixed the database".
    – user3638629
    Jan 8, 2018 at 12:24
  • 7
    "Pure engineering is not research": I'm not sure about that. Fixing up a database might not be considered a theoretical development in the bioinformatics world, but it's certainly a very relevant piece of information to be conveyed to the field of researchers that use the database. Thus, it might not be considered "research" in one field, but I would think it should be considered information that should be distributed to the field of researchers that use that database.
    – Cliff AB
    Jan 8, 2018 at 16:29
  • 3
    "Pure engineering", like restructuring the database so it is properly normalized, probably won't do much for your research career. However, identifying problems with published data--and finding ways to mitigate these--could yield something publishable: @skymningen linked to one such example above.
    – Matt
    Jan 8, 2018 at 22:14
  • As a point of evidence that what could be considered "pure engineering" can be considered research, here's a paper in a respected statistics journal that's basically a bug report on Excel.
    – Cliff AB
    Jan 10, 2018 at 4:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .