Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A recent question in Information and Security Stack Exchange,

How to know whether a textfile has been edited or tampered with

raises a very good point: if one wants to archive experimental data in an open, text-based format, in the long term, it is desirable to have tools to prevent the tampering of the files, or at least detect the tampering if and when it happens.

The answers at infosec are good, but they feel somewhat abstract and hard to implement for a busy PI whose hands are already full trying to run a research lab. It may be, on the other hand, that solutions already exist that fulfil at least some of the requirements, or that they will appear not too long in the future; it's certainly reasonable to suspect that some form of tamper-resistant data or lab-book signature scheme is already in use at least in commercial research organizations.

This question is relatively hard to pose correctly, as there is an inherent vagueness in the requirements, and it is probably better to keep it general. The main problem is, given a set of text-based data files, how to signatures, or similar devices, that can be used in the future to provide guarantees that the data has not been tampered with. Are there any specific solutions that will do this in an accessible way?

share|improve this question

closed as off-topic by jakebeal, David Richerby, vonbrand, scaaahu, D.W. Jan 11 at 4:44

  • This question does not appear to be about academics within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

1  
Do you want to track changes (see: version control) OR do you want to ensure that a file is unaltered (e.g. due to malicious users, connection failures; see: md5 checksum)? – Piotr Migdal Jan 10 at 19:27
    
Also, if you haven't taken it into consideration for the storage, you'll want to look at ways to avoid/detect/correct for data rot. – guifa Jan 10 at 19:30
    
I don't have anything particular in mind - I mostly think it's an appropriate time to have a parallel thread here to the InfoSec one, focused more on usable solutions than abstract signature schemes. I think both tamper-detection and change tracking solutions would be interesting in this context. – E.P. Jan 10 at 19:45
5  
@E.P. What is your question here that is about academia? This seems to me to still belong on infosec. – jakebeal Jan 10 at 19:47
5  
I'm voting to close this question as off-topic because it is a question about information security, no part of which is specific to academia. – David Richerby Jan 10 at 22:01

In addition to good recommendations on the InfoSec SE site, some of which are not that difficult to implement (for example, digital signing), I would suggest another rather straightforward solution (unless your data is extremely large or extremely sensitive): simply use one of Git repositories, such as GitHub or GitLab, or research data repositories, such as Zenodo or figshare.

Git and Git-based services now support large files, plus you will have an added benefit of versioning for data files that you can match with your research workflows to enable reproducibility.

Check my related answer on some arguments for potential preference for Zenodo vs. figshare.

share|improve this answer
1  
A private Github repository (they are in general very inexpensive) is my method of choice. They 1. Can be funded by your department with most likely no objections to the cost, 2. Allow for versioning of files with timestamps and version comparisons, 3. Offer collaboration features for your colleagues and have access controls if need be, 4. Are hosted so you don't have to worry about backing up data, 5. Are private to only the collaborators and can be made public if you feel like it. However, Github and similar services are don't work well with non-text data (no file diffs can be done). – Chris Cirefice Jan 10 at 19:47
    
@ChrisCirefice: Private repositories is what I implied in my answer (and, BTW, research data repository services offer them as well). As for diffs for non-text (binary) data files via Git services, your statement is not exactly true: 1) you can use Git attributes (see "Diffing Binary Files" on this page); 2) for image files, there are various approaches, such as GitHub's built-in one or this one. – Aleksandr Blekh Jan 10 at 20:23
2  
The OP explicitly asks about text files, so Github should be good. Plus: I wouldn't trust any file format other than TXT and CSV to be legible in the long run. File formats change. Worst of all are proprietary file formats where you may not even have the necessary license in five years. Bottom line: store everything possible as TXT or CSV. – Stephan Kolassa Jan 10 at 20:24
    
Unfortunately, using GitHub isn't really a long-lasting solution. It only lasts as long as Github's servers do, and depending on the success of a corporation isn't exactly future-proof. – Roger Fan Jan 10 at 20:34
2  
@RogerFan: Your point is valid (in the context of long-term preservation). However, there are, at least, three approaches that alleviate the problem: 1) instead of hosted cloud versions, use of on-premise versions, i.e. GitHub Enterprise or GitLab EE; 2) deploying archiving solutions to several solid cloud infrastructures, i.e. Amazon Web Services, Microsoft Azure and Google Compute Engine, and keep data in sync (chances of all three companies disappear are pretty slim); 3) use Zenodo, which is built around CERN infrastructure and backed by EU, which guarantees the long-term data preservation. – Aleksandr Blekh Jan 10 at 20:55

A common tools for checking data integrity is to use MD5 checksum. If you have unix-based system, you can do it form command line:

$ md5 some_file.csv

or (md5sum, depending on your system).

It gives some hexadecimal number (like dc50353b4a1e5d99cb49b65e33b18916) which will (almost certainly) change with any changes to the file.

share|improve this answer
1  
The only problem with this is that you need to maintain a reliable and trustworthy way of disseminating the correct hash to end users. In this short term this is pretty easy, but if you want to archive data for potentially years or decades it can be difficult. – Roger Fan Jan 10 at 19:43
    
@RogerFan Short term: just put md5 on your website. Long term: same, but in archive in which you are storing your data (and which archive is good is another question). – Piotr Migdal Jan 10 at 19:56
6  
@PiotrMigdal: Storing md5 hashes where you store data files is akin storing keys in the open in the vicinity of locks they protect. Bad people can simply tamper data files and then regenerate corresponding hashes and replace the original ones with them, all in the same archive. Unless you track filesystem changes, you will have no idea that the data files have been tampered. – Aleksandr Blekh Jan 10 at 20:29
1  
@AleksandrBlekh No, it is not the best analogy. Unlike keys, you want to have as much people having a copy of it (which, in short, is the only way to ensure it is uncorrupt). And if you put your md5 hash in article (publication) content, or some text, it is much more likely to be cross-archived that a (potentially big) file. What is ironic, git does not protect against malicious use (one force-push and you are done). – Piotr Migdal Jan 10 at 21:24
    
@PiotrMigdal: I agree - it is not the best analogy. However, I was just trying to make a point, which, by the way, assumed a centralized archival solution (perhaps, I misunderstood you, as I thought that was your assumption). Anyway, it seems to me that the solution you're talking about is distributed and, thus, IMHO might be implemented, using blockchain technology. As for potential malicious use of git via force-push, I don't understand how it could be malicious, if all repository operations are logged. – Aleksandr Blekh Jan 10 at 22:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.