Organizing data and files

Question

Over the last four years I did a bunch of experiments (biotechnology, protein purification). Some of them are more valuable, some of them were just poorly designed (e.g. temperature not controlled, etc.). Some of them turned out to be important one year after I collected the data.

The data is highly heterogeneous (microscopic images, ELISA, SDS-Page, HPLC of various suppliers, settling velocity, filtration tests etc.; formats are doc, tex, xls, txt, csv, tiff,py, ipynb, png,pdf and proprietary formats like leica`s lif format)

Usually, I started a new folder on our network drive for each task (like afm_particles/2015_01_13_particle_1). Over time I gathered about 50 Gb of heterogenous data.

Usually, I do not alter the data in place, but load it into an Ipython notebook and do the calculation and plotting with python. The calculation files are stored in the same folder as my data. I have a file called INFO.txt in every folder with a brief description of the data within the folder, experimental conditions etc.

Usually, I know where I can find my data, but recently I wanted to change a plot for my thesis and spent about 15 minutes searching for the right sub-folder. In the end I searched for the name of the file and found it. A colleague of mine quit his job a year ago, and if I wanted to reproduce his experiments, I would have to invest several hours in order to find the specific file containing the raw data (hopefully with physical units stated somewhere).

We do write quarterly reports and paper-based lab journals (some more thoroughly than others). However, navigating through my own data is challenging. Not to mention someone else's data.

How do you handle heterogeneous data collected throughout a project (3 - 5 years) ?

Have you found a solution satisfying for both, people who have to document it and people who might continue the project ?

It would be great if I can browse the collected data without going through all the subfolders. For example by some form of index with information for each folder like: title, short description, usefulness of experiments, worked out as expected, totally crap, totally different than expected but interesting.

Are there tools capable of such a thing (Linux, Windows) ?

Answer 1

I have been using a very simple system and am very happy with it:

• have one projects directory
• have one private directory
• have one life directory

Inside these directories, I create one new directory per year, for example projects.2015. Inside this directory, each new sub project gets its own folder that starts with the date at which I begun this new project, for example 20150902 rendering performance analysis. Everything related to this project will be placed inside this directory, no matter when it is done.

When your projects get too complex, you can always add a text file describing the content of your files in more detail, and place this file inside the projects.2015 folder. However, do not place anything else in this folder! Only folders starting with a date and maybe one file giving more information.

This system has worked for me for about ten years and I see the following advantages:

• very easy to implement
• does not depend on software that may or may not be available in 5 or 10 years
• works on Mac, windows, Linux, etc
• can be backed up easily, for example by placing your files on your cloud storage folder
• future ready: when I started online backup, for example, did not exist yet and we do not know what will exist in the future; but system will probably be compatible.
• very comprehensive: chances are you will remember what year and maybe what month you did a specific project, so it will be very quick to find your stuff.

Finally, if many projects contain a similar structure, stick to an identical organization of your data. Whatever your structure, sticking to a system will make it easier to retrieve your files later. For example, inside your folder 20150817 afm study 1055nm grating and any other measurement folder, you could have the following sub folders

• 00 notes
• 01 code
• 02 raw data
• 03 processed data
• 04 plots

Personally, I often use numbers because this orders the list of items in my desired way and because in Unix, Mac, etc, I can just type cd 03[tab] and the system auto completes.

Whatever you end up choosing as a system, my recommendation is that you look for something trivial to implement that does not slow you down. The more trivial the logic, the more likely you will stick to it over the years and benefit from the ordering you have.

Have fun

Answer 2

I am going to answer the "I wanted to change a plot for my thesis" part. I write in LaTeX.

I create a symbolic link to the plot file's directory. When you have to share the LaTeX, such as during submission to a journal, you need to collect all the images together. I do this using a shell script that copies the figures into the same directory as the LaTeX. It also helps to have everything in one directory for some version control systems.

Answer 3

I have been thinking about this myself as well. I did a few projects before I started my PhD and learned a lot during those. Now I have several folders which each have specific files in it. I have a folder for References, Data, Experiments, Presentations, Conferences, Manuscripts, Protocols, Information, and a few more. For each experiment I do, I make a new Word-file. This file is saved using the date of the experiment; I often have an Excel file per experiment as well which has the same/similar name and '_setup' in the name. I have one Excel file with each experiment written in it (date and brief description). My 'Data' folder has a subfolder per technique, so I have one for ProteinGels, Chromatography, CD, DLS etc. In there, depending on how many files, I made subfiles per year. All data is saved with the date of the experiment in the file and usually a description of some sort. I have two back-ups of all my stuff that I save at different locations in case my laptop gets stolen. Don't want to do the PhD again ;).

It's still not 100% but I can easily find experiments and the data. I also have a labbook where I write things as I go, and I scan this when it's full and I start a new one. The labbook for me is just daily notes about making solutions etc because it wouldn't work for me to write details of every experiment in there. I do find it hard to keep an overview of everything I've done and this is something that I can improve on.