I started my PhD 7 months ago, and as I generate more and more data, and do more and more analysis, my folder structure is getting out of hand. I wanted to ask for best practices and opinions on how to organize my files in order to not lose track of everything and quickly find what I need. I am a geoscientist and have lots of analytical data as well as programming scripts. Along that I also have
README.md for some analysis and snippets of ms word or plain text, if I wwrite something down to remember for a paper possibly.
I've made it a habit not to edit raw data at all and to backup regularly for obvious reasons.
Right now my simplified general structure looks a bit like this:
├── data │ ├── analysis │ │ ├── isotope_temperature_reconstruction │ │ │ ├── report.ipynb │ │ │ └── script523.py │ │ └── light_micro_growth-line-analysis │ │ ├── img1.svg │ │ └── regression.py │ └── raw │ ├── isotopes │ │ ├── run2019_02_19 │ │ └── run2019_02_24 │ ├── light_microscope │ │ ├── sample_xy123123 │ │ └── sample_xy123124 │ └── sem │ ├── sample_xy123123 │ └── sample_xy123124 └── documents └── paper1
I know the system my files are organized in doesn't really matter as long as it is consistent. However I am facing some struggles:
- The data usually varies in "quality" and "ripeness". I have data that resulted from:
- Some trivial test -> won't be used ever again
- Is for calibration -> Doesn't belong to any project, but matters in many cases
- Is directly and only needed for a certain publication
My problems with this are:
- I find myself often linking and copypasting my data all over the place, because it is not where it's currently needed. As a consequence i also make edits only in certain places and not everywhere, and lose track of whats the most recent file. I also lose track of what files are handled programatically and where I edited something "per hand".
- I have a lot of duplicate python/R/whatever scripts that I copypaste wherever needed. I think this is the easier part to resolve by modulating code and putting it into version controlled system wide libraries.
- I sometimes have snippets of word or plain text that contain relevant research insights, but are scattered all over the place, because they are not directly related to a paper / data.
So I am looking for suggestions to address these problems, as well as suggestions for general file and data management, and general organization at a researchers main desktop machine.
The only problem that I feel adequately solved is my literature management because I just use zotero and let it organuize all my papers in a coherent folder structure. (It also makes it easy to search via tags, which would be super cool for data files)