1

I have separated my question into sub-questions as suggested in the comment by user “padovapadova“ ( see my original post)

I started my PhD 3 weeks ago. I already read a lot of information, research papers, publication and so on regarding my topic, but I still don’t have a complete final idea about my topic. Before applying I was quite sure about my PhD topic idea, but after discussion with other members of research group, my supervisor and attending meetings… I think I have to work on my first idea. You probably had such experiences and can share your advices, tips, hints which were helpful for your first year in PhD

How do you organize your work on your project? Do you save your files on cloud (Dropbox, Mega, OneDrive, iCloud, Google drive…) or just on your PC?

1
  • 1
    If you have been around computers enough, you should know you need to keep things in at least two places, preferably geographically separate.
    – Jon Custer
    Oct 17, 2022 at 14:23

3 Answers 3

2

Here is my foolproof aspiration for organizing my academic data. Take what you will and find what works for you!

Step -1: No Software

A few years ago my university shifted from physical lab books to a fancy digital note-taking software. I thought it was going to save my life! I have always been, and still am, incredibly disorganized. I thought having all my note-taking happen in a simple browser window would turn my research life around.

It did not.

Now, if you find a piece of software that works for you, by all means, use it and never let go! But if you have trouble adopting one, you may be suffering from digital cognitive friction. After all, in theory, you just need a stack of notebooks to stay organized. In theory that's all it should take me. And yet I am not even physically notebook-organized, which means that any successful organizational system must be even simpler than opening up a notebook and writing.

No logins. No menus. No side-by-side panes. We need to go all the way back to basics.

Step 0: Open a new folder

And pat yourself on the back. Organizing a project from the beginning is much better than trying to organize a sprawling disorganized year-old festering mess.

Step 1: Simple folders, simple files, simple filenames

Simple folders mean things are stored by kind. Here's a sample directory structure:

flux-capacitor_1985
|
|--papers
|  |--Einstein_relativity_1915.pdf
|  |--Brown_gigawatt-energy-requirements_1955.pdf
|
|--notes
|  |--DeLoreans-are-cool_1980.md
|  |--possible-teenage-collaborators.md
|
|--invoices
|  |--Libyan-Glowies_1984-06-21.docx
[...]

If you can organize things more deeply into subfolders, good for you -- if you find yourself agonizing between "physics" and "mathematics" for Poincare's papers then don't bother. It is better to have a folder full of thousands of well-named simple files than to have an folder empty of files you couldn't organize because you couldn't pin down a sub-organization system to your liking. Remember that if they are well-named simple files then your computer can search through their contents for you. (This is an adaptation of the YODA principles for data organization.)

Simple files are, as far as possible, plain text. Store written documents as Markdown plain text files, which are both very readable in Notepad and very pretty when formatted. Store tables as column-separated plain text files for easy AWK-ing. Store plots as plain images (and generate them from scripts which have the filenames coded in, so you can search your scripts directory for a plot's filename and have the relevant plot's name come up). Of course, you will have PDFs of journal articles, PPTXs of presentation slides, and binary output from instruments or programs you can't control -- but that should be just about it.

Simple filenames make it easy for either you or your computer to decide what's in a file based on its name. Here is an excellent presentation on how to name files -- read and apply it.

Step 2: Frequently backup

There are all the usual cloud solutions like Google Drive, Dropbox, OneDrive ... But if you have an external storage server, you can use SyncThing, and if you want to backup to a thumbdrive you can use FreeFileSync.

Notice that since you have organized your files using folder structures and filenames, the organization automatically travels with any backup.

Step 3: Version control

Are you not organized? Then here's the next level: turn your project folder into a version-controlled repository. Version control means you'll be able to track and revert changes across everything you do. No accidental deletions, no uncertainty about when a script suddenly stopped giving the "correct" results, and no more worrying that you won't be able to backtrack your steps.

If you can learn Git, you can even host your repository on GitHub! And guess what -- it automatically formats your Markdown files for you!

One Step At A Time

Notice how each step is worthwhile in itself. This way, no matter where you stop, even if you don't get all the way to creating a fancy-as research compendium, you can pat yourself on the back for being more organized than me than you would otherwise have been!

Also notice how the steps synergize with each other. Simple folders and files mean you can automatically apply version control, and mean that a GitHub repo will be as perfectly navigable as your desktop folder. They also make it easy to back up your work.

Also notice that this is a bare framework on which you can add things. You can absolutely run Mendeley or Zotero off your papers subfolder (and thus make sure, by the way, that it can't corrupt anything else). You can experiment with Jupyter Notebooks or the DataLad system or anything else you please -- again, having a simple folder of simple, well-named subfolders and files can only help. You can almost certainly do better! But these suggestions, I think, are a good starting point.

All the best!

1

I routinely work with dozens and hundreds of gigabytes of data and have to perform quite a bit of housekeeping after each experiment session to ensure we have enough backups. If you can offload at least a bit of that to the infrastructure at your lab/university or not handling it in the first place, great!

Overall, my organizational rules would be:

  1. A lab should have a centralized data storage and a backup system. Students come and go, you sometimes need to revisit old data and results. I have found that by-year->by-project folder structure works better than the other way around; it feels easier to navigate (YMMV!).
  2. Raw data should be stored separately from the rest of the artifacts you produce and no one should ever write over it, including in the same folders.
  3. Split work into stages and keep a document describing what goes where alongside the processed data - you do not want to revisit it three months after and spend a week figuring out if you should take files from "preprocessing" or "stage 2a" folder - it may make perfect sense in the process, but you will forget this structure in a heartbeat. Write it down.
  4. You may (and should) have a working copy of your data locally, unless there is a binding agreement like NDA prohibiting it. Better safe than sorry. That said, this is more of a backup solution in case EVERYTHING goes wrong with the existing lab storage, do not spend a ton of time and effort trying to keep it as neat as the "main" storage.
  5. Notes you are taking will be mostly temporary, I find it impossible to keep them loaded into my brain at all times. Go through them periodically: weekly is a good periodicity for meetings and for writing down what seems the most important at the time. If you do not produce an article in a few months, write down your findings in a similar, more detailed fashion to keep things orderly. Keeping one "master document" describing what you know about the current project up-to-date can help.
  6. As others have already pointed out, text files are the most valuable. Thankfully, they are extremely lightweight and between code, .md (which is fantastic indeed!), and text documents you should be using cloud and, possibly, version control for storage.
  7. Keeping your knowledge base healthy is what I struggle with the most personally. Reference managers help a bit, but it seems the bulk of it gets to stay inside your head for you to be able to navigate your field.
0

I'm going to suggest what you want is some minimal kind of thing that takes as little time as possible. Don't become a sys-admin to support your studying. The place you should be trying to cram in as much information is your brain rather than some computer disk someplace.

Get something that works without too much trouble. Make sure you can store binary files so that you can zip stuff (or whatever your favorite file compression software) and shove it in the archive. And make sure you can take it with you after your degree.

Depending on how much data you produce, you could probably just carry around a thumb drive or some such. $20 of thumb drive on an attractive lanyard of some kind, and you can probably hold more text than you can type in your life.

Organizing the files then depends on how many and how big and how often they change. What works for me is, at the end of the day, to grab all my files from my working directory and copy them into a named-for-the-date directory on an archive drive. If I am working intently I may shorten that to every 4 hours or even every 1 hour. Storage is cheaper than my time in most circumstances. A simple named-for-and-saved-every-day file system is usually quite convenient if I need to back up or restore.

I tend to work on the hard drive on my desktop. Then I archive daily to the LAN drive on our office system. It's backed up automatically. Then periodically I will copy the whole thing to something like a thumb drive or a second LAN drive or some such, depending on the project.

Your experience may vary. If you are producing 100 GB files every hour or two then storage may be a challenge.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .