5

I'm in the throes of writing my first substantial research paper in the field of Information Science. The topic looks promising and my advisor seems pleased with the work. Because of the nature of the work, there are a lot of intermediate steps involved in obtaining the raw data, preprocessing the data, then running the experiment proper and recording the results. These intermediate steps involved a lot of programming on my part, building one-shot tools to solve problems along the way to conducting the experiments.

I understand that reproducibility is a huge issue in computer science and I wish to lessen the burden upon my future self and upon my lab mates if/when the time comes to follow up on this trail of experiments. Certain software has to be installed, path variables have to be configured, firewall settings have to be set up to allow transference of data - and similar technical problems that wouldn't be appropriate to include in the resultant research paper proper.

How do I record all of these technical and procedural steps such that a future interested party (be it myself or one of my lab mates, or perhaps even an outsider to the lab who has an academic interest in the resultant paper and software) can reproduce the experiment? Do I leave myself open to being "scooped" if I set up a website with an academic blog or wiki in it?

3

These are excellent questions! You are especially wise to recognize that your future self will have unfortunately hazy memories of what parameters you chose and why. (I've also been mulling this process myself, and was considering asking a similar question!) Your question is definitely related to prior questions here, but I think those don't necessarily cover collecting data, and some of the answers haven't aged as gracefully as others because they deal with "try this kind of software."

To the extent that your advisor and lab already have a workflow set up, try to work this into it. Is there a shared server for documents, or an application like Dropbox or Box? Do you have a shared Zotero group for bibliography and files? Do you use Git or other version control software? (Version control may be less relevant to you if you're the only one working on the code, whereas it will be critical if multiple people fiddle with it.)

Keeping notes about results

Many scientists are trained to use research notebooks, and there have been interesting discussions about those; aeismail suggested Colin Purrington's tips on maintaining a lab notebook. From looking at an excellent question across tools and a question about Python in particular (though it can be extended), Jupyter is the next tool I will look into for myself, and it may be what you're looking for.

For my own dissertation, for tables and charts I created in Stata, I had a clear file naming scheme about where I outputted and saved them. It was easy (conditional on already knowing LaTeX) to make a LaTeX doc that pointed to those files and could update with the newest version of the results.

File naming and organizing hierarchy

TCSGrad pointed to Jason Eisner's advice on file organization. In particular, I like his advice there of putting in files or file names or folder names the tags: to-do, how-to, logbook, acknowledge, and send-to. I also highly recommend having, in your project folder, a folder called "Raw Data". This is the original data, as downloaded or found or whatever. In "Raw Data", have a readme doc, whether it is a text file or spreadsheet or whatever, that says where each bit of Raw Data came from, when, under what conditions, etc.

To capture the software setup and system parameters, perhaps have a similar Setup folder (with the executable files for software you install, and a similar readme). (Docker is one tool to capture and easily replicate the correct computing environment. A group like Software Carpentry might be able to provide guidance.)

A separate "Data" folder can have your cleaned versions of the raw data, and folders for "Code" and "Output" should be similarly separate. (Datasets you create should still go into Data, probably.) You might even find it useful to have separate folders for (nicely formatted) "Tables" and "Figures".

Whenever you submit something or have a major draft, try to save a copy of all the things you're relying on, as-is.

Making the notes available?

This prior question asks about the (dis)advantages of an open lab notebook, and there was some good discussion.

|improve this answer|||||
1

Yes, you definitely DO leave yourself open to being 'scooped' and plagiarized if you make a blog or wiki site..

Whatever you do and however you take notes or make records FOR YOURSELF and your adviser, DO NOT PUT THEM ONLINE. you cn make hand written notes or you can make word files or whatever and Save them, possibly burn those records or onto CDs or whatever has replaced Zip files in the year 20 years. part of my data is 8mm tape cassettes from Sun work station, but i know i probably was not given all the 8mm tapes... The text of my thesis or on ZIP files. i didn't have my own PC until 2004. I did have the hardware for the Zip files on that PC, but now i am on mey second PC. which i bought over the counter in

30 years from now. however you record and save your data and research notes or notebooks NOW, Is the hardware you use NOW STILL GOING TO BE AVAILABLE and usable IN the future.? Hardware DOES become outdated. You've probably never used 3.5 inch floppy discs. I have.

My thesis was interdisciplinary i had mainly geologists and my thesis committee , but i also had one geographer, who happened to be the GIS expert i was WELL aware that my "audience" would NOT be familiar with all the image processing, remote sensing, and geographical cartography and computer and GIS terms and I made very sure i explained what something like "normalized vegetation index" meant and which Bands of Landsat satellites images were used t create an NDVI image were used and how image processing worked statistically with the data in a satellite image. Data is MEANINGLESS until it is organized analyze and INTERPRETED to become information. oneof of my thesis committee members, a geologist, said my thesis was well written and not too full of jargon.

HOWEVER you record your data and how you organize it, DO NOT put it online outside of a very limited sub-network, and NOT on a blog or wiki page that becomes publicly accessible.. You DO NOT want to put ANYTHING online because your copyright right WILL be violated.

When you get to the the literature research par of writing your thesis or dissertation, or ANY paper for publication, i suggest you spend some time with one or university reference librarians. Ask them how library digital databases work and what citation indices are, and how you can access them.. Publish or perish may be trite, but it is VERY true and yes, IF someone CAN "scoop' you They WILL I've seen one person a Ph.D. candidate, who had to stay in school another 2 or 3 semesters because some else got part of their dissertation published FIRST. and THAT was in the 1980s., BEFORE the internet took off in the 1990s and early
2000s.

|improve this answer|||||
-1

How do I record all of these technical and procedural steps -- including installing certain software, setting path variables, and firewall configuration -- such that a future interested party...can reproduce the experiment?

You can write a script that automates the necessary steps and reproduces the results. (Writing that script after you've produced your results will be difficult. Try writing the script as you go along.)

(Anything more sophisticated is surely complimentary.)

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.