18

Currently, on all of my projects, I do my data analysis, create my figures, put them into a word document, and then I start writing. I say things like "We saw a 35% reduction in the effectiveness of..."

100% of the time, these numbers change. We reach the discussion section and a co-author requests we change the analysis this or that way, or finds an error in my thinking.

Then, I need to go back and re-do the analysis. Then, I'll need to find every point of data that was potentially affected and manually change it in a word document. Occasionally, this back-and-forth leads to the introduction of errors.

I would love a writing platform that would allow me to integrate my data into the writing process. Instead of writing

"We saw a 35% reduction in ..."

I would say:

"We saw a <% print(reduction.round()) %>% reduction in.."

Of course I could do this from scratch on my own computer, but I then lose the ability to collaborate.

I'm wondering if anyone has had this problem, and how they have solved it?

8

While other answers have given some very good suggestions, I wish to focus on the part "if anyone has had this problem, and how they have solved it?" of the question.

I use Sweave and can only speak for this particular method. My general thoughts are that:

  1. Yes, it's awesome.
  2. However, the time to make the two sets of code to work may not necessarily be shorter or less miserable than revising the statistics and tables by hand. It has some learning curve. So, I'd suggest considering using this method if you have i) some documents that need to be repeatedly created or the data are repeatedly being appended, like periodic reports, or ii) some analysis that involves a large amount of repetitions.
  3. The benefit really shines for tables and graphs. Yet I found that embedded text can be troublesome. For instance, weird sentence like "the mean energy intake increased by -1357 kcal at the end of the study."
  4. As an extension of the above, sometimes the restructuring of the analysis can be so drastic that the codes will need to be revised extensively. And you'll have two sets of code to revise and two sets of bug to catch.
  5. In my own circle of colleagues, it's hard enough to have them keep the statistical syntax in a standardized format. I will not even ask if they use LaTeX, not to even mention Sweave.

Having said that, it is indeed very satisfying to see a 100-page PDF analysis report being revised with one click. I'd suggest at least find a suitable environment to try once. By the way, Sweave can also work with Stata and SAS (statweave), quite versatile.


Now, back to the root cause. I'd like to share with you how I minimize this Sisyphean situation.

  1. Remember, if you do no take charge, coworkers will take charge for you. Some statements to express firm decisions about leaving and entering a certain stage in the analysis process can be forceful and yield productive results. This is also true if you are just a student and they are your supervisors. Some reasonable assertiveness goes a long way.
  2. Put all the data set details, variables, research questions, proposed analyses, and some reasonable amount of "plan B's" on what I call a DMAP (Data management and analysis plan.) Pay particular attentions to: i) how missing values will be handled, ii) how outliers are defined in the key variables of interest, and iii) recoding scheme if any categorization is to be done. Gather input from all of them. Once finalized, carry out the analysis.
  3. In the next meeting, share analysis report (but NOT write up). Prepare a descriptive statistics package. And then according to the research questions, lay out the main findings in the same sequence. After each summary output, state 1-3 main "talking points" that will be the foundation (or topic sentences) of the Discussion. Show only necessary output and make sure to make them reader-friendly. Highlight or bold the parts that you want them to focus on. Have the group contribute their thoughts on revision or sub-analysis. Revise the DMAP. Have the previous DMAPs handy to avoid the "you said, I said" situation.
  4. Repeat steps 2 and 3 until no more input was given. Be very clear that "you are going to finalize this analysis and start writing the Discussion." Are there anyone not replying your e-mail and can potentially disrupt this finalization? Deal with them individually before moving on.
  5. Go on to craft the Discussion based on the talking points that have been previously agreed upon.
  6. Along the process, keep clear documentation. Keep your syntax files and analysis report files clear and dated. Include section numbers corresponding to the research question, page number, and line number. Date and sign (provide name and e-mail) all your reports and syntax files.

The main point is: do not write the Results and Discussion and distribute them before the analysis is finalized. You may draft them in private, but never circulate them while the analysis is still actively being evaluated/revised. Doing so provides too many distractions to the group, and it's just going to end up with a hot mess.


In my own experience 75% or more of the so-called sub-analyses are what I call "brain farts." They are a healthy sign that the brain is working, but not pleasant if happening too frequently. Most of them are "what if's" and they can be out of control especially if the results do not go with how they want the world to work.

Yet, 1 out of 8-10 times the suggestions can be good. I usually will take the pain to revise the analysis plan and restart the process. Leave the writing, and come back to deal with it with the new analysis is finalized.

Finally, some catch phrases.

  • "That is a great suggestion, however it's seriously deviated from our original research questions. For the sake of being succinct, I'd write this idea down and we can pursue it in another setting."

  • "Sub-group analysis? Yes, but be prepared that it's going to be underpowered and please don't keep you hope too high."

  • "Sub-group analysis? But the interaction terms are not even significant and I can tell you to rest assure that the two groups will not show any difference."

  • "Another parameter? Another scenario? Sure, let's get this done with, once and for all. Let me know all possible parameters you want to try now. I will just loop through them."

  • "No, it's not related to our hypothesis."

  • "Would you like to follow up with that suggestion? I can send you the codes."

10

Take a look at R Markdown. It allows to generate files (Markdown, HTML, LaTeX, PDF or even some interactive Shiny slides) based on text, LaTeX formulae and code (not only in R - you can use other languages as well!). For a smooth start you can try using a real-time editor editR.

enter image description here

Alternatively, you can use IPython Notebook, it is easy to share, but harder to collaborate on or convert into a nice LaTeX.

10

One possible workflow is using R for producing the results, LaTeX for writing the report, and Sweave to integrate both. With either TexStudio or LyX (or any text editor that supports track changes) as writing environments and Dropbox, you can set up some sort of "collaboration".

6

First of all, let's set the terminology straight. The approach that you're planning to use and seeking applications and workflows for is usually referred to as reproducible research, which, in turn, is based on the literate programming paradigm (introduced by Professor Donald Knuth).

The previous answers are nice and to the point, however, they cover only a limited range of tools. In particular, if you're not interested in having fine control of generated data-driven document in LaTeX format, it might be more feasible to use much simpler RMarkdown (or other Markdown variants) instead of Sweave or knitr (for converting generated documents between different formats, pandoc is very helpful and is pretty much the golden standard for such tasks). Also, there exist more comprehensive (but, not necessarily, more convenient) software that attempts to help in automating and managing the whole process of producing data-based written artifacts (reports, manuscripts, etc.). To learn more about such tools as well as other reproducible research aspects and related tools, please see my relevant answer on Data Science SE site.

  • Pretty sure knitr can use markdown as well as LaTeX. – Flyto Jun 1 '15 at 4:59
  • @SimonW: My point is that, while knitr can handle both source types, it doesn't make sense for someone, who doesn't want/need the flexibility and fine level of control of LaTeX, to use knitr (an extra layer), when RMarkdown is enough. – Aleksandr Blekh Jun 1 '15 at 5:55
  • @AleksandrBlekh RMarkdown uses knitr – ikashnitsky Aug 24 '16 at 20:41
  • @Ilya: I know, but that's not the point. It is the fact that RMarkdown provides more limited functionality (control of fine details of a document) than a combination of knitr and its LaTeX directives. Hope this clarifies things. – Aleksandr Blekh Aug 25 '16 at 2:37
  • @AleksandrBlekh sure. Now I re-read your previous comment, and the context became clear. True, with HTML5 most of the time you don't really need LaTeX – ikashnitsky Aug 25 '16 at 8:56
1

Almost everyone who has written a quantitative paper has been confronted with the problem of reading an old draft containing results or figures that need to be revisited or reproduced (as a result of peer-review, say) but which lack any information about the circumstances of their creation.

Kieran Healy describes a workflow that uses R, Sweave and Emacs org-mode or Knitr to tackle this problem. Dropbox or github can be used to track versions and collaborate with co-authors.

1

My understanding is that IPython and Mathematica can be used to prepare interactive 'notebooks' that can be dynamically updated based on arbitrary computation.

1

RStudio (I highly recommend to follow their blog) does amazing job redefining the R workflow with a particular emphasis on reproducibility (see rmarkdown).
Quite recently they added notebooks, much like those in Jupyter (formerly IPython). The newest features are always available in the preview release.

0

This is an old question, and the answer largely depends on the tools you are using for analysis and writing, but I recently came across a new program to satisfy the need for reproducible research. In the past I have used RStudio and RMarkdown files for analysis, but I'd still end up copying and pasting numbers into Microsoft Word files - for those using LaTeX with collaborators this might not have been as much of an issue (by using Sweave or knitr), but I am predominantly writing with others in Word.

StatTag is a new Microsoft Word plugin that enables you to connect statistical analysis files with Word documents so that the analysis can stay up-to-date. StatTag works with Stata, SAS, or R (or Rmarkdown) analysis files. I have found the introduction and documentation pretty clear. There is also an overview and demonstration from one of the creators at the recent useR conference here. I'm hoping to use it in future projects to lead to more reproducible research.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.