Equivalent to agile software development but for research?

Question

In software development, there are many development methodologies with a focus on speed/efficiency: Agile, Scrum, DevOps, Kanban etc.

Are there similar methodologies in scientific research? (something like "Agile Research")

A couple of related ideas that come to mind are: design of experiments (how to vary independent variables for optimum statistical inference and how to sequence experiments) and MLOps (similar to DevOps but in machine learning). While those address some aspects, they aren't a complete methodology for doing research.

Answer 1

As someone who has used, taught, and published things about agile software process, I can give you some ideas. I don't know about tested successful existing practice for applying the ideas to research, however.

First, the practices that agile developers use are mostly not applicable to research, though consistent "pairing" probably is. You really need to go back to the principles from which agile was developed to find appropriate principles and practices. For example, it is unlikely that a research team has a "customer (product owner)" in the sense of an agile software project, unless, perhaps, for product research. So, I'll assume you are interested in basic research, leading to an opening of new ideas and theories. I'll also assume, as in agile software development that you are interested in the work of teams.

The goal, all along was to create high quality software in a flexible, changing, environment using continuous feedback on the product as it developed. There was also a reluctance to guess about the needs and to therefore need to make corrections due to bad guesses, though enable changes due to changing conditions. How do you hit a moving target effectively and (relatively) efficiently?

The principles, often stated, are: We value

• individuals and interactions over processes and tools,

• working software over comprehensive documentation,

• customer collaboration over contract negotiation,

• responding to change over following a plan.

As you can see even these principles need some restatement for research. These were devised by experienced developers to guide the creation of the actual practices.

But note that research, being an attempt to look into the unknown, has some of the same underlying characteristics. It isn't so much that the target moves, but that it is unknown, so our view of it at any moment might change.

So, what would agile research principles be? What would researchers value?

Let me suggest a few, though I can't claim to be either complete or especially near the mark. This requires some group thinking to come to a consensus. It would also require being put to the test.

But, I suggest that we value:

• collaboration over individual work,

• insight over individual results,

• flexibility over established process,

• (questionable) small projects over big,

• peer feedback over holding details back.

And note, importantly, that these aren't binary choices. It isn't that individual work isn't valued, it is just a matter of shifting balance. And, the simple statements above need a lot of expansion and elaboration.

I'm not sure I've got it right yet, and won't on my own, but it might give you a starting framework that you could use to develop such a framework.

Answer 2

No. Not that I am aware of. Here's a couple of things.

Generally speaking, academic research is relatively independent when compared to a product dev/delivery cycle. Agile methodology "pays better" in more complex group workstreams. So the necessity of codifying practices in my PhD program is muted. I'm sure there are larger academic research projects that would definitely benefit from more rigor in the process, I feel that's the exception.

At the risk of being controversial, agile is business first and methodology second. The training costs money; the trainers get the money. Academic groups don't line up to spend that money because it's not required. Additionally, there is less sell pressure from the agile trainers, for whatever reason. Perhaps this is a good business opportunity?