Is it possible to measure/evaluate one's progress or development in quantitative terms?

Question

I often end up frustrated over the pace of my work, or rather the lack of "worthy" results. During my feedback talks with my colleagues and superiors I usually get good feedback but I find it distressing that most of it is qualitative arguments.

I sometimes feel like I have not progressed as much as I would have liked to, but not sure how to assess whether or not I have developed "enough" over time. Which led me to wonder if it is possible at all to measure how well a PhD student is progressing.

The usual measures in the community appears to be:

• number of publications
• which journals the publications appeared in (or rather the impact factor)
• number of hours in the lab (regarding how "hard-working" one is)
• number of credits taken from courses during phd studies

I personally find none of the above to be a good measure. Publications are a fact of research, or the goal rather. But they should not a be measure of how well a PhD candidate is doing in research. I believe the pragmatic demand on "more publications" has essentially lead to overall lower quality and novelty in individual publications. But even without that subjective comment, it should not be a revelation to anyone here that the amount of publications (and especially in which journals they are published) is more dependent on the seniors on the paper rather than the grad student who wrote it.

As for the other two measures I point to, they are just too naive variables to mean much. I mean you can be in the lab for 18h a day, but not learn much new or even worse not even remember those things you have learned. Besides one can also argue whether or not it's actually better for a grad student to be obsessed with number of hours in the lab, or courses taken.

Summary: Is there a good way to measure your progress through-out your studies? How can I evaluate my development as a scientist, in quantitative (and unbiased) terms?

Answer 1

To reduce anything to a quantitative score, you need a valid metric. There are very few valid metrics you can use in research, as you state in your question. The only other half-common metric that you didn't list above is Papers read/annotated. We all know reading is really useful, and you should definitely aim to read X papers a week (where X is some random number that makes sense in your field). That said, the goal of reading is to gather information, and how much information was gained (and retained) is a lot harder to measure.

That said, I have two half-answers:

• Make up your own metric based on hours of productive work. At the end of each day, just write down in an excel sheet or something how many useful hours you worked that day. You can use some service like Rescuetime to help you in this, or the Pomodoro technique, or just simply buy a stopwatch and keep track yourself. At the end, do a simple # useful hours/total hours worked to see how productive you think you're being. That'll probably be more useful than anything else you'll come up with.

• Make task-specific metrics. I tracked my progress on my thesis using a custom shell script I wrote that tracked how much text I added in a given time period and plotted it out. (Yes, I probably spent more time making the shell script than I gained in motivation from using it. Whatever.) I tracked progress on one of my projects by how many datasets I had analyzed. I tracked progress on another project by how much coding I completed each day. These are much more useful than broad, overarching metrics.

Answer 2

posdef, this is something that I've struggled with as well. Instead of knit picking over what the definition of 'is' is, I would like to offer the approach which I have used. Your mileage will vary but I've found this approach to work well for me and it may work for you as well.

First, the crux of the problem for me was that progress is either analog or digital, qualitative or quantitative, right? This is what we are led to believe and I think that in the case of education, it is not true. There are discrete quantitative measures by which you can, and have, gauged your progress. 40 classes to get your Bachelor's degree. If you complete 20 classes, you're half way there. Quantitative progress. If you're half way through and your GPA is 3.5 then you can make a qualitative assessment of your empirical data. 50% complete, doing well with room for improvement. So throughout our undergraduate work there is a pretty consistent set of standards and metrics by which to measure our progress and the quality of our work.

Graduate school, for me at least, has been somewhat different. With a Masters program you've usually have either seven-eight classes and a thesis or ten-twelve classes and some kind of a project. For the first portion of the program you can track as before, but then you get into the core of your research or development and encounter something, which I think you are alluding to, 'the perception of quality'.

• "How many papers did you write?" 10 - High(Quant)
• "How good did people think they were?" 4 - Low(Qual)
• "How many hours did you spend in the lab?" 100 - Low(Quant)
• "How good was your lab output/finding?" 10 - Exceptional(Qual)

I haven't started the doctorate yet, but I can make an educated guess that this only intensifies with candidate work.

What it really boils down to at this point in your academic career, at least in my experience, is 'how well is your work received?' and how to do you track that to evaluation your own progress. What I've done is to take a two pronged approach to each aspect.

On the quantitative side I've set up a simple database with my course work, grades, number of publications, number of lectures, number of citations, everything I can think of to track my progress. You could also do this with a spreadsheet pretty easily. The basic tenet of this approach is 'What do I have to accomplish & what have I accomplished."

On the qualitative side I've asked professors, facilitators, leads, reviewers, even peers in some cases, to evaluate my work against the task objectives. Usually you get something like "you're doing fine" but if you can get more detail, do so. A question that I like to ask is "Would you feel comfortable with me teaching your syllabus?" This seems to get their attention. It's interesting because it puts the qualitative assessment back on to themselves and forces them to think of your mastery of the material in terms of dissemination rather than assimilation. "Do you feel comfortable with me teaching this material/running this lab/managing this team that has your name on it?" Good bad or ugly I write it down and give it a 1, 2, 3. 1=no faith. 2=some faith. 3=complete faith. If, after 6 courses (for sake of argument) you have a qualitative score of 15+ then you know that more than half of your superiors have faith in your mastery of the content that you are consuming or presenting.

To be fair... I'm a bit of a numbers junkie and this may not be the kind of system that works for you but it has worked for me so far.

Best of luck.