Take the 2-minute tour ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

I am trying to compare PhD training programs of different universities, institutes and departments in my field of interest. One of the comparisons I made is where did the previous students go after they finished their PhDs; how many papers did they write during their PhDs; and how much are they cited 1-2 years after completing their PhDs.

Now, comparing the citations counts is difficult. They vary quite a bit. How many times should a successful young scientist be cited 1-2 years after completing his or her PhD? How about an average one? What is the minimum number to be able to continue in academia? I understand that exact numbers are difficult to provide but I would guess you senior scientists can provide good estimates.

share|improve this question
4  
Such a short time after publication the number of citations will mostly represent how well a paper was advertised rather that its quality. –  Bitwise Dec 7 '13 at 16:55
    
Try and check sucessful scientists: In some fields (math/TCS), only like 10-20% of their articles really get significantly cited, most have up to ~5 citations. If you really want to evaluate their publication quality, look for eaxmple at cooperation scheme: Do they publish only student+supervisor? Do they publish in larger teams across the dept/from other departments/universities/countries? If they do, look at the profiles of their coauthors: Good scientists wouldn't publish with students from bad departments. –  yo' Dec 10 '13 at 21:52

4 Answers 4

up vote 6 down vote accepted

I believe that scientometric indicators of scientific productivity and impact are at best, limited if the overall objective is to compare different college, departments, universities, institutes, centers (or any other academic organization) in one discipline. This is coming from someone who has had rigorous training in bibliometrics/informetrics/scientometrics.

This is because there are massive variations in citation patterns within a particular field. This is especially pertinent if the field is rather interdisciplinary. Note though that in any given field, there is a general level of agreement among scholars in that field which departments are "better" than some other ones.

For instance, in Human-Computer Interaction (HCI), critical theorists and ethnographers get, on an average lower numbers of citations per article published than say, ubiquitous computing folks.

However, purely on an exploratory basis, there could be several ways of looking at departments:

  1. Where are graduate students from a particular department publishing in? Are they consistently publishing in top tier venues? e.g. In core HCI, I would look at CHI/CSCW conferences. When considering, the subfield of usable privacy, I would, in addition, look at SOUPS/IEEE Privacy & Security.

  2. Among these publications, are they receiving best paper awards or best paper nominations?

  3. Where are students from these departments being placed on a consistent basis? Are they being placed in industry or academia? Where, specifically? Google or a no-name startup? Are they getting tenure track positions or primarily post doctoral fellowships? Which other departments?

  4. Are some of these students recipients of prestigious fellowships during their doctoral studies? e.g. NSF Graduate Research Fellowship or Facebook Fellowship? How many?

  5. Bibliometric Indicators:

    • Average number of peer reviewed articles? (full journal and conference papers)
    • Average number of citations per article?
    • Co-authorship network analysis (which gives a nice idea of collaboration networks)
    • Average number of student grants received or otherwise?

You can go on and on and on.

The idea is that, there is always some way to standardize or normalize any given construct. Sometimes, that is an useful approach. But, strict adherence to bibliometric indicators at many times, make you mistake the noise for the actual signal.

Consider the bigger picture and use metrics to buttress your concerns. That might be a more holistic approach.

share|improve this answer

Although this approach to measuring is plausible, I fear it is not what you want. Although of course things vary by field, the papercount per se, or even "impact factor", or ... any other easily-measurable thing is not quite what will affect things.

At least in mathematics (my field), a smaller number of very-good papers is much better than a large number of irrelevant papers. One might imagine that this would be measured by "impact factor", but it is absolutely not reliably so. As an extreme case, consider what happens when an important problem is finally and definitively solved: there may be fewer follow-ups, fewer citations, simply because there's little more to say.

And there's time-lag: I have many examples of 10-20 year (and longer) delays before people widely realized the interest of a given paper. Srsly, surely we're not trying so much to be of-the-moment-celebrities, but to make a long-term impact.

One could go on-and-on about the unreliability of appealing-but-fatally-flawed metrics... but the real point is that a less objectifiable judgement is necessary to evaluate the "merits" of a program. Stats on the students is probably not quite what you'd want, even though it might seem to be. E.g., at more elite places, there is vastly greater variance among students... so the "outcomes" are harder to interpret meaningfully... and, for you yourself, what those other kids did or didn't do is operationally irrelevant:

That is, your level-of-performance, that causes (or doesn't) respected experts to commend you to their peers, is what will get you a job, or not. The "published papers" thing is a side-effect.

share|improve this answer
1  
This was a good answer too! –  selecting_phd_program Dec 7 '13 at 22:33

You should decide on your PhD institution by the quality of the tenured faculty, not their students. Great students make great co-workers, but it's the quality of your supervisor that can make (or break!) you. If you have good relationships with faculty at your current institution, you should ask them what, in their opinion, are the best departments to apply to.

You should review the publications of faculty members you would like to be supervised by, to see if their research interests you. The papers that are first-authored by the faculty member (they are still active in research and don't spend all their time supervising and teaching, right?) will be particularly useful in this regard.

share|improve this answer
2  
In most fields, the faculty in all of the top 200 schools would be coming from the top 20-30 programs, and thus will all be highly qualified to teach and supervise. However, it is the quality of the student body, and the peer effects associated with it, that make programs superior. MIT can afford to select the best of the creme; UMass Boston will have to do with the grad applicant leftovers that were not placed to better places. And the faculty, that may only be a notch worse at the latter, would have to offer simplified courses because their students won't be able to digest the "real" ones. –  StasK Dec 7 '13 at 0:45
1  
@StasK - in that case, surely the best universities would be able to select the best faculty, as well as the best students? Think carefully about why the university continues to be at the top - great faculty. Of course, excellent researchers are not only found in the top-20 universities, which is why I try to emphasize the importance of your supervisor while making my answer independent of country and discipline. Not all PhD programs have a significant coursework component. –  Moriarty Dec 7 '13 at 1:23
8  
I disagree. The success of past graduates of a program is a far better indicator of the likely success of future graduates than the success of current faculty. Lots of incredibly strong, active, funding-rich faculty are terrible advisors. –  JeffE Dec 7 '13 at 2:09
1  
+1 to all three of you, because all your comments hold some validity. :) I somehow think that StasK and Moriarty have overemphasized their points though... (I complete agree with JeffE having witnessed what he describes happening to friends.) –  usεr11852 Dec 7 '13 at 2:33
    
@JeffE I concur. However, the value of PhD advisor goes beyond teaching purposes. If one intends to pursue an academic position, a top research group would likely be a better choice even if that scientist is not a perfect advisor. By being in a successful research group, a student can work at the very frontier of the field and can potentially make a great impact; a successful scientist often have significant influence in the field and many academic connections. These alone don't make their students good scientists, but certainly can be great boost to the career of new scientists. –  Xiaolei Zhu Dec 7 '13 at 5:28

I believe your first two criteria are far more relevant than the third one. Two years after the defence is too early to evaluate scientists based on citations. Moreover, the average number of citations varies a lot from one field (or even subfield) to another.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.