19

Yesterday, during some idle time in the office, a colleague of mine has discovered that I am currently ranked at position 20 worldwide by Microsoft Academic Search for my subfield regarding the last 5 year impact. While certainly flattering, this ranking is also quite clearly bogus, as many much more eminent academics are ranked far behind me in this list.

The Occam's razor explanation that there simply is another researcher of my name in my field can be discarded, as Microsoft Academic profile does indeed list my, and only my, publications (as well as a profile pic that they seem to have grabbed from an old university web page).

So my question is the following: how does Microsoft Academic Search actually generate rankings (of individual academics, but there are also rankings of journals)? It seems clear that not all citations and publications are considered equal (otherwise I would not end up so far to the front with a comparatively meager set of papers and citations), but how do they decide what is "worth" how much?

I am mostly looking for answers that refer to papers or web links coming from MS insiders on their ranking algorithms. Barring that, some well thought-through speculation from outsiders is also ok :)

Edit:

I just discovered that Microsoft Academic Search seems to employ the notion of a field rating for conferences and journals, and the rating for some of the venues that we prefer seems to be unreasonably high. However, that still does not fully explain my curious case (and it still leaves open the question of how these field ratings are generated in the first place).

  • 9
    +1 Interesting question. Perhaps Microsoft sent employees back from the future and therefore knows you are an upcoming star scientist who will change the world as we know it? – Marc Claesen Jan 29 '14 at 9:51
  • 2
    For the first name I tested, the university is completely wrong, many publications are missing, many are listed more than once, and many are erroneous and belong to somebody else in an entirely different field (say physics versus economy). – gerrit Jan 29 '14 at 9:55
  • Would love to see someone chime in on this one. – Jonathan Landrum Jan 31 '14 at 19:38
  • 1
    Having poked around my own entry in Microsoft Academic Search, I'm somewhat afraid the answer is "disastrously poorly". – Fomite Feb 4 '14 at 18:55
8
+200

how does Microsoft Academic Search actually generate rankings?

Since Microsoft Academic Search, like Google Scholar, is a proprietary vertical search engine and its algorithms and ranking systems are not open source, you're unlikely to get a detailed "how" answer unless a disgruntled former Microsoft employee decides to chime in.

Microsoft Academic Search does make some details of its rankings and results system available, however. This page offers a basic explanation of the search engine's approach. For both Microsoft and Google, citations play a major part in search result rankings, which has subjected academic search engines to criticism in the past. As the Academic Search page above indicates,

the information associated with Microsoft Academic Search author profiles is derived from the tens of millions of scholarly publications that are currently indexed by Microsoft Academic Search. Most of these publications have reference lists that Microsoft processes. The indexed publications and reference lists help create a snapshot of individual authors' publication history, productivity, and impact. As more content is indexed within Microsoft Academic Search, the accuracy and completeness of the author profile data improves.

Because Academic Search rankings can be distorted by factors including citation counts, incomplete publication indexes, and the field rating system you mentioned, it will inevitably produce some inaccurate results.

Speculation alert: it seems to me that the lion's share of ranking is accomplished through automatic citation list analysis, which is not necessarily a good indicator of

  • Prominence in one's field,
  • Number of publications (instead, it seems to emphasize how often a publication is cited elsewhere).

Finally, Academic Search is still in Beta, and I suspect that their rapidly growing index may not always be in perfect step with their ranking algorithms - in my personal experience with the search engine, there do seem to be cases of highly-ranked scholars or papers suddenly plummeting, then re-appearing again, which I assume is a consequence of ranking algorithm tinkering.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.