12

[EDIT (2016-11-11) based on a comment]

I have been invited as a reviewer for the IEEE International Conference on Image Processing (ICIP 2016, on image processing, in a broad domain sometimes associated to computer science). The ancient system was based on ticked keywords (by benevolent reviewers) in a list (and on the area chairs' expertise).

The novel system is called TPMS. It is based on the paper Toronto Paper Matching System: An automated paper-reviewer assignment system. It uses a machine learning ("statistical matching") technique based on a sample of the reviewer's publications. It has also been used for CVPR 2016 (Computer Vision and Pattern Recognition conference).

  • Has the TPMS system been successfully deployed?
  • Do reviewers or editors have a quantitative evaluation of the efficiency of such systems?
2
  • 1
    a request for feedback is not a question that can be answered. maybe you could instead ask whether the TPMS system has been successfully deployed. Nov 10, 2016 at 16:40
  • Nice suggestion Nov 10, 2016 at 16:45

1 Answer 1

6
+50

Laurent Charlin's reference (http://papermatching.cs.toronto.edu/webapp/profileBrowser/about_us/) suggests that it's been adopted by "over 50 conferences", including the following conferences:

  • European Conference on Computer Vision
  • Conference on Uncertainty in Artificial Intelligence
  • Conference on Neural Information Processing Systems
  • Conference on Computer Vision and Pattern Recognition
  • International Conference on Computer Vision
  • International Conference on Machine Learning
  • Artificial Intelligence and Statistics Conference

And it also "keeps the profiles of 8,000 reviewers". In machine learning circles, this is just about as good as it gets with non-mass-market data. The size of the training pool and the fact that it's been continuously worked on means it is likely to have become a fairly reliable tool.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .