10

[EDIT (2016-11-11) based on a comment]

I have been invited as a reviewer for the IEEE International Conference on Image Processing (ICIP 2016, on image processing, in a broad domain sometimes associated to computer science). The ancient system was based on ticked keywords (by benevolent reviewers) in a list (and on the area chairs' expertise).

The novel system is called TPMS. It is based on the paper Toronto Paper Matching System: An automated paper-reviewer assignment system. It uses a machine learning ("statistical matching") technique based on a sample of the reviewer's publications. It has also been used for CVPR 2016 (Computer Vision and Pattern Recognition conference).

  • Has the TPMS system been successfully deployed?
  • Do reviewers or editors have a quantitative evaluation of the efficiency of such systems?
  • 1
    a request for feedback is not a question that can be answered. maybe you could instead ask whether the TPMS system has been successfully deployed. – Aaron Brick Nov 10 '16 at 16:40
  • Nice suggestion – Laurent Duval Nov 10 '16 at 16:45
4
+50

Laurent Charlin's reference (http://papermatching.cs.toronto.edu/webapp/profileBrowser/about_us/) suggests that it's been adopted by "over 50 conferences", including the following conferences:

  • European Conference on Computer Vision
  • Conference on Uncertainty in Artificial Intelligence
  • Conference on Neural Information Processing Systems
  • Conference on Computer Vision and Pattern Recognition
  • International Conference on Computer Vision
  • International Conference on Machine Learning
  • Artificial Intelligence and Statistics Conference

And it also "keeps the profiles of 8,000 reviewers". In machine learning circles, this is just about as good as it gets with non-mass-market data. The size of the training pool and the fact that it's been continuously worked on means it is likely to have become a fairly reliable tool.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.