Performance of the Toronto Paper Matching System (TPMS) for (conference) paper review assignment

Question

[EDIT (2016-11-11) based on a comment]

I have been invited as a reviewer for the IEEE International Conference on Image Processing (ICIP 2016, on image processing, in a broad domain sometimes associated to computer science). The ancient system was based on ticked keywords (by benevolent reviewers) in a list (and on the area chairs' expertise).

The novel system is called TPMS. It is based on the paper Toronto Paper Matching System: An automated paper-reviewer assignment system. It uses a machine learning ("statistical matching") technique based on a sample of the reviewer's publications. It has also been used for CVPR 2016 (Computer Vision and Pattern Recognition conference).

• Has the TPMS system been successfully deployed?
• Do reviewers or editors have a quantitative evaluation of the efficiency of such systems?

Answer 1

Laurent Charlin's reference (http://papermatching.cs.toronto.edu/webapp/profileBrowser/about_us/) suggests that it's been adopted by "over 50 conferences", including the following conferences:

• European Conference on Computer Vision
• Conference on Uncertainty in Artificial Intelligence
• Conference on Neural Information Processing Systems
• Conference on Computer Vision and Pattern Recognition
• International Conference on Computer Vision
• International Conference on Machine Learning
• Artificial Intelligence and Statistics Conference

And it also "keeps the profiles of 8,000 reviewers". In machine learning circles, this is just about as good as it gets with non-mass-market data. The size of the training pool and the fact that it's been continuously worked on means it is likely to have become a fairly reliable tool.