How can the audience be sucessfully involved in best-paper awards?

Question

Me and a colleague are co-organizing a conference next year (which will be hosted in our city). Neither of us has ever organized a conference before. It is a yearly conference that so far has happend 3 times at the same location. The original organizers wanted to grow the conference and make it more international and have started to look for collaborators and co-hosts in other countries - this is where we come in. The field is a multi-disciplinary engineering sub-section.

The original organizers are present and offer support and guidance, but give us a lot of freedom concerning structure, specific topics, etc. I would very much like to implement a small best paper award (this would be the first instance of the conference that has such an award), as I think it is nice (especially for early career researchers) to get recognition this way. Everyone is on board with the idea but noone has any experience on how to chose the awardee. I asked my PhD supervisor (who is not part of the conference organization crew), and he told me that he only ever had the steering comittee chose who to give an award to.

We would like to give the audience a chance to vote, too, as a good presentation should not only be of great quality content wise, but also presented well and understandably--a matter that an audience might be more fit to judge. We wonder, though, if this is very uncommon (I have never experienced any instance of "public" voting for an award at a conference) and will lead to biased voting because some presenter might know more people and is more popular.

So is audience-voting for best-paper awards something that actually happens and what should one consider when implementing it to avoid skewed and biased results?

Answer 1

Yes sure, at some conferences participants are asked to vote for best talk, best poster or similar. It is probably not the norm, but it does happen.

At one rather well-known and also high quality conference in my field, all poster presenters give a 3 minutes talk about their poster, and after the following poster session votes are collected from the participants. Traditionally, most presenters give a serious presentation, but some make a quite a fuss and are more on the humorous side. The latter tend to have a better chance to win, and it is not always based on scientific merit. But that is no big issue from my perspective (a rubbish poster would not win, of course). It is just a different emphasis than when for example a pre-determined jury decides.

Also, the conference is not so big (approx. 500 participants) and some big research groups come with basically all their members (maybe 30-50 people, just an estimation). Members of these groups also have a better chance because they may agree before who they vote for, and an extra 50 votes easily can decide the outcome. These posters are often high in quality, but not necessarily the best. This is more problematic from my perspective because participants from other groups have a much smaller chance to win, no matter how brillant their poster is.

Whatever the method is to decide who gets the prize, it is always a debatable decision. There is usually not the one best contribution, but quite a few very good ones that all would deserve the prize. I am not sure if there can be an awardee selection completely ruling out any bias.

Answer 2

I think you are conflating two mostly-independent objectives.

1. Decide the award based on certain criteria (such as the quality of the presentation) and avoid it being a popularity contest.
2. Give the audience a chance to influence who should win the prize.

The second one is perfectly justifiable but I am not sure it has anything to do with the first one. If you want to do it, replace the "award panel" in what follows by "the whole audience".

For (1), I would prepare a (short!) list of criteria and ask the award panel to score each presentation on all criteria. The way to combine those scores into a decision (best weighted average?) should be announced in advance to the award panel. Here's my attempt at a three-point list for oral presentations:

• The work is original, covers new ground and/or establishes new techniques in the field
• The work is carefully executed, adheres to strict ethical and methodological guidelines, and/or includes appropriate discussion of possible limitations
• The presentation is clear; the work's scope, methods and results are easily understood from the presentation

The point of a scoring schedule is not (as in exams) to communicate feedback to the presenters. Rather, it is to force panelists to assess each item as objectively as possible, to minimize conscious and unconscious biases. You’re going to feel that the student you taught in undergrad class made a better presentation than the one you never met before, but presumably that bias is higher when asked "which were the three best presentations" instead of "score all the presentations based on criteria X, Y and Z". The latter requires the scorer to be satisfied with the relative rankings of all presentations.