I am assisting in arranging a small conference in a highly mixed topic. There will be hard scientists, soft scientists, makers/creators/artists, researchers from many different fields etc.

I'm assisting in a number of small ways - one is how to decide on who should be able to present, and under what conditions/criteria. This is more difficult than I had imagined.

We will have no issue getting enough applicants, but we do have issues sorting through them. In the past, they've relied on a very quant-oriented process, however they've felt overall that this hasn't translated into the best pairing of speakers on panels, strength of tracks, etc. This year the proposal is to rely on more intuition, and qual-style evaluation (does it align with our goals, does it incorporate diversity, etc.).

Because we are quite small, there isn't the budget or capacity to get a wide panel of experts who can decide on quant-based merits for every applicant. We do have about 15 people who are active in the various tracks of the conference.

I'm trying to make this general enough to be helpful to others in the future, so the question remains simple: how does a very small conference evaluate conference proposals? Are there any best practices?

  • Could you elaborate more what is the "quant-oriented process" you used before? Because being in soft science (History), there are very few quantitative ways to measure the quality of our works. – Cochise Dec 1 '18 at 2:52
  • @Cochise this is a great point. I wasn't involved at that point so I can't speak specifically to their past methods, but my impression is that based on a number of criteria that they felt were 'hard' criteria, each evaluator would assign a score of each category and add them up and average the scores, to compare one candidate to the next. Of course this is inherently biased as well - it's not like they got 8/10 questions correct and were therefore sent on. – Gryph Dec 2 '18 at 3:45
  • I am voting to close as unclear, since the correct answer seems to depend entirely on what the organizers want to achieve. – jakebeal Oct 6 '19 at 11:22