I noticed lately the increase of conducting shared tasks, what are they and what is the rationale behind them? why do we need them since we already have workshops and conferences?

  • 3
    increase of conducting shared tasks In what context?
    – Nobody
    May 28, 2016 at 13:47
  • 1
    Like SemEval and qats2016 (qats2016.github.io/shared.html), to name two examples. May 28, 2016 at 14:07
  • 1
    OK, so what do you not understand about the QATS 2016 one that you linked to? It appears to be an event like a hackathon where academics get together to actually work on a problem. Lots of workshops and conferences are talks only and no joint work. A shared task appears to bring the work back to a workshop.
    – Bill Barth
    May 28, 2016 at 14:18
  • What the rational behind doing so? May 28, 2016 at 14:19
  • 7
    The rationale is to get a bunch of smart people together to actually solve or at least make progress on open academic problems.
    – Bill Barth
    May 28, 2016 at 16:01

2 Answers 2


The rationale is to get a bunch of smart people together to actually solve or at least make progress on open academic problems. I've never been to a "workshop" that actually did any work on the problem. Conferences are typically just people giving presentations and not really working on problems in my experience.

  • 4
    I've never been to a "workshop" that actually did any work on the problem. -- I have, but they're rare, and they have to be organized with that explicit goal in mind from the beginning. In particular: no talks.
    – JeffE
    May 28, 2016 at 19:46

Shared tasks are organized to tackle specific problems which are challenging to tackle by spare research groups for several reasons. Typically, it's the availability of data. One key element of a successful challenge is the goal: it has to be challenging, specific and possibly lacking data. Natural Language Processing (NLP) is one of the fields that has a very long tradition of challenges, to the point where plenty of advances are made because of their organization. Why?

Take the case of clinical NLP challenges. As a researcher, it won't be easy to have access to authentic de-identified clinical records from hospitals. They are hard to get, legally challenging to manage and distribute, and very expensive to annotate (they can't be annotated using Mechanical Turk or similar, you want domain expert to do the job). Finally, in the era or Machine Learning (ML), you don't want 100 clinical records, you want much much more. This is a scenario that prevents almost any research group to invest months of data acquisition for the sake of tackling a specific task. Moreover, if a research group interested in this endeavor exists, it will hardly release such data for free. It's more likely it will keep studying those data and get as many publications as it can out of it. But this is a pity because is cutting out everybody else from the discovery process.

In those cases, shared tasks are the solution. They are sponsored by several institutions, each one contributing at some level: providing experience, data, secure hosting and distributing infrastructure. With those pull of forces, interesting unexplored but important problems become attractive to researchers. As a plus, almost always, at the end of the challenge, the data become publicly downloadable to everybody (not just the attendees).

In terms of work, as a participant, the major part of the work is done before the workshop in your own research group. You work for months on a methodology that would solve the proposed problem. You attend the workshop only to meet the other participants, to present them your results, and share your tricks.

I've played both roles: participant and co-organizer. It's tons of work in both cases, but it's also a great pleasure to crack problems nobody ever attempted before, or to declare problems solved. ^_^

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .