3

I noticed lately the increase of conducting shared tasks, what are they and what is the rationale behind them? why do we need them since we already have workshops and conferences?

  • 3
    increase of conducting shared tasks In what context? – scaaahu May 28 '16 at 13:47
  • 1
    Like SemEval and qats2016 (qats2016.github.io/shared.html), to name two examples. – Iris Garden May 28 '16 at 14:07
  • 1
    OK, so what do you not understand about the QATS 2016 one that you linked to? It appears to be an event like a hackathon where academics get together to actually work on a problem. Lots of workshops and conferences are talks only and no joint work. A shared task appears to bring the work back to a workshop. – Bill Barth May 28 '16 at 14:18
  • What the rational behind doing so? – Iris Garden May 28 '16 at 14:19
  • 7
    The rationale is to get a bunch of smart people together to actually solve or at least make progress on open academic problems. – Bill Barth May 28 '16 at 16:01
7

The rationale is to get a bunch of smart people together to actually solve or at least make progress on open academic problems. I've never been to a "workshop" that actually did any work on the problem. Conferences are typically just people giving presentations and not really working on problems in my experience.

| improve this answer | |
  • 4
    I've never been to a "workshop" that actually did any work on the problem. -- I have, but they're rare, and they have to be organized with that explicit goal in mind from the beginning. In particular: no talks. – JeffE May 28 '16 at 19:46
4

Shared tasks are organized to tackle specific problems which are challenging to tackle by spare research groups for several reasons. Typically, it's the availability of data. One key element of a successful challenge is the goal: it has to be challenging, specific and possibly lacking data. Natural Language Processing (NLP) is one of the fields that has a very long tradition of challenges, to the point where plenty of advances are made because of their organization. Why?

Take the case of clinical NLP challenges. As a researcher, it won't be easy to have access to authentic de-identified clinical records from hospitals. They are hard to get, legally challenging to manage and distribute, and very expensive to annotate (they can't be annotated using Mechanical Turk or similar, you want domain expert to do the job). Finally, in the era or Machine Learning (ML), you don't want 100 clinical records, you want much much more. This is a scenario that prevents almost any research group to invest months of data acquisition for the sake of tackling a specific task. Moreover, if a research group interested in this endeavor exists, it will hardly release such data for free. It's more likely it will keep studying those data and get as many publications as it can out of it. But this is a pity because is cutting out everybody else from the discovery process.

In those cases, shared tasks are the solution. They are sponsored by several institutions, each one contributing at some level: providing experience, data, secure hosting and distributing infrastructure. With those pull of forces, interesting unexplored but important problems become attractive to researchers. As a plus, almost always, at the end of the challenge, the data become publicly downloadable to everybody (not just the attendees).

In terms of work, as a participant, the major part of the work is done before the workshop in your own research group. You work for months on a methodology that would solve the proposed problem. You attend the workshop only to meet the other participants, to present them your results, and share your tricks.

I've played both roles: participant and co-organizer. It's tons of work in both cases, but it's also a great pleasure to crack problems nobody ever attempted before, or to declare problems solved. ^_^

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.