I am a PhD student in computer science: is it worth spending time organizing a contest for a conference (e.g. a shared task for a conference in natural language processing)? I think that would be a great experience to have, and fits very well my research interests, but I am afraid it might slow down my research and therefore delay my graduation date.
Often in graduate school there are an abundance of neat opportunities and experiences; it can be difficult to decide which tasks to pursue. You'll need to consider the short and long-term career benefits of organizing the conference, as well as how it will be viewed within your program. In regards to short term career development, this may be an opportunity to provide an academic service activity for the conference, though there are often less labor-intensive ways to add that type of experience to your CV. However, if you're interested in a career in management or administration and want to gain experience in developing or implementing organizational activities, this might be a valuable long-term experience. Also, if the group task will result in data that you can later use for publications or products, it may be worth the investment; it may even serve as a unique research opportunity.
Whether you pursue this activity will likely also depend on whether your mentor is supportive. If you are concerned that this might delay your research and your graduation date, you will need to decide not only if you can live with pushing back graduation, but if your mentor or program will support that as well. If you are able to pitch this opportunity in terms of career development benefits, products, or even positive publicity your mentor or program may be supportive. They may even provide you with some undergraduate assistance to reduce the burden of coordinating the event. Before you have that discussion, make sure you have a clear picture of the pros and cons, as well as what (minimal) supports would allow you to take advantage of the opportunity at the lowest possible economic, time, and labor costs.
To add to the other answers here, you will frequently be encouraged to "contribute back" during your graduate training. The idea here is that you are giving back to the institution, and you can potentially list these activities on your CV.
The issue with this is that you're already giving to the institution. You're producing research and/or teaching, and you're much cheaper labour than anybody else (postdocs, professors), and require much less training and supervision for how much you do contribute. So, you really don't owe the university or department your time.
As for adding other activities to your CV, there is some benefit to this, but it is incredibly minimal and short-lived. No amount of other activity is going to replace a single paper, irregardless of where you are in the authorship list. A paper where you are author #19 out of 20 is still better than organising a dozen things. Once you move on beyond graduate school, the other activities that are expected of you change, and you'll find that your grad school volunteering isn't so impressive. You'll be expected to review papers or serve on committees, and not so much organising contests.
User30295's answer covers most of the "hard" considerations. I'll add some "soft" ones here.
First of all, organizing such a meeting will put you in contact with lots of people "who are somebody" in your research area. It is a stellar opportunity to get known. Such contacts will be invaluable if you later get stuck and need the right hint to overcome a thorny patch in your thesis (and later work!). You will also presumably meet (and work with) lots of other students. You will have first-hand access to some cutting-edge research.
But keep in mind that organizing such a meeting will also mean that just for the talks you are most interested in, you will be called away to solve some dumb foulup (yes, Murphy's law is unrelentless). And to be remembered as "the brilliant organizer of that contest that ran like silk in 2016, and who solved my domestic problems promptly" is more important than a talk or three.