It fills an acute need
Organizing a conference takes significant effort, but it also usually fills a need within a community. New conferences generally arise when some subfield or a geographic community within a field has a critical mass of research that they want to exchange, but that is poorly served by the existing options.
If ten years ago the FooBar conference occasionally got papers like "FooBaz is a nice new thing" but now there are many FooBaz papers that get rejected with reason "The paper is okay but outside the scope of FooBar" then everyone in the FooBaz community would benefit from a specialized conference on FooBaz - and if it is a success, then it often gets repeated and turns into a [semi]yearly tradition.
If you want to make progress in an emerging field, you are motivated to invest personal effort in making the field a success, to advertise your research and the related research that's growing out of it. If it's an estabilished field, then often you form some organization that unites the relevant scientists and is able to get funding and administrative resources for the explicit purpose of advancing that field - which involves organizing conferences.
It's a shared effort
I've actually never seen a scientist organizing a conference. Usually it would get managed by some university, institute or other organization (though through active initiative of their scientists) - it would piggyback on their existing administrative capacity. Similarly, organizing the papers and organizing the event is usually split among separate people so that the workload is manageable.
Also, I've seen many conferences that get 'rotated' among the community. If a conference makes sense, then multiple research centres are interested, otherwise you can just make a local seminar. If organizing a conference requires your [institution] involvement only, say, every 6 years, then it's not so tedious to make it intractable.