Okay, as you say, this is very broad, and possibly argumentative. So, I'll try to section off my answer for your various sub-questions, and talk not so much about how I do come up (and organize) research ideas, but how I see it done by everyone (including me).
Coming up with ideas
The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but rather, “hmm... that's funny...” — Isaac Asimov
It's probably very akin to asking a large number of artists “how do you come up with inspiration?”, i.e. you can probably get one thousand different answers, and yet not useful answer at the same time. However, there are some elements that I think are common to all. You can't “trigger” new ideas to come into your mind, but you can put your mind into the right disposition to host these new ideas: recognize them and welcome them. Below is a list, certainly partial and limited, trying to detail my perspective in this matter:
Be challenged! Nothing sparks ideas more than being confronted with contradiction, healthy criticism, a spirited debate, maybe a bit of competition. Some people manage to do that by themselves, arguing against their ideas and improving them. I myself (and most of the colleagues and students I have seen) need an echo chamber, someone to discuss things with. If they're not exactly from your field, all the better, as they may have unusual/naïve/silly questions or expectations.
To give an example, some of the most “successful” ideas I have had came while answering questions, for example from a PhD student or colleague, and replying by “no, it doesn't work like that… in fact, it's probably always guaranteed to be false, because… see, it's linked to X… or maybe it's not? hum…”
Be curious! Ideas come from problems. Identifying worthy problems in your field of research, and dissecting larger issues into of specific problems of manageable scope, is at least as hard as coming up with new ideas. In the end my feeling is that, especially for a researcher, all ideas are the result of one’s curiosity.
Manage to get some free time for thinking (and not: teaching, supervising, tutoring, reviewing, writing, sleeping, …). Body and mind. Sure, an idea can pop into your head any time, but it's probably less likely to happen when you teach basic calculus all day that when you get some time to really think.
Know your field, know where a new development need to occur, what is currently missing. Read review papers, search for such ideas through people's articles or blog posts, discuss with senior colleagues who have a comprehensive view of the field, …
One of the ways you can come with ideas is by analyzing how different groups work in your fields, seeing what has been addressed and avoided, what big questions are still open, and how you can link between different works to build a coherent global picture… This is not always successful, but it usually generates some good ideas along the way!
Explore more or less closely related fields, and see if there is something from your background that you could apply to their problems, or ways you could build something together. Such ideas tend to be very strong, because you can oftentimes apply an entire branch of knowledge (ideas, methods, algorithms, etc.) to a very different problem. In that case, the added value comes from your different perspective, as you might try things that others would not think of.
Ways have been devised to come up with new ideas on a given topic, either alone or in group sessions. Brainstorming is probably the best know such method (and might be the most popular, in one form or another), but a really large number of creativity techniques have been developed. They can be applied both to enhance creativity or to boost problem solving efficiency.
Organizing ideas
A quote often attributed to Kant: “someone’s intelligence can be measured by the quantity of uncertainties that he can bear”. If that true, that has serious consequences for research. Accepting that your mind can only efficiently support a finite number of ongoing research ideas, you have to come up with ways to write them down, organize them, prioritize them, come back to them later, etc. Just as you cannot juggle with as many balls as you'd like, such “external” tools will help your brain focus on the ones that you assign high priority (or the ones to which it gives high priority; the brain works in funny ways).
Most people use very low-tech tools for that:
Notebooks, either sorted chronologically or thematically; in the later case, open a series of blanks pages for each new project/idea, and flip through the book whenever you want to check on them. I use a Moleskine (WP) for that purpose; having a nice, leather-bound notebook somehow helps me “value” it more and treat it with care (always have it with me, actually use it).
Post-it’s scattered through one’s (real or virtual) desktop. Downsides are obvious.
More people than I thought actually don't use any tools, and just keep all in their mind. Apparently it can be done, but I don't advise it.
But more complicated methodologies have been devised, that are supposed to help you with it:
- Mind mapping, either on paper or software-based.
- Using todo-list flat or two-dimensional todo-list software, or more complex task-tracking software (see, e.g. Trello).
- The software side of this question is already covered (though possibly not extensively) here on this very Q&A site.
Finally, don't underestimate the possibilities opened by delegating: people in charge of a specific project or sub-project (PhD students or post-docs) can be tasked with maintaining a list of ideas by all contributors of the project, to come to later on.
Answers to your miscellaneous smaller questions:
What proportion of your ideas for past papers come from; (i) colleagues, (ii) intentionally browsing the literature for ideas, (iii) on the spot inspiration, (iv) conferences, (v) other?
Most ideas are hardly “traceable” to one source or another. A given idea might have formed in my head during a conference, seeing how people were failing to address a certain issue, then crystallized during a discussion with colleagues, but would never have occurred to me if not for a literature review I had performed a few months before.
I'll come back a bit later and continue working on this answer :)
