I make a simple distinction based on what I'm looking for. Essentially, am I tackling a problem that I know about, or am I looking for a new problem/area to expand into?
I'd suggest you to have atleast this broad an idea before dipping into any material, otherwise you may easily get lost in the sea of information available today. The risk of getting too deep into something and forgetting why you got there on the first place is very real.
EDIT: For this problem, I'd recommend taking a look at this question.
Now, if a problem is known, I'd go straight to research papers and get an idea of what approach people follow, what is being tried, what gap areas exist, and so on. I'd go to a book only if (a) it is referenced in a part of the paper I find relevant, or (b) the explanation proposed hinges on something I know very little of. In (b), a book is good because typically it is written in a more pedagogical style and hence easier to understand.
On the other hand, if I'm looking for an area to expand into, I'd first consult books on related topics to acquire some understanding before jumping in to literature. Yes, it is time consuming, and reading papers directly seems like a shortcut ('you anyway need to read them once you select your problem, so why not start there'). Nevertheless, I personally believe that a firm understanding at the beginning is necessary to attempt formulating your problem.
Just to reinforce, let us consider these two situations and see what happens if we reverse the approaches.
(a) You know the problem already, but start off with a textbook- you will spend a lot of time reading. Since a good book is engaging, you will find it hard to put down, and may start jumping from the chapter you were interested in to another chapter that is referenced somewhere (another quality of good books). At the end, you've spent a lot of time, gathered a lot of information and hopefully some knowledge, but you are no further on your research problem.
(b) You don't know a field, but start reading literature- you may get lost in jargon. Also, you may think a contextual statement is generally valid, leading to misconceptions. Finally, you will rely completely on the approach and methodology of a particular author with a particular agenda (i.e. to advance a particular viewpoint/theory) and therefore adopt all his/her biases. A book will generally be written in a neutral way (the agenda is to disseminate information), which will help you to critically analyse any research result.