If I publish a book on my research topics while employed for an academic institution, can they claim part of the revenues I get from royalties ? My question is relative to Europe, but if someone knows also for other countries, I will welcome additional answers.
In general, no. (This is based on my experience in the US.) One's employing institution does not have any claim on royalties from books written while a student or faculty member. However, it's possible that there might be exceptions: for example, sometimes a university may help financially in the publication of a book, and this might be reflected in the publication contract. Of course, in such circumstances the book is not expected to make any money, which is why the university is helping out in the first place.
A related and very common phenomenon is for the publisher to hold the copyright on an academic title. My own book is like this. I am entitled to royalties, but the press holds the copyright for some defined period. Again, this is due to the terrible economics of publishing academic monographs.
There have been cases where universities have tried to assert very broad rights over the intellectual property of their faculty employees (e.g., lecture notes as well as books, etc), but I think these have generally failed. It does still happen: here's an example where the University of Louisiana is trying to broadly claim rights to scholarly output, including royalties from books. These sweeping assertions of rights have sometimes been motivated by the desire of administrators to claim a share of some of the genuinely lucrative things now produced by some university researchers, such as patentable biotechnologies, with books caught up in the net but not really directly targeted. Patents are an area with real money at stake, where the university's investment (in laboratory space and so on) is much higher, and where university claims on income from work done while employed are strongly and successfully asserted. Books, not so much.
Perhaps you should check with your own institution as some universities (at least in the UK) have a policy on this issue. Also your employment contract might already cover this. For example, in several UK universities the copyright of course notes and lecture notes lecturers create often belong to the institution, not the individual lecturer. And the same might apply to books they write unless individually negotiated.
Reiterating what others have said: there is danger here, but at "better" universities in the US, the venal impulses of university administrations have at least recognized that they'll not make gazillions of dollars on grabbing royalties on textbooks or monographs, ... so they just let it go.
One should pay attention to the local rules. Yes, the UK rules in the last few years are disturbing... Maybe they've changed.
In the US, the contractual idea is that if one does something "specified, under contract", that the product is owned by the "entity" that engaged one to produce the work. Universities have been a teensy-bit more ... fair... about this kind of thing, but one should look around.
But we should address the dangerous cases: yes, some colleges/universities will claim that whatever you write/produce/do is their property. (I can't help recommending that you send them stool samples... maybe daily...)
In summary, obviously, try to use common sense. Yet be alter to the (duh, human nature) problem that "things are more complicated on the ground".
The net is that novices are coerced to give up "rights". This is ill, but I cannot change it. Apparently we must all "cope". (Sorry I can't give better/happier advice" It is disturbing ...)