In general, no. (This is based on my experience in the US.) One's employing institution does not have any claim on royalties from books written while a student or faculty member. However, it's possible that there might be exceptions: for example, sometimes a university may help financially in the publication of a book, and this might possibly be reflected in the publication contract. Of course, in such circumstances the book is not expected to make any money, which is why the university is helping out in the first place. 


A related and very common phenomenon is for the publisher to hold the copyright on an academic title. My own book is like this. I am entitled to royalties, but the press holds the copyright for some defined period. Again, this is due to the terrible economics of publishing academic monographs. 

There have been cases where universities have tried to assert very broad rights over the intellectual property of their faculty employees (e.g., lecture notes as well as books, etc), but I think these have generally failed. These sweeping assertions of rights have sometimes been motivated by the desire of administrators to claim some of the genuinely lucrative things now produced by some university researchers, such as patentable biotechnologies, and so on. That's a lucrative area with real money at stake, and one where the university's investment (in laboratory space and so on) is much higher. Books, not so much.