What you list on a CV depends on the purpose of the CV. The idea that you have a single CV isn't really accurate. For example, using an NIH CV/biosketch for a job application would not be a good strategy. I believe that a "long" CV should have everything on it. I add things to my long CV as I do them which makes creating shorter CVs tailored for a particular purpose much easier since deleting is much easier than adding.
There are many things for which there is a benefit of listing talks that you have given at other departments/labs on the CV, the issue is really what "heading" they come under. The most critical thing when divining things up is that each item on your CV can only be listed once. You cannot list a conference you attended, gave a talk at, had an abstract published for, and a paper published for four times. It can only go on the CV once.
I try and only divide things into sections based on "concrete" criteria. It is clear when you are giving an invited talk at a conference, but it is not clear whether a departmental talk is invited or not. Similarly, it is not always clear whether a departmental seminar is a job talk or not.
The relevant sections on my CV are (a) Conference Papers, (b) Conference Abstracts, and (c) Research Presentations. Each "presentation" only goes under one category. I limit the "abstract" category to conferences that produce archival abstract books that people have a fighting chance of finding. Talks and poster presentations at conferences that don't have archival abstract, departmental seminars, talks to research groups, and job talks, go under "research presentations". I have a separate teaching presentations section for teaching based job talks.
I split the papers and abstracts sections into posters/talks and invited/submitted (someday hopefully I will add key note). The presentations I split into internal and external. I like to keep track of my internal presentations for annual review CV since it lets me demonstrate departmental citizenship. I tend not to split the presentations into posters/talks or invited/submitted since as I mention before, it is too difficult.