I organized our department's seminar series recently. This is what I did:
know your audience
Are you trying to build collaborations with your department? Are you trying to expose your students to new ideas?
I'm at an all undergrad institution. We wanted speakers who might be recruiting grad students. So I looked to bring speakers from institutions where we've sent students in the past. I also wanted to expose the students to research areas not represented by our small faculty.
You know what you want. Who do you want to hear from?
what's the budget
We had very little in the way of budget, so I focused on nearby institutions. Luckily there are a hand-full of institutions within a 3 hour drive. Plus, it's way less trouble to get a speaker to come down for an afternoon. It's a pain for everyone when dealing with plane tickets and a hotel rooms.
I also looked around for some additional money. There are often specific instances where a funding agency like NASA or NSF will have money to send speakers around. You may not get to pick who you get. I was able to bring in someone from across the country who would have been way outside our usual budget.
People you already know are more likely to respond, but you won't make new connections if you don't try.
I dug around on department webpages, looking for anyone who seemed interesting and emailed them. I also asked if they could recommend one of their grad students or post docs who could come in their place if they could not.
Many potential speakers keep a list of available talks on their personal webpage. Ask for a specific existing talk, or a small modification to one. That's less work for them: more likely to come.
fill in with personal connections
After I got some spots on the agenda filled, I contacted the people I know. I can more easily convince them to come on some day that no one else wanted.
If you still have holes, go back to your closest neighbors. There is some grad student who can come (especially if you offer free lunch).
invite a mix
Hearing about the same thing week after week is boring. Try to invite a mix of topics and/or speakers.
- junior/senior faculty / private researcher / post doc / grad student
- men / women
- big research university / small college
- local / far flung
In the end you're in charge, so do what you want. If your colleague doesn't like the schedule, let them put it together next time. :)