As noted in the comments, there is no single answer, because when you are applying for a highly competitive graduate program you need to be both an "excellent student" and have additional strong qualifications that set you apart from all of the other excellent students.
That said, GPA is definitely a negative filter, and one can still extract some guesses. Here are MIT's requirements for entry of its own students into the internal EECS 5th year M.Eng. program: 4.25 on its 5-point scale (equivalent to 3.25 on the usual 4-point scale), plus at least 4.0 (3.0) in the most recent term. Since these are internal requirements, you can assume it is likely that external applicants for graduate programs will be measured a bit more harshly since they will be less well known.
Assuming a 4-point scale, I think that a safe rule of thumb for Ph.D. admissions is that if you've got at least a 3.75 both overall and in your field, you can stop thinking about GPA and start worrying about the rest of the package. If you're below 3.0, it will be extremely hard (though not necessarily impossible) to convince people that you are a strong enough student, even if you are very good in non-GPA aspects of admission. In between, the lower your GPA, the more you have to shine in other aspects in order to overcome GPA-based concerns: e.g., 3.6, not so hard, 3.25 very hard. This are rough heuristics, of course, and every case may differ...