Is the prestige / difficulty of my undergraduate school taken into consideration when I apply to graduate school? I understand that it is not an excuse my lower grades (however, my GPA is respectable). I am wondering if this has an influence.
The admission committee I serve has 11 members and there will probably be 11 unique answers to this question. Generally, it matters, but the degree it matters and how much it'd factor into the decision varies member to member, committee to committee.
There is thought such as "A grade B+ in prestigious university is better than a grade A in a so-so university." And there is thought such as "This so-called prestigious university tends to inflate grades, so a mediocre grade is going to be a bad news." The amount of subjectivity and variation can be astounding.
To complicate the problem further, what is "prestige?" Generally, people can name up to 10 prestigious universities, and then the rest are somewhat hazy. Some committees attempt to use acceptance rates as a representation and yet it's far from perfect: program by program the acceptance rates can differ, and such fine information is seldom available.
So, they do care, but it's difficult to predict how much. I'd say don't worry too much about the existing GPAs. If you can improve it, take some relevant courses and get some good grades, if not, then present the best aspects of yours through showcasing your course combination (e.g. did you take more challenging courses over the easier?) and trend (e.g. did your grades improve along time?), clear sense of a long-term plan, a convincing personal statement, and impressive open exam scores (such as TOEFL and GRE in the US) to order to leave a good impression.
Yes, it will not be a huge factor, but if admissions are competitive for the program you are applying to, imagine another application identical to yours in every way except your institution: that might be the only way for the admissions committee to make a decision. Furthermore, graduate schools can be rated on the pedigree of their students, both before admission and after graduation. Not all is lost, however - you have many other ways to distinguish yourself, and I would never recommend someone choose an undergraduate institution solely for its prestige when applying to graduate schools.
Broadly speaking, I think the importance of your undergraduate institution is very dependent on the institution you are applying to - the most prestigious institutions are probably the ones that care the most about prestige.
Lastly, if you have already completed/are near completion of your undergraduate program, you don't have much control over this anymore - focus on what you can actually influence instead of what you cannot.