You are asking the question incorrectly. The grades alone don't tell whether the course is too easy. After all, upper division students have already proven themselves in earlier courses and the ones still there should be better "on average" than those in the first courses.
To improve the question, ask yourself what it is necessary for students to actually gain and know from your course. Ask how you are measuring that. Ask whether the students, by your own measurement techniques actually have demonstrated that knowledge. If the answer is that they are learning the material then giving out lower grades "on average" just disadvantages many of them for no reason at all.
But if their answers on exercises or exams indicate that they aren't learning the material then it might be that higher grades is disincentivizing them to study hard. In that case you need to provide proper incentives. That might be things that lower the average, but it might be other things as well.
For some, but not all students, a bit of "stick" will get them moving. For others, carrots are much more effective.
But the point of it all is learning, not grading. If your grades don't measure learning then you have a bigger problem than you think you do.
But the main question is answered by comparing your syllabus to the requirements of the course and the field and making sure that you are fairly measuring that the students are able to demonstrate competence.
And, if you do conclude that it needs to be more rigorous, then the ways to improve things is to give more assignments, readings, exercises, whatever, depending on the field. Make them harder, require better solutions, etc. Learning requires hard work, just as does excellence in, say, athletics.