There are several issues to address in this story.
1. Is the grading really based on what the instructor remembers?
Remembering who contributed during the lectures is like remembering who answered the questions right in the midterm. The latter seems foolish since there is a hard proof that you have attended the midterm, written down the answers and there is an answer sheet. However, the former can also be kept as a document (for instance, putting a mark near the student's name on the attendance sheet), and can be checked later on just as a midterm exam paper.
In this case, it is unfortunately your fault not to keep track of participation points. You should have appealed against the method your instructor follows from the beginning and made sure that every contribution you have made is recorded.
2. Your instructor might be giving marks on the worthiness of the comment made
Not every comment during the lecture is a contribution. For instance, in a math class, an instructor tells that two plus two is equal to four. A student who says "Oh, than two plus three equals to five!" might not have made any contribution, but a student asks "Why is two plus two is equal to two times two?" might have asked a good question that improves the course flow.
3. You might think that the other students were inactive but they were actually more active than you
Inline with the second issue, some of the students -- who got more participation marks -- might have asked really interesting questions for the sake of the course subject, but you might have missed them.
4. There are witnesses
If you truly believe that you desire the participation grade, then you should talk to the other students that also think just as you. A bunch of them will probably be at your side when you appeal to the grading.
5. What is the university policy?
Remember that an instructor can always tear apart a midterm paper with the attendance list, and claim that the student has never attended the midterm, and the attendance sheet is lost.
Is there any rule against this situation? If so, then there should be a rule against "remembering the contributors".
What I am trying to point out that reading a story subjectively would be misleading. Since it is you who takes a low grade, you as well might be over-subjective about the issue. If you are not a senior student, then you can always take the course for improvement and take a lesson out of this incident.