Why it matters
In my field, cum laude or rite (the two worst German PhD grades) are rarely ever awarded and if they are, it’s usually a sign that there were major issues and a strong hint that the PhD should not consider any career aspirations in academia. Such PhDs often end in positions which mainly care about the title for its own sake (e.g., to impress customers). Your field may differ a bit, but even in law (which is known for using the entire spectrum of grades), such grades would not bode well on an academic career.
I would be careful with taking the position that the grade does not matter. As already said, the vast majority of PhDs that stay in academia have either summa cum laude or magna cum laude (the two best grades). Since getting a summa requires at least a bit of luck in many examination rules¹, many people do rightfully not care about the difference between the two. On the other hand, not having either is likely to raise some eyebrows and may prevent you from getting your desired position without you ever knowing why. Even if you find stories of people who had a successful academic career in your field with a cum laude, beware of survivor’s bias.
Moreover, many job offers in Germany (in particular at government agencies) explicitly require a very good PhD². This includes professorships to the extent that it is in the text of the respective laws (Landeshochschulgesetz), e.g., the one for Northrhine-Westphalia states (translation mine):
Einstellungsvoraussetzungen für Hochschullehrerinnen und Hochschullehrer sind […]:
[…]
besondere Befähigung zu wissenschaftlicher Arbeit, die in der Regel durch die Qualität einer Promotion nachgewiesen wird; […]
Preconditions for being hired as a professor are […]:
[…]
an outstanding qualification for scientific work, which usually is evidenced by the quality of the doctorate; […]
Now, this does say “usually”, which means it is technically not impossible to get a professorship with a bad PhD, only more difficult. But that already can be a severe disadvantage and may mean that you are ending up with a considerably worse position.
One function of such rules and phrases in job offers are things like avoiding nepotism or having to hire a bad candidate due to lack of better ones. Still if the criterion is phrased respectively, it may be legally impossible to hire you since you fail a hard criterion. If I found out that the successful candidate for a public job I failed to get only has a rite, I would certainly suspect foul play and consider legal action.
All in all, this grade may very well be career-ending.
¹ I know of cases where somebody did not get a summa only because they did not perform top-notch in a one-hour oral examination on their field of study that was unrelated to the topic of the PhD. Consider a theoretical cosmologist PhD being asked about lasers.
² Including your grades on your CV as well as providing a copy of the actual PhD certificate is the default in the hiring process, so you cannot really hide it.
What to be scared of
I am now considering to appeal to the university. But, I am scared.
… of what?
It sounds like you have little left to lose here.
If your PhD gets re-evaluated from scratch, you are very unlikely to end up with a worse grade.
There is a small risk that you earn a reputation over making a fuss about this, but those who care about this will also likely care about your grade.
On the other hand, if I were to consider you for a position and asked you for the story behind your grade (and I probably would), I would rather be negatively surprised if you hadn’t made a fuss.
What to do
Now, every faculty in Germany has its own PhD rules and therefore it is difficult to make any general statement here.
However, there are several aspects of your story where my PhD rules could have kicked in:
There would be two further members of the committee who can intervene in case of major disputes or grades being given with questionable reason or an accusation of academic misconduct (using material without a reference) against which you could not defend yourself.
The entire committee (four professors) grades the defense.
If the two main examiners differ by more than one grade, a third examiner would have to be consulted.
If your thesis is deemed to have major flaws, you have to be given the opportunity to amend it.
Grades for the thesis have to be given independently and cannot be changed afterwards. (It sounds like the “bad” examiner wanted to give a cum laude, so it cannot be the middle, unless he changed it to a worse grade later.)
There is a written record of committee’s decisions.
Nobody else would have been admitted to attend the committee making the decision. It is particularly weird that students can attend, but you cannot.
I recommend that you thoroughly read your faculty’s PhD rules and consult with whoever oversees the process (usually the dean). Also talk to the “good” examiner (your supervisor, I presume) to get a more detailed story of what happened.
Note that most PhD rules also forbid you to just submit your thesis to another university.
Caveats
Beware that you very likely did not get the full story so far.
For example, attendees of the committee’s decision tend to primarily report the most egregious aspects even though other points had been made.