There are very well known American and British universities, which have a very good reputation in a lot of different fields. Obvious examples being Cambridge, Harvard or Princeton.
But what about the state funded German universities, how is their international reputation? Why are the German schools ranked so poorly in international rankings like the Shanghai ranking?
University-wise reputations are somewhat of a self-amplifying phenomenon:
University-wise reputation is an established thing. This is opposed to the reputation of a department or faculty.
Students and academics want to be at a well-reputed university. Thus, well-reputed universities can be more selective and get better students and staff.
Better students and academics at reputed universities raise the level of teaching and research through existing qualities, higher teaching levels, cross-fertilisation, researchers having access to better students, etc.
Reputed universities actually are better as they produce better graduates and research, attract more funding, etc.
University-wise reputation becomes a reasonable criterion and gets further established. Go to Step 1.
However, without such a process, university-wise reputations hardly makes any sense: Most interactions in universities happen within departments or at least within faculties, and thus there is no reason to assume the qualities of different faculties correlate with each other. For example, the idea that a university with an excellent science faculty also must have a good law faculty is absurd (except for the above mechanism).
In some countries, such a process has happened; in Germany, it mostly didn’t. Germans rarely think about the reputation of a university, but rather about the reputation of a department, and there are indeed strong variabilities between those. This is at least partially due to historical factors, but there also some systemic factors:
In the German education system, students specialise on a field upon entering university. This reduces the interactions between departments as compared to other systems.
German universities are primarily funded by taxes not by fees and donations. People do not make donations to their alma mater; they pay taxes. Reputed universities cannot raise higher fees. There is no rich-get-richer amplification furthering the above process.
The German culture is rather egalitarian and particularly holds to the ideal of providing free and equal education to everybody (or in case of universities, everybody with certain prerequisites). The concept of an elite school or university is not generally well regarded. (Mind that whether those egalitarian ideals are actually achieved is another question.)
A considerable amount of research in Germany happens at dedicated research institutes (mostly Max Planck, Helmholtz, Leibniz, and Fraunhofer Institutes) that are usually only loosely affiliated with universities. Whatever reputation these institutes acquire does not fully rub off on universities (in public perception as well as methodic rankings).
Note how the German universities that have a somewhat generally good reputation tend to be located in beautiful and expensive cities and thus are more attractive to students with rich parents (which despite all egalitarian tendencies have better prerequisites). Finally, mind that there is a (disputed) initiative that may initialise the above process, but even it does, I would expect it to take decades to show effect – in particular if you measure success in Nobel prizes (see below).
Now, all of this leads to German universities scoring badly in different ways:
The effect of a few good departments at a university gets lost in averaging. The good departments in Germany are simply not clustered at a single university.
Scoring high in a university ranking is not such a relevant factor for universities, and thus they have no incentive to game those systems.
Even top departments are not that attractive to students that they can be as selective as a globally high-ranking university. Students simply select their place of study by other criteria.
A considerable portion of research successes happens at research institutes and not at universities and thus does not boost any university’s ranking.
For ARWU (Shanghai) in particular: This ranking mainly counts the extremes (Nobel prizes, high-impact papers, etc.). Broadly speaking, it looks at the highest percentile of research happening, not the median, average, or similar. Most of the aforementioned points are particularly bad for achieving such extremes, while hardly affecting the average quality of research and teaching.
There are lots of biases in international academic rankings, they tend to favour US and UK universities because they use criteria which are mostly relevant for a market-oriented academic system, such as the US/UK ones.
In many European countries the academic system is not market-oriented, or at least not as much as in the US and the UK: students fees are very low because higher education is mostly funded by public money. Universities are seen as a public service offering education to society as a whole, rather than a kind of commercial institution which sells knowledge and qualifications to those who can afford it. As a consequence Non-US/UK universities are not as incentivized to attract international students, they don't make big efforts to play the competition game since their income doesn't depend (at least not much) on their international ranking.
Of course this is only a simplified explanation, but this is the main reason why US/UK universities perform better than German universities (among others) in international rankings.
[Edit] Originally this was only a quick answer to explain what (I think) is an important structural and cultural difference with respect to international rankings between UK/US-like academic systems and others. My point was only to emphasize this difference to the attention of prospective students looking at these rankings, knowing that this is a common source of misunderstanding among them. I didn't imagine that the question and my answer would attract so much attention. I gladly admit that this is a rather simplistic answer, and I'm happy to see that other answers have done a much better job than me at analyzing other aspects of the question in detail.
A significant portion of this phenomena may simply be due to core problems in the methodology of these rankings systems. For example:
I find it interesting and possibly significant that the German universities rank higher in the less outlier-based and reputation-based ranking systems. Remember also that top 100 is still quite high, given that there are approximately 1000 graduate schools and thousands more 4-year colleges in the USA alone.
Furthermore, even the "objective" measures like "citations per faculty" and "number of papers in top 1% of field" are going to be dominated by the high tail of the distribution, rather than the actual educational and research opportunities provided to the median undergraduate or graduate student.
Thus, if the German universities have not been optimizing for these metrics, as suggested by other answers, I would expect the best ones to show up only in the medium-high rankings, even if they are extremely good institutions (as they have been).
Just writing up an additional point discussed in the comments to Erwans excellent answer. US universities vary massively in terms of funding and any excellence or quality measure you can think of amonst each other. In comparision, German universities are much more uniform. This means that if you only look at the very top, you will see a few US universities but not a single German one. At the same time any German university would count as fairly good compared to the average US university.
In terms of funding this is a result of politicial choice in Germany. If you want a university that scores in the top 10 internationally, a necessary but not sufficient condition is an amount of funding similar to the current top 10. No German university is anywhere close to that amount of money and the German government, which provides the bulk of their funding, is (currently) not interested in giving out that amount of funding to a single university. There are a number of programs to increase excellency but they are all targeted more broadly.
There are undoubtedly many things going on here, but some of this is almost certainly attributable to the broad sweep of history. Academic inquiry flourishes best under some sort of liberal democracy. Germany was the originator of the modern university system, but for most of the 20th century, the political environment in some or all of the country was very negative. During the Nazi period there was a "brain drain" of academic talent, e.g., many leading physicists who were Jews fled to the US or UK. Taking TU Dresden as an example, it was ruled by the Nazis, then "largely destroyed" (according to their web site) by the allied bombing. Immediately after the war, there was just a struggle for daily survival in Gernany. TU Dresden was then rebuilt but run under totalitarian communism until reunification.
You don't just get those years back for free. Often in economics if there is a setback, that simply becomes the new starting point for continued exponential growth, so where some country that didn't experience the setback gets $Ae^t$, the country where bad things happened sees $A'e^t$, where A' is just smaller. It's true that sometimes destruction can actually have paradoxical effects, such as allowing a fresh start. (IIRC this happened sometimes when wars destroyed obsolete rail systems.) But that's not a guarantee.
So liberal democracy matters in many ways, including this. That's one reason why the trend toward populist authoritarianism in places like Poland, the US, and Hungary is such a bad thing.
Edit: pointed out by by "henning -- reinstate Monica", my answer is not true as I wrote it since by using something called "Leistungszulage" you can negotiate top-ups to your standard salary.
One factor which has not been explicitly mentioned, but is somewhat related to the different amount of funding: In Germany, salaries of professors are regulated. This means that there are three types of professorships (W1 = Juniorprofessor, W2 and W3) for which there are fixed salaries no matter on who you are or where you have this professorship (they do change depeding on your years in the job, though; they also perhaps fluctuate in miniscule amount between the different "Bundesländer", but one can really ignore that for my point since the difference is too small). What this means is that if you are performing much better than the average in your field and can negotiate for a higher personal salary in a position at a US university, this option does just not exist in Germany. The best you can go for is negotiating e.g. more doctoral students or perhaps funding for your research/working conditions. I think this provides a high disincentive for over-performers to settle in Germany and might thus affect the outlier-measuring metrics used by many of the comparison schemes.
I would like to add one more point:
Top Institute like Max Planck, Helmholtz, Leibniz, Fraunhofer Institutes and many more offers Research (master, PhD, etc) in the German language, where hardly any international students gave more choice. Even these Institution and other Institution offers in the English language, which gives more completion for admission.
