Is there any research/study/survey that tried to compare the earnings generated by open-access papers (researchers/institutions pay when publishing) vs. paywalled papers (researchers/institutions pay when reading)? I.e., for a given paper, do publishers make more money through the open-access model, or through the paywall model?
For subscription earnings, the most prevalent figure is in the ballpark of $5,000/ per paper. It is obtained by dividing the total revenue of the STM journal subscription market by the number of published paper in the same year. Some sources: this blog post and link therein provide estimate based on precise sources; this STM report (2015) mentions an estimate of £3095 per paper, with a rough decomposition of the cost.
Remember this is an average, I know a publishing house who would typically have prices of the order of $50 per page (in math, so about $1000 per paper) - but hey, they do a much better job than Springer or Elsevier, so of course they are 5 time cheaper. Heu, wait...
For Open Access article, things are way more complicated. First, it is not true that OA means that "researchers/institutions pay when publishing": Green OA is quite independent of journals (authors deposit their postprints in a repository) and many Gold OA journals, including high-standing ones, run on voluntary work or subsidies or both and do not ask for APC (Article Processing Charges, aka publishing fees). Here, it is my duty to introduce Fair Open Access.
Second, there is the case of hybrid journals (some papers are OA, the others are closed to non-subscribers). Figures have to the best of my knowledge not be compiled and publicly available.
Third, even for those Gold OA journal relying on APC, I do not know of a good source to get an average price; I have seen prices from $500 to $5,000, possibly more. But it is true that currently Gold OA seems quite lower priced than subscription on average. However Nature have said that if they where to turn OA, they would "have to" charge several tens of thousands of dollar per article. So one can guess that OA may have developed first in the lower-cost segment of the market, rather than being intrinsically lower-priced.
Daniel Pollock at @DeltaThink notes that OA publishing accounts for 20-25% of STM publisher output, but only 6-9% of revenue #stmfrankfurt
...implying that the earnings are (currently) lower for OA papers.
Daniel Pollock apparently is a publishing consultant. so those numbers are probably relatively accurate.
I would expect this to change now that more and more countries are signing Big Deal-type agreements with publishers on APCs.