81

I find it hard to write a literature review, even though I have Google, the Internet, and all free, very easy to search for information tools. I have to search for days to find a piece of information that is related to my research. I really would not be able to do that without the help of the internet and the technology.

This really makes me wonder how people in the past did that:

  • Did they have to read the entire book to find a piece of information? I know they might have used the index, but still this wouldn’t have given them the details of what they were looking for.

  • Does that mean people in the past (before the Internet) worked harder to achieve their degrees?

  • Does that mean that research before relied less on references?

  • 25
    An interesting observation from 2003: in the field of language teaching methods, the campus bookstore gives the impression the field has only existed for ten years; the library gives the impression nothing has happened in the field for twenty years. – WGroleau Jun 25 at 6:18
  • 6
    Footnotes were useful for tracing ideas backwards in time. This results in a tree structure. One of the early uses of computers (1960s) was to take some 20,000 footnotes from professional journals and sort them in inverse fashion, resulting in a data set that could be used for tracing ideas forward in time. Not an answer to your question, but an interesting tidbit. – Walter Mitty Jun 25 at 10:08
  • 17
    While I certainly agree with the premise of this question, it makes me feel older than I'm comfortable with... – Ken Zein Jun 25 at 13:59
  • 52
    Your imagination may be exaggerating the difficulties. Things weren't really that hard when I was younger, although it got a little inconvenient for a while after the destruction of the Great Library of Alexandria. – Ben Crowell Jun 25 at 17:59
  • 7
    Anecdote: As a first-year grad student in early 1967, I heard about measurable cardinals and wanted to know what they are. I was unable to find a definition until several months later when Joe Shoenfield's book "Mathematical Logic" appeared. Dana Scott's paper "Measurable cardinals and constructible sets" had appeared several years earlier, but I didn't know about it. – Andreas Blass Jun 25 at 21:06

13 Answers 13

107

We depended on libraries and librarians. Grad students would spend hours in, say, the math section of a good academic library, going from book to book and taking copious notes (on paper, of course).

But, often enough, the next paper we needed to look at wasn't in that library at all, so you would go to the librarian and ask for a loan of the resource from another library.

But it was also an important technique to use the librarian as a 'knowledge expert' who could, and would, suggest things for you to look at and places to look. I don't know if librarians still get the training to do that.

And, of course, you could ask your colleagues for hints about which rocks to uncover to find the gems.

  • 58
    When I needed to access papers I started by collecting coins. Some went to feed a parking meter. The photocopy machines got most of them. – Patricia Shanahan Jun 24 at 13:28
  • 4
    I made good use of the interlibrary loan system - even though the uni I was at had a very good engineering library, I was after books by Judge, Ricardo, Cummins etc all famous names then and still now... oh, and most important : we could read... I ask students which books they have read and most come back with “I only read the condensed version”... – Solar Mike Jun 24 at 17:32
  • 11
    In the mid nineties, while still at school, I did my work experience week at the Bodleian Library in "the stacks". Requests for books would be written on slips of paper and fired via pneumatic tube from the reading rooms across the road. The work was to go and find the requested books, load them into wooden boxes, and put these boxes into the automatic delivery system - an impressive mechanical machine, which would 'read' the destination according to the position of a slider on the front of each box. Now it's all been dismantled and only exists in the minds of people such as myself. – Aaron F Jun 25 at 8:41
  • 12
    Ah, none of the old-timers have yet mentioned the dreaded micro-fiche/micro-film. A true joy (not). – Buffy Jun 25 at 11:58
  • 3
    @Buffy I was trying hard to forget those... smudges, old gum over the one bit you needed, creased, oh and just when you found what you wanted, the bulb would blow with no spare until tomorrow... – Solar Mike Jun 25 at 18:51
48

One point that the other answers have passed over is that there were various services the libraries subscribed to which surveyed the literature and provided abstracts and cross indexing of the primary journals. Science Citation Index was mentioned in a comment, but there was also Science Abstracts, which had been published since 1898. These were hefty print volumes, and consumed a significant fraction of the library's shelf space and budget.

You'd typically start with a paper your mentor pointed you to, use the Science Citation Index to see who had cited that paper, then look up those papers in Science Abstracts, march off to the shelves to find the appropriate bound copies of the journals, and then plug nickels into the copy machine. If your institution didn't have that journal, well, that was what the interlibrary loan was for.

  • 4
    There were also abstract collections for each discipline, so you had Biological Abstracts, Zoological Abstracts, etc. Chemical Abstracts looked like this. – user13380 Jun 25 at 9:16
  • 2
    In my undergrad education in chemistry just 10 years ago, we had a class that taught us how find research using indexes like Beilstein. I found it quite fun leafing through old books to find answers. I personally find it much easier to browse and visually scan scientific literature in print than on screen. – ipetrik Jun 25 at 23:39
31

(Comment extended to post:)

My impression is that part of the answer is "they didn't", or more precisely "they were only as good at it as their own knowledge and that of their communities". In particular, at least anecdotally, many things in mathematics were discovered in parallel for lack of easy communication and inter-visibility. [This is complementary to @Buffy's answer, which explains why people found anything at all.]

I have tried to put this on solid footing by comparing numbers of references in a 1970 issue of a mathematical journal vs. in a 2019 issue of the same journal. This turned out to be surprisingly nontrivial. Firstly, it's not clear if I am comparing apples to apples, since most journals have changed their publication criteria and sometimes even their (implicit) subject within these 49 years. Secondly, papers in almost every part of mathematics have gotten much longer (by a factor of 2.8 in my sample). Thirdly, empirically, Project Euclid bans your IP if you load more than about 20 PDFs in rapid succession, and Sci-Hub is slow and has captchas. There might be a way to do such research using MathSciNet, but I am nowhere near proficient enough at its use.

So I ended up comparing Proceedings of the AMS (due to their long back-catalog of freely accessible issues), and came up with this ("reference" means "bibliography item", not "place where a bibliography item is being cited"):

I picked 13 of the papers from Proc. AMS 141 (2019) #12 (more or less picking the first 13, except I skipped a few from the Abhyankar cluster since he writes and cites in rather idiosyncratic ways). The average paper has 1.5 references per page:

$\dfrac{\dfrac{11}{7}+\dfrac{22}{13}+\dfrac{19}{15}+\dfrac{14}{6}+\dfrac{15}{13}+\dfrac{31}{15}+\dfrac{18}{13}+\dfrac{9}{5}+\dfrac{10}{4}+\dfrac{9}{11}+\dfrac{9}{7}+\dfrac{8}{10}+\dfrac{5}{6}}{13}= 1.5$

I picked 13 of the papers from Proc. AMS 24 (1970) #1. The average paper has 1.2 references per page:

$\dfrac{\dfrac{2}{5}+\dfrac{2}{3}+\dfrac{5}{9}+\dfrac{2}{4}+\dfrac{3}{2}+\dfrac{5}{5}+\dfrac{4}{3}+\dfrac{5}{5}+\dfrac{6}{4}+\dfrac{13}{9}+\dfrac{4}{2}+\dfrac{6}{5}+\dfrac{7}{3}}{13} \approx 1.2$.

Should we really compare references per page? There are good reasons to assume that the number of references per page should decrease as papers get longer, since the references cited in Section 1 won't normally be disjoint from the references cited in Section 2. As a consequence, the discrepancy between the above numbers looks even starker.

I can only explain this discrepancy in two ways:

  1. the literature has grown much larger, and not just by the addition of new disciplines but also by more people writing about the same discipline;

  2. (as the OP observed) finding relevant references in the literature has gotten easier thanks to the Internet, Google Scholar, etc.

I don't know how to properly disentangle these two causes.

  • 3
    Notice that today, too many papers are published (and most of them repeat a little bit the ideas of previously published papers by same authors) I heard that the average paper has today just 2 readers. In 1970 it was probably more. – Basile Starynkevitch Jun 25 at 11:25
  • 3
    @BasileStarynkevitch Does 2 readers include the reviewers? :-/ – gerrit Jun 25 at 14:48
  • 2
    This is something I forgot, but in the article mentioning that "too many papers" I think it was no.... If you have precise references, please give them here! But as a "semi-academic" (working at something claiming to be a research institution since 1985) my impression is that the productivity of researchers has decreased during my professional lifetime. We collectively spend more time on searching funding, communication, but less time on scientific research. And that might explain why scientific progress is decreasing w.r.t. 1980s – Basile Starynkevitch Jun 26 at 3:42
  • 1
    Likewise, I have the impression that the collective technical knowledge of Europe is decreasing ... if you know any references (e.g. academic papers in macro-economics or research policy), please share them with me basile@starynkevitch.net – Basile Starynkevitch Jun 26 at 3:45
  • 3
    It's more of a starting point. A proper analysis would begin by figuring out how the number of references should grow with the length of the paper (by comparing papers of different length from the same time), and then controlling for that confounder. Also my n's are a bit on the low side, and in hindsight it looks like the medians are more informative than the means here. – darij grinberg Jun 26 at 13:13
20

Although I haven't really known this time myself, I think it's important to mention the much more crucial role that conferences and journals used to play in the dissemination of specialized knowledge. A researcher would usually try to attend the conferences of their field and get their local library to subscribe to the relevant journals in order to keep up to date with what people are doing in their research community.

At the time researchers would preciously keep these massive conference proceedings whenever they attend a conference. I know a few senior researchers who still have their offices shelves full of old proceedings and journal issues. In order to keep up one had to follow the series of conferences/journals relevant to their field, but it was feasible because there were not as many publications (and publication venues) as nowadays. They didn't have to read the entire volume but at least they would read the most important publications and make a note of potentially relevant ones. To some extent they relied less on references indeed, and they relied more on the reputation of a journal or conference to cover the recent progress of the field.

The question of whether they worked harder is subjective. The nature of bibliographical work was different and certainly more time-consuming, but researchers were not expected to process more than what is humanly possible: overall the research system was proportionate to the technical constraints. Like for many things, I think that people used to make more out of fewer resources: exploiting the resources they had access to in a deeper and more extensive way, whereas nowadays we often can only afford to skim through the massive amount of literature... I could keep rambling about disposable research but that would be very subjective and out of topic ;)

  • 4
    I can remember the process of reading one paper in a journal, which would refer you to several others, which would do the same, until you'd built up a nice network of interlocking references about a subject. It was more time-consuming, but there was also a greater possibility of discovery -- the book or journal next to this one on the shelf might be useful even though it wasn't directly linked by reference. – Joe McMahon Jun 25 at 0:47
  • 1
    @Joe McMahon: This process of using references, references of references, references of references of references, etc. is so ingrained in me that it's hard for me to think of what else one would do, but I suppose in this day of google searching (which of course I do a lot of) it might not be the primary method anymore. Usually for a very specific topic, a few iterations of this leads to a few dozen references (takes a few hours in a library), and finding something relevant that "hardly anyone knows about" was difficult. Often, this happened a year or two later while looking for something else! – Dave L Renfro Jun 25 at 7:11
  • 1
    Certainly when I did my PhD (1972-5) I would read the relevant journals when they came out, as well as browsing previous issues. My supervisors would learn at conferences who was working in my field, and get people to send me advance copies of their publications. And then I would follow the citation trail. – Michael Kay Jun 26 at 15:08
13

It was a LOT harder before the internet. Time spent traveling to and from the information source is eliminated by instant transmission of data.

If people were lucky enough to be able to spend long stretches of time in a library, the amount of information they could find was, for those days, a "lot." There were these things called "card catalogs." They were furniture, basically, in which long, little drawers were kept, and the drawers were organized by subject, and inside each drawer were hundreds of little cards, and you'd find information about one individual book on each of these cards and determine if you wanted to go walk sometimes long distances and up elevators or stairs to "the stacks" (where the books were) to find this particular book or not. (Google Dewey Decimal System.)

You'd write down the ID info about all these books onto what would often become long lists. Then you'd go to another part of the library, where the books were ("the stacks") and pull books, many of them hardbound, off of shelves. Lots and lots of books sometimes. Yes, they'd actually read "the whole book"!!! (Couldn't help but laugh hard at that question.) And sometimes not. And they'd underline sections in pencil (and ink, unfortunately), and place books face down with pages open to pages they needed to look at as their hands went to and from the typewriter, and they'd put bookmarks in between pages in an effort to be able to go back to references and material they wanted to put into the project they were working on, as they attempted to create an effective structure without Select, Copy, Cut, Paste.

If they couldn't type, they'd write it out by hand on pads of paper, and cut paragraphs and sentences with scissors (I'm not making this up) out of one section so they could rearrange the sequence of their text by taping it and gluing it to other cut-up pieces. (Google the word "mucilage.") Before this stuff called Whiteout was invented (by a mom, apparently, with stuff in her kitchen), typists had no way to correct an error on a page; they'd have to start the entire page over again.

The invention of the IBM Selectric typewriter was revolutionary and a gift from heaven because you could correct backwardly about 12 spaces. You could check books out with a library card, and you had to return them usually within two weeks or pay a small fine. And if you didn't return the book, you had to pay for the book. (One of the many good things that broke down in the social decay of the last few decades was that people started stealing books from libraries, something nobody ever did before.)

Some libraries had little tiny rooms (e.g. New York City) that they'd let serious researchers use privately for months. And they could get "a lot" (by the standards of the day) done. But it was a small fraction of what you can get done now. Someday, when you're in your 60s or so, a young person is going to ask a question that will make you laugh because you can't believe things have changed so much that young people don't know what something used to be like. It was a lot, lot, lot harder to get a degree, build a business, make almost anything... than it is now. I mean a lot, lot harder.

  • 13
    But the upside is that in those days, it was much harder to find complete nonsense by some fraud who is not afraid to share everything he thinks he knows about something. – WGroleau Jun 25 at 6:23
  • 2
    No, the frauds were just given more credence. Old studies are blown out of the water every day that people in the 50s, 60s, and 70s took for granted. Don't give academia so much undeserved credit. – some_guy632 Jun 25 at 8:36
  • 2
    Time traveling (2nd sentence) unfortunately is still not working. – henning Jun 25 at 8:43
  • 5
    "people started stealing books from libraries, something nobody ever did before" [citation needed] – Stephan Kolassa Jun 25 at 8:48
  • 5
    Stealing from libraries or churches goes way way back - why do you think they chained books to the tables? Those books had a lot of value.... – Solar Mike Jun 25 at 19:07
7

To add to the other answers: Many publishers and professional societies produced bound paper indices of journal articles such as Math Reviews by the AMS.

6

I'm surprised that no one has yet mentioned preprints of research articles. At least in mathematics, it used to be when you published an article in a reputable journal that you were given a number of paper copies of your article that you could give to colleagues. These in turn were often photocopied. Almost every time I went to my advisor's office in grad school I would emerge with another paper (either one of his own preprints or a copy of a preprint that someone else had given to him). By the time I had finished my dissertation, I had amassed a collection of a couple of dozen papers which were directly related to my topic.

When you went to conferences, many people would bring with them preprints of their own papers. You might emerge from such a conference with a half dozen new preprints.

Nowadays I don't accumulate many paper copies of articles very often (though I might still print up one I download if it strikes me as particularly important). But over the years (roughly) 1988-2003, I must have accumulated over two hundred preprints (and copies of preprints) that were given to me by colleagues. Even now I have a file cabinet full of them.

TL/DR: Before the internet, networks of researchers would keep much of the relevant articles in circulation.

5

Does that mean that research before relied less on references?

Plural-of-anecdotally, I've referred to several mathematics papers from the 1940s and 1950s and it's quite common for a 20-page paper in that era to have between zero and two references in the bibliography. Even today, papers in pure mathematics don't tend to cite a whole lot but I'd say that something like ten to twenty citations would be the common range for a 20-page paper.

As for your title question, it was very common for researchers to not find articles before the Internet and the computer era. For example,

The Bellman–Ford algorithm [...] was first proposed by Alfonso Shimbel (1955), but is instead named after Richard Bellman and Lester Ford, Jr., who published it in 1958 and 1956, respectively. Edward F. Moore also published the same algorithm in 1957, and for this reason it is also sometimes called the Bellman–Ford–Moore algorithm. (Wikipedia)


The Floyd–Warshall algorithm [...] was published in its currently recognized form by Robert Floyd in 1962. However, it is essentially the same as algorithms previously published by Bernard Roy in 1959 and also by Stephen Warshall in 1962 [...] and is closely related to Kleene's algorithm (published in 1956)... (Wikipedia)


[Dijkstra] re-discovered the algorithm known as Prim's minimal spanning tree algorithm (known earlier to Jarník, and also rediscovered by Prim). Dijkstra published the algorithm in 1959, two years after Prim and 29 years after Jarník. (Wikipedia)

  • 1
    Nice post, making the same point as mine but providing complementing evidence. Now imagine our current way of scholarship from the vantage point of 2050 when at least mathematics will have a sort of semantic web... – darij grinberg Jun 25 at 16:42
  • 1
    Additional anecdote: In 1971 I needed an algorithm for rasterizing a straight line between two points. Bresenham had already developed this in 1962, but presented it at the ACM national convention in1963, a year in which no proceedings were published. IBM picked it up in 1965 in its in-house Systems Journal, but neither the local uni nor my alma mater subscribed to this. Technically it was published but I just couldn't find it. - I was compelled to develop the algorithm independently. I would have been embarrassed if I had tried to publish. – A. I. Breveleri Jun 26 at 13:30
  • 1
    I suspect reinvention and rediscovery are just as common today as they ever were. Google only helps if you know the right terminology, and in abstract fields like maths and computer science, the same concepts can go by different names. Indeed sometimes an "isomorphism" is discovered that shows that two things previously thought to be completely different are actually fundamentally the same underneath. – Michael Kay Jun 26 at 15:16
  • 3
    Another anecdote: Wilkes and Needham implemented one-way password hashing on EDSAC, but thought the idea so obvious that it only got a footnote in Wilkes's book Time-sharing computer systems. A lot of subsequent researchers missed that. – Michael Kay Jun 26 at 15:21
  • 2
    @DavidRicherby That's true once the field is well-established. But if the algorithm was first discovered in the 1890s by someone responsible for scheduling the routes used by postal delivery workers, and was expressed entirely in terms of geography and street addresses, then it's quite likely that a computer scientist would miss it, google or no google – Michael Kay Jun 26 at 15:26
4

Another resource was memory, the one inside your head. We would remember a bit of fact we read n years ago, remember what we were reading n years ago, and start there.

4

Chem Abstracts worked well. Look at the volumes (physical, take down from shelf). Then go to the bound journals (in volumes). See if article is good...if so xox it and stick it in a file folder.

Trace footnotes back (look at articles, copy relevant ones, sometimes going couple layers back...you find quickly the seminal reviews). This is even same process done now.

If uni doesn't hold something, send in an ILL request. (Librarian hunts it down in other libraries.) Articles would often be faxed. Books mailed.

Books tend to be dated and less useful than review articles. But still, they are much quicker to skim (just physically to do so...and you have the TOC and index). Often looking at what is physically next on shelf is useful.

P.s. Now that Google is here, I'm actually amazed how BAD many kids are at looking anything up on Google. Very little "fu". I mean there are wizards also. But very frequently, people who can't even do the basics. See it all the time in classes, work, forums, etc. Q&A sites an issue too. (Nothing wrong with asking for help after making an effort. But those to lazy to Google are a plague that drive discussion quality down and take community for granted.)

4

As a math grad student in the 60's I thumbed through the orange Mathematical Reviews when it came to the library every month. I could read a paper there, or write the author a postcard asking for a reprint. I still have lots (including reprints of my own papers) but rarely use them because it's easier to find a pdf on the net than my hard copy.

Nowadays I often don't even have to know where to go for the literature. A query on stackexchange returns proofs and references.

4

"In the old days", I think there was much more emphasis on, and valuing of, scholarship, in the sense that there were (usually older...) people who had good memories, and had paid attention, and knew of many things. Published and preprint-only. I myself was struck by this possibility most powerfully as a grad student at Princeton long before computer look-up was possible: many of the math faculty were aware of an amazing terrain of current and older work.

In some regards, this was very important there, because very many of the most important books and journal volumes were checked out, all the time... and grad students had very low "recall" priority. So to be able to find out what was in those unavailable sources, informally, very quickly, by talking to scholars, was excellent.

Unsurprisingly, the high-level "pre-processing" done by very-capable scholars, as opposed to search engines, allowed many huge speed-ups in searches... and I think still does so.

And, with live scholars, as opposed to current fairly-generic search engines, one of the fundamental difficulties... not knowing the keywords... was easily overcome.

Indeed, from what I can see, quite a lot of the "research" posted on arXiv has failed to connect with much prior research, visibly due to failing to connect to the proper key-word (or author) world. Understandable, but "computers" do not magically solve that problem.

One of the biggest changes is simply the possibility of typing things up oneself, in presentable form, and easy error-correction. This was a significant bottle-neck as late as the late 1980s. The even bigger change, unimaginable in 1985, was the possibility of "publishing" by simply putting things on-line. "Organizing" this, if we pretend that's what we're doing, to any degree, seems to require new ideas or concepts, that are not yet "here". E.g., for math, arXiv is very useful, but: lots of pointless stuff, and many things don't appear there. (I periodically check the web pages of a list of people to see what they're doing... in addition to looking at arXiv daily.)

But, yes, "in the old days", it was important to "be on the inside", and/or be connected to people who were, etc., to get the "preprints" which would not have been (could not have been) publicly available for a year or two (due to publication lag in math... which sometimes was infinite).

And, as other answers have noted, in those days, in math, conferences really were the places where new things were announced... that might not appear in "publications" for a year or two, if even that!

We can also mention the vagaries of "physical mail", especially transatlantic... :) Things could take a month or so, or never show up at all. And, too, long-distance phone calls in the U.S. were pretty expensive. People tried to arrange to charge their grants... :)

One summary of the state of things was that there may have been a more coherent common body of "known" things, that specialists/experts all were aware of... and novices were trained-into that scholarship.

3

Most of the methods used have already been pointed to. I'll add that Index Medicus was de rigueur for life sciences, as it doesn't look like anyone has pointed that out.

Oddly, I find that having to use the Scientific Citation Index taught me an understanding of how important tracking citation of seminal work can be in a way that graduate students don't really seem to understand.

The biggest difference I find between now and then is that the immediate ability to know how many times a paper has been cited without having to page through volumes and volumes of the SCI means that you immediately know what the seminal papers in a field are. It makes it much, much, much easier to bring oneself up to a workable level of knowledge in an area that isn't your own.

Of course, the other difference is the hours in front of the copy machine in dark library stacks.

On a separate point, another factor that might influence the number of citations in a paper over time is how EASY it is these days to pop a reference into a paper, with the advent of powerful reference managers that interface seamlessly with text and word processors. Think about adding that last minute reference when it meant retyping a whole document, changing all your superscripted numbers!!!

protected by eykanal Jun 27 at 19:50

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.