Is it acceptable to heavily reference blogs, forums, and other informal media?

Question

I am pursuing a specific research question. I have thoroughly surveyed existing research on the topic, and found dozens of researchers working on the problem, but was disappointed by their work. Their research:

• Does not consider the full magnitude of the problem.
• Constantly tests old, insufficient methods.
• Overlooks significant details, so the test results are meaningless.
• Lacks innovation.

I found lots of interesting ideas posted around the Internet. In blogs, forums, and USENET, I found people with some clever new ideas to approach the problem. These people had a genuine stake in the problem, so I found their ideas actually brought the problem somewhere meaningful. These informally-posted ideas need testing and considerable refinement. They are far from perfect, but many times better than what the academics are dealing with.

I would like to prepare some trials and publish some papers, centered around a number of these ideas. It is only fair that I give credit to the authors of those ideas. Essentially, I need to give credit to lots of anonymous people who posted their ideas informally. I have never read an academic paper containing highly informal references. Can I include references like this in my paper?

MutantTurtle17. “My Amazing DIY Tin-can Refugee Shelter.” MyBlog. 2014.
    Retrieved from http://...

SimCityFan2012. “RE: RE: Look at this!” Shelter Designs Forum. 2013.
    Retrieved from http://...

Answer 1

Does your paper really test & verify lots of ideas?

You state that "I would like to prepare some trials and publish some papers, centered around a number of these ideas." How many of those ideas do you expect to actually implement per a single paper?

If you implement, evaluate and contrast three novel ideas from blogs&forums, preferably including a solid comparison against a baseline published method; then that's just three informal items that you need to cite, in addition to the current academic publications.

There are references and references

If your paper assumes something, or claims non-obvious things, then you need 'proof' of it outside of your paper that should come from references. Those references need to be trustworthy - preferably respectable peer reviewed publications.

However, if your paper uses references for giving credit to ideas or pointing to original sources, then that's an entirely different class of reference, where blogs and forums are just as acceptable as, say, referencing archives of private informal letters that are used in studies of literature or history.

If you have never read an academic paper with a lot of informal references, then it is because it is very dependent on the field you're studying - for example, a thesis about racial stereotypes in online media would reference many informal sources as examples; while a thesis about particle physics wouldn't have any.

Answer 2

This is an actual problem that I also struggle with in some aspects of my research. There are problems in which the blogging and industrial world is, sadly, miles ahead of the scientific state of the art. However, citing an abundance of blogs and other non-reviewed resources is rarely a good idea. A few citations of web resources are usually ok, though. Hence, my (imperfect) solution to the problem currently is to cite the 2, 3 web resources that are best suited for my paper, and try to find academic resources that cover the rest of the ground as good as possible.

That being said, this situation is certainly a possibility for you. If you can take the ideas from these forums and blogs, and bring them on a sound scientific basis (e.g., through user studies or formal analysis, whatever is appropriate for your research) and publish it both scientifically and informally (e.g., in your own blog), there is a good chance that you make a strong impact on both the scientific side and the blogging community. At the end of the day, people tend to remember not only who originally threw a revolutionary idea or concept out there, but also (sometimes even more so) the person that made the revolutionary idea work (or, at least, clearly showed that it works).

Answer 3

Maybe one way of dealing with having lots of informal references would be to divide the bibliography into sections, so that the reader can easily see the different types of reference (acknowledgement versus justification).

Or alternatively maybe put them all in an extended acknowledgements section (since databases won't be able to do much with blog citations anyway, perhaps it wouldn't matter so much if they don't appear in the official bibliography, provided the reader is sufficiently informed of who came up with what).

Answer 4

If the post knowingly belongs to some well known researcher or otherwise a known, notable person, such post can be cited, because even "personal communication" at the end can be a reference. However it is not good as a proof that something questionable is true as this is not a peer reviewed article.

If the author of the post is anonymous or not a scientist, such source is not trustworthy and is only suitable as a raw input data for analysis in social research.

Answer 5

I am a mathematician, former editor of several journals (currently, just one). Suppose I were to receive a paper submission with an accompanying email saying something along the lines:

I have thoroughly surveyed existing research on the topic, and found dozens of researchers working on the problem, but was disappointed by their work. Their research:

Does not consider the full magnitude of the problem. Constantly tests old, insufficient methods. Overlooks significant details, so the test results are meaningless. Lacks innovation.

I found lots of interesting ideas posted around the Internet. In blogs, forums, and USENET, I found people with some clever new ideas to approach the problem. These people had a genuine stake in the problem, so I found their ideas actually brought the problem somewhere meaningful. These informally-posted ideas need testing and considerable refinement. They are far from perfect, but many times better than what the academics are dealing with.

Or/and, checking the bibliography list, I see many references to Reddit, Wikipedia, Quora, blogs by people I never heard of...

My crank-meter would go to something like 99% and my first reaction would be:

Should I even bother soliciting a quick opinion (let alone a referee report) on this paper? Or should I check if the supposed solution of the 4-dimensional smooth Poincare Conjecture uses "simply-connected" instead of "homotopy-equivalent to the 4-sphere?"

Answer 6

This is just a personal opinion, but let's run it up the flagpole and see who salutes.

When deciding whether to use (and therefore cite) information from a particular source, ask yourself: who has checked that the evidence and arguments presented in this source really support the conclusions this source reaches?

For a peer-reviewed paper in a journal or conference proceedings, the answer is "a couple of independent experts in the relevant field, in a formal procedure in which they're fully focused on the task". That's great, you can go ahead and use (and therefore cite) the information with only a pretty cursory check of plausibility on your own part.

For a Wikipedia article with many independent authors and a lively talk page, the answer is "a large community of Wikipedia users, some of whom are experts in the relevant field and some are not, who may not be fully focused on the task in a formal process, but who at least have a clear pathway to correcting any errors they discover". Again, this is pretty good - maybe you need to be slightly more careful in your plausibility check than you would with a peer-reviewed paper, but you're still good to use and cite.

For a book that's been through many editions, the answer is "a large community of readers, some of whom are experts in the relevant field and some are not, but who don't have any particularly clear or reliable pathway to correcting any errors they discover". Before you use (and therefore cite) this, you're going to have to do a bit of work checking the evidence and arguments really support the conclusions yourself.

For a book in its first edition, or for a newspaper or magazine article, the answer may be "a single editor who was more concerned with style and grammar and spelling than with the substantive validity of the arguments". Before you use (and therefore cite) this, you're going to have to quite a lot of the work of checking the evidence and arguments really support the conclusions yourself.

For a Wikipedia article with a single author and a moribund talk page, or for a blog or forum post, or for an ordinary web-page, the answer may be "no-one". You can still use (and therefore cite) it, but you're going to have to do all the work of checking the evidence and arguments really support the conclusions yourself, and put enough details in your manuscript/assignment to convince the referees/examiners who are evaluating your manuscript/assignment that you've done that work.

(In all cases, part of that work of checking might be done by investigating whether multiple independent sources reach the same conclusion.)