17

I am working on a master's thesis in natural language processing wherein I analyse various tokenisation algorithms. These algorithms keep a large fixed-size set of recognised words (more specifically, word pieces) in storage, on the order of tens to hundreds of thousands. Along the way of my analysis, I noticed in one of the non-English tokenisers I am studying that it had quite a few word pieces which could only be completed to form English pornographic terms (let's say ussy, inces, cking and so on). The reason would be that they were trained over web corpora that contain such webpages.

There exist lists of swearwords online that contain some of these words. However, there are also a bunch of words that qualify as pornographic genres (maid, POV...) that aren't inherently pornographic, were it not for the fact that they are English words appearing online in a corpus that largely consists of a different language and that has the obviously pornographic words already present in it. As further evidence, the names of some porn websites also occur in the tokeniser's vocabulary.

I find myself in a strange and slightly blush-making predicament, now: not afraid of the internet, the only place I could actually find a comprehensive list of pornographic genres and website names was, well, on websites that also host videos of those genres. Since this one experiment fits very nicely with the rest of the thesis and I find the results significant and interesting, I would like to feature it.

Proper academic methodology requires that my experiments be reproducible... but how do I cite my sources in the most appropriate way possible? I have only found one paper that speaks on this topic, which obfuscates their sourcing:

We filter out documents that have at least three types or ten tokens from a list of words highly used in pornography. The list was derived from the analysis of pornographic pages harvested in a previous crawl.

The lists of swearwords are all Node.js packages, so citing those is no problem. For the rest of the list, concretely, how does one cite porn.com in a scientific work, as appropriately as possible? Hyperlinking to it seems like a bad idea, and showing the URL to me seems "tasteless", although not showing the URL feels too handwavy and might raise even more eyebrows ("I use a list compiled from a variety of websites known to me").


This is at a Western-European university, so there won't be a moral inquisition if I include these results, but that doesn't mean it's not something to be nonchalant about. I want to respectful to the reader but academically precise.

13
  • 10
    "(let's say ussy, inces, cking and so on). The reason would be that they were trained over web corpora that contain such webpages." Hmm, I happen to know a French environmental activist and Star Wars fan who lives in Ussy, seriously dislikes fracking and seriously likes princess Leia... Ok, that's not quiet true, but maybe the creator of the dataset parsed a newspaper article about my imaginary friend. (Sorry, just couldn't resist. ;-) ) May 25 at 19:22
  • 9
    fussy, winces, checking, lacking and so on. May 25 at 19:26
  • 7
    @JochenGlueck I don't know, sounds a bit sussy. May 26 at 3:23
  • 4
    Some colleagues of mine do research on web security. They routinely scrape and refer to porn or darknet sites. I don't think they have ever seen this as problematic.
    – xLeitix
    May 26 at 9:25
  • 4
    @EthanBolker another example (and a known old problem).
    – Trang Oul
    May 26 at 14:07

6 Answers 6

26

Citation is not the same thing as describing your methods. It sounds like you need to describe your research methods, not use citations. Those research methods may include listing URLs that you accessed to gather data.

You will want to make sure that your methods are not against the terms of service of any of those sites you used. Just because you are able to view certain content does not mean you can do whatever you want with it; some uses may fall under the concept of "fair use" or similar depending on what legal jurisdictions apply to you.

I'm not familiar with your field, but I've certainly seen popular press that makes reference to well-known pornographic websites, and they do not bother to obscure anything about their names. These are big businesses, and you'll find them referenced in newspapers and news television.

I think there is a certain level of advice that I want to give along the lines of "grow up and get over it". If your research is going to involve pornography, you're going to have to get a lot more comfortable with your research involving pornography. Urologists, gynecologists, and obstetricians can't get all giggly every time someone refers to genitals, they work there.

If you have a reasonable research question that involves pornographic language and you've used pornographic websites in work towards those questions, you need to be sufficiently explicit about it to allow someone else to recapitulate your work. That's the standard in research. Long lists might make more sense in an appendix or similar data supplement, but if you got your list of naughty words from "porn.stackexchange.com", and you think your work is rigorous enough to be published, then mentioning that your words came from "porn.stackexchange.com" should be just fine. You can try searching Google Scholar even for the specific websites you have in mind to see how others have referred to them.

4
  • 1
    I do want to note that this is only one aspect of a dozen-or-so hypotheses about tokenisers that the thesis covers; it was really an accident that this one became about pornography. If I would have noticed an overrepresentation of words that could have only originated in Star Wars scripts, then I would have investigated that instead. The field is, very generally, about parsing any text from any domain, which is rarely the websites in question.
    – Mew
    May 25 at 20:29
  • 1
    @Mew Sure, but at least the impression I got from your post, possibly not the correct one, was that at some point you felt it was necessary to find a comprehensive list of pornographic genres and that this was important for some aspect of your research. If it wasn't important, why do it? If you want to study language in specific contexts that exclude pornography, why use a corpus that includes them? May 25 at 20:47
  • 2
    Yes, your impression is correct, I did need such a list. All I'm saying is that I'm not writing a thesis about pornography, so the comparison to e.g. urologists gives undue weight to it. The thesis is about tokenisation -- splitting words into pieces -- and one section of it is "overrepresentation of certain parts of the internet corpus people have used to train these machines in the past". In a parallel universe, I might have asked for the best way to cite a list of medical terms instead (and the thesis wouldn't have been in medicine).
    – Mew
    May 25 at 23:25
  • Urologists sometimes work there--they are concerned with the whole plumbing chain, not merely the emitter. May 28 at 21:13
13

There is no room for bashfulness in research

As the other answers here point out: (a) if you require information for your research that can only come from pornographic websites, then you are going to have to source that information from pornographic websites; and (b) if you source information from pornographic websites then yes, you either need to cite those websites or at least record their use as relevant sources in the information in your methodology. In some cases where you are sourcing large numbers of individual webpages you might just cite the general sites themselves and have supplementary materials setting out all the specific links, etc., for the data sources (as opposed to listing them in the bibliography). In any case, the rule is that your reader should be able to track your data back to its original source to verify the accuracy of your work.

Beyond the academic requirement to show your sources, good data science practice is to save your data/sources locally if possible (within reason) and have a good system for tracking your data back to its original source. This might even mean that you would be well advised to save some pornographic pages or search outcomes in your local data folders to ensure that you can demonstrate the source of the raw data (in cases those pages change or become unavailable later).

Here I think you have actually identified an opportunity for some useful additional research that could form a separate paper. The predicament you describe seems to be one that many researchers examining language models would run into, and if you are correct that there is a sparsity of existing research on pornographic terms then you have a golden opportunity to be the one to do that research and write a useful paper with supplementary materials giving a list of terms that other researchers could use in the same situation you are facing. If you were to do this then you would need to wade through some pornographic pages for the initial project, but you would save later researchers the trouble of doing this. It would also mean that your existing language-modelling paper (and any other language-model papers in the future) could directly cite your other paper on the analysis of pornographic terms without having to cite any pornographic websites itself.

2
  • Good suggestions. I hadn't thought of saving these particular sources locally (but that folder will be a conversation-starter for sure!). The result could indeed fit in an ACL shortpaper, but considering this thread has been viewed >2500 times already, the odds of somebody plagiarising the idea are now non-zero.
    – Mew
    May 26 at 23:16
  • 3
    ideas cannot be plagiarized, texts and work can. Don't overestimate the size of your research field and don't underestimate the head start you have because of your existing work. May 27 at 13:55
7

The basic rule in research is that if you use it, you cite it.

So, if you use data taken from a site, you should, indeed, cite that site.

I'll guess, however, that the basic structure of your research (vulgar words) will lead people to expect that you didn't find them in "proper" places. And, those places/people that would object to such a citation will object to your work in any case, unless you avoid using those words, even if obfuscated, in publications.

And, you are using such sites not for pornographic purpose, but for analysis of language. Those sites aren't secret from the public in any case.

2
  • Do you have suggestions as to the format of the citation? As I raised in my question, it seems bad taste to hyperlink the URL, for example, lest someone actually click on it. An inline URL? A footnote without further explanation or NSFW warning? A BibTeX citation?
    – Mew
    May 25 at 19:03
  • 3
    Not really. An active link certainly isn't required, but it needs to be specific. And any online resources need to list the date of last access, since things online change.
    – Buffy
    May 25 at 19:14
5

Not claiming that this answer is canon or anything, but, if it were me, I would go for a solution that allows me to cite/link to the resources (for verification/reproducibility purposes), without necessarily including any of the vulgar content to the paper directly.

In other words, something along the lines of what you see in websites/tv shows when you try to enter and they warn you of potentially offensive content, e.g. something like this at the end of your bibliography:

[28-35] Please note: some of the resources used for this research unavoidably contain content that may offend readers, therefore citations to these resources are provided as a separate bibliography section. Readers wishing to access this for verification and reproducibility purposes can do so here: http://link.to.some.textfile.online

(where 28-35 refers to the citation number in the text, and where you make sure to lump all the 'risky' citations under those numbers at the end of your bibliography)

Furthermore, you could probably consult the journal editor on this issue; they may have a policy on this already, or advise you of their preferred solution (e.g. a particular repository to upload your bibliography section to).

2

Don't

If I was doing a computer analysis of language from novels from the 1850s I would not include the list of titles in the paper itself. I would include it under supplemental material.

Since you are doing a crawl, I presume you are using a large number of websites. If there is a large list of websites then it is appropriate to not name them in the paper itself, but to just describe the websites, and include the list of domains or whatever in the supplementary material.

3
  • 2
    I'm not doing the crawl. Someone else did a crawl, then trained a model over the resulting corpus, and this model is the standard of its kind for its language.
    – Mew
    May 26 at 23:09
  • Then I'd just direct the reader back to the someone else for the details. "We use XXX corpus gathered and used in YYY from a web crawl of pornographic web pages. For details see the supplementary material in that paper."
    – Daron
    May 27 at 9:47
  • 1
    But I'm not the one using the corpus. Imagine you are using a language tool like ChatGPT, about which you are told that it was trained on general German text. Does it speak English? Not inherently, but you notice that any time you ask it to produce an English answer, it strings together a bunch of pornographic words each time. You hypothesise that this is due to an overrepresentation in the training data, but all you have access to are the internals of the model (and even if you have access to the training data, you want to know the fraction of the model focused on these data). That's me.
    – Mew
    May 27 at 11:18
1

Why not? I work in econometrics. Causal inference, specifically. There are two quasi-experimental papers I've wanted to do that directly or indirectly involve PornHub. So long as your work is academic, then this shouldn't matter if the question matters. Moreover, we're all adults. So, it's okay.

2
  • Some people earned their PhD at age 17 or younger. See this. Your assumption "we're all adults" may not be valid. I do agree with this answer, though. +1
    – Nobody
    May 29 at 14:37
  • Yep they did. I know it's invalid. I wanted to add this qualification. So, I guess 99.9% of us are adults. But yeah, either way, even if we're not all grown, we all are expected to be mature. May 29 at 20:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .