2

I am investigating a complaint that comes up on social media "often" for a particular institution. The complaint is about a particular type of bad interaction with the institution. I could not find any research that said how often this interaction occurs, and because I have access to administrative data, I decided to do a quick analysis showing how frequently this interaction occurs. In writing up the manuscript, I want to say something that amounts to "this is a common complaint" in order to motivate the topic.

The issue is that I cannot quantify how often this interaction is complained about. I have many reddit/facebook/twitter posts that I can link and cite, but this seems arbitrary and a bit odd to be citing a series of posts. I certainly do not have the capacity to find the relative volume of these reddit/facebook/twitter posts. This is similar to this question, but rather than "word on the street" it is a thing that happens to people with unknown frequency (hence this analysis) and some proportion of them discuss it online.

9
  • Have you asked for a librarian's help?
    – Bryan Krause
    Commented Dec 21, 2023 at 22:29
  • 1
    I'm on the fence about whether this is off-topic as "content of research," but I suppose I would be convinced enough to hear something like "we found X posts from this school year using the search term 'foo bar'" in the motivation. Commented Dec 21, 2023 at 22:42
  • @AzorAhai-him- I suppose that's fair, but that's a huge task that would have to be manually done for what is supposed to be one line of text. It's also not possible to get the relative volume of posts, unless you work at the social media company.
    – Liam
    Commented Dec 21, 2023 at 22:51
  • When twitter was still twitter you could easily get this information using their API. But since the last acquisition and rebranding it became very expensive to get this kind of data from X. Maybe you will have more lucky with Reddit and other social media.
    – The Doctor
    Commented Dec 21, 2023 at 22:54
  • @TheDoctor That's true, but unfortunately a lot of it is facebook where this was never really possible.
    – Liam
    Commented Dec 21, 2023 at 22:57

1 Answer 1

2

This is something you can do, but you should be as systematic as you can

In principle, there is nothing wrong with doing research that cites evidence from social media or administrative sources in order to give a conclusion. But as with any other type of research work, you need to be systematic in how you collect and collate your data (and make it available to your reader if possible) and you should describe your sampling method, results and conclusion, adequately hedged by any uncertainty that arises. We typically want to avoid bald assertions of "word on the street" which are unbacked by any kind of empirical evidence. Sometimes it is only possible to get empirical evidence through a sampling process that is not ideal, but that is still better than nothing, and it is a reasonable approach if there is no better alternative. Here is what I recommend you do in this case:

  • Step 1 (Determine an appropriate sampling scheme for data collection): You say that you have found many posts on Reddit, Twitter and Facebook that contain the complaint at issue. In view of this, you should take some time to consider whether there is any mechanism by which you could systematically sample from these sources in a way that would identify this data, but also identify all the null cases of posts that do not contain the complaint at issue. Ideally, you will be able to figure out a search method that has sufficient coverage to obtain all posts relevant to the university and you will be able to go through them and identify which posts make this complaint, which posts make a different complaint, and which posts are null cases that make no complaint. If you can't do this, think about the second-best option, third-best option, etc. Once you have considered this, settle on your sampling method and write it up clearly.

  • Step 2 (Collect and compile your data): Using the chosen sampling method, collect all the data you are able to get by saving all the relevant social media posts and administrative records at issue and viewing them to categorise them and extract relevant data. Create a proper data-frame containing the relevant information from these posts (one row of data for each post with variables for date, time, poster, link to post, etc.). (Best to save all the posts locally in addition to saving links, so that you have a record that is not dependent on an outside source.) Unless there are privacy issues preventing this, your data should be made available to the reader, either as supplementary material or in an appendix to your paper.

  • Step 3 (Describe your sampling method and data): In your post you have noted the drawback that this data is only available in a way that makes it impossible to systematically determine its magnitude relative to other like things, which detracts from your ability to say that this type of complaint occurs "often" in an objective sense. What you have described is anecdotal data that can be collected via searches of administrative data and social media posts using various forms of "convenience sampling". Instead of just asserting that this type of complaint occurs "often", instead describe your sampling method to your reader, describe your data more specifically, and then give your conclusion as an inference you are drawing from that data. Link to your supplementary data/appendix so that the readers can see the data for themselves.

  • Step 4 (Use your data to make a conclusion, hedged by uncertainty): Once you have described your sampling method and the results in the data, you can use these to form a conclusion. You might be confident that it demonstrates that this complaint occurs "often" and you should feel free to make this conclusion, but you must ensure that the reader understands how you made that inference. Because of the nature of the sampling method, you should also draw the reader's attention to potential problems that might call that conclusion into question or make it uncertain.


Here is an example of what this might look like in your paper when completed.

The present author observed anecdotal evidence of social media posts complaining about flatulent staff members at the university, which led to the hypothesis that this form of complaint occurs often in external social media commentary on the university. To examine this, we conducted a search of social media posts on Reddit, Twitter and Facebook. Our search methodology involved running search queries to identify posts mentioning the university and examining the top 500 search results on each platform based on selected keywords. Our keywords were selected to identify posts pertaining to feedback about the university, but so as to be neutral/unbiased as to positive or negative feedback and to be neutral/unbiased as to the category of any complaint/adverse feedback in the resulting posts. As designed, our search resulted in a total of 1,500 social media posts across the three platforms we used (further details of search method shown in Section 6.2 below). We examined each post to determine whether or not it raised a complaint or adverse feedback about the university, and if so, we categorised the nature of the complaint/adverse feedback.

Full data for our analysis is available in Appendix B.2. In Figure 6.4 below we summarise this data in a bar-plot showing the counts of complaints/adverse feedback across complaint categories on the three platforms. As can be seen from the figure, there was a relatively high count of complaints about flatulent staff; this category of complaint was the highest on Reddit and Facebook and the second-highest on Twitter (coming after complaints about wild monkeys living in the chemistry labs). We also examined time-series data of the occurrence of complaints, shown in Figure B.2.1 in Appendix B.2. This showed a pronounced spike in complaints about flatulent staff coinciding with the great Baked-Bean-eating contest held each April at the university.

[Figure 6.4 - Barplot of social media post data]

Based on this data, we infer that complaints about flatulent staff in the university occur often and are a major source of adverse social media feedback from former students and visitors to the university. As a caveat to this finding, we note that our analysis was confined only to the three identified social media platforms, and the sampling method (which used a "first-500 posts" methodology) ignored any social media posts that did not show up in the top set of queries; this may have been heavily affected by our specific search keywords and the search-ranking algorithms for the relevant sites. Notwithstanding these limitations, we think that the search-ranking is likely to be representative of the popularity of the relevant social media posts and so we have reasonable confidence that the identified complaint is the most common complaint about the university. The interested reader can scrutinise our method by examining our sampling methodology in Section 6.2 and the data in Appendix B.2.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .