1

Google Research recently published a paper, Pushback: Characterizing and Detecting Negative Interpersonal Interactions in Code Review and in the Method section 2.2 Interviews about Code Review Pushback, the paper details it's method for selecting the 14 developers used in the interview portion of the research.

It states:

Participants were recruited in two samples by randomly inviting developers from the company’s human resources database. Although Google has developers in multiple countries, all interviewees were based in US offices. The first sample was developers who replied to the recruitment screener (𝑛 = 7), consisting predominantly of white, male, and senior developers. As this uniform sample might not include experiences more common to less tenured developers or those from underrepresented backgrounds, we rewrote our recruitment materials to be more inclusive and adjusted our sampling script. Consequently, the second sample (𝑛 = 7) included more junior developers, included racial/ethnic diversity, and consisted entirely of women.

Unfortunately, the study did not include any more information about what adjustments were made to the recruiting materials.

I am troubled by this methodology for many reasons and am unsure of how to feel about it. I would like to cite this paper in my own research, but I'm dubious of it's credibility.

Here are my concerns / questions:

  • If Google's demographic comprises of majority white, male, senior devs, wouldn't that mean a random sample would be representative of the group?
  • What is meant by underrepresented backgrounds? Google makes no mention of what this means. I genuinely am not clear.
  • Wouldn't a proper research paper explain in detail what changes were made to the recruitment materials to make them more inclusive and most importantly, what adjustments were made from the sampling script.

The way the paper reads, it implies that Google adjusted their selection script to exclude white senior males. Nevertheless, it would be impossible to replicate this study without these details.

Please help understand a context or view-point and the validity of the aforementioned research paper and concerns so that I can make an informed decision on how much, if any, of this paper I can use for my academic research project.

Thank you.

  • 1
    Unfortunately, the contents and methods of research are off-topic here. – Azor Ahai Apr 17 at 3:18
  • 2
    Oversampling or stratified sampling is a common and well-established technique in survey or interview research. – ff524 Apr 17 at 3:30
  • Use it if any bias makes no difference to your study, if it makes a difference then don’t use it and find another paper. – Solar Mike Apr 17 at 6:46