Health sciences researcher out of my area of expertise. I have an assistant calling businesses at random, using a publicly available registry, to ask about their automated external defibrillator (AED). We are sampling without replacement.

My question is, what happens methodologically if we call one of the registered AEDs (selected by random) and the person/informant can give us info on multiple AEDs on the registry that have not been selected as part of our random sample? We're in this position with a large business who has 10+ AEDs, all of which are registered, but our randomization only selected one of the 10+ devices. It seems crazy not to get info on all the devices to me.

Can/should we include these devices that were not randomly selected? I'm having trouble understanding the implications of sampling these and including them in our analysis. Including these other devices seems like snowball sampling instead of just simple randomized sampling.

Any help would be extremely appreciated. Thank you!!!

closed as off-topic by scaaahu, Scientist, Daniel R. Collins, Enthusiastic Engineer, user3209815 Jul 23 '18 at 6:42

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.


This isn't my field, but statistically speaking, you will possibly invalidate your study if you change the parameters mid stream.

However, you can gather the extraneous data, since it is cheap to do so. But isolate it and use it only as a guide to future work and, perhaps, to give yourself confidence (not statistical confidence) in the outcomes.

If this is dissertation work, then knowing what you could have done differently is worth exploring and maybe including. But let the experiment you designed be carried out faithfully. Otherwise, at best, you will just confuse yourself with the results.


Good news. The approach you are describing is a named sampling technique: cluster sampling (technically, cluster sampling with probability proportional to size). The bad (?) news is you will likely need to consult with a statistician to see if your study exactly conforms to cluster sampling.


You can keep the data, you just need to change your analysis a bit. I imagine you have something like a large spreadsheet, where you enter your results per device. But I assume that the results you want out could be something "per location", that is, how much is the device used in different types of places, wear and tear etc. If you now include additional devices form one location, the randomness in your selection goes out the window, as you, in your survey, have a larger representation of locations with many devices.

But you would be crazy not to use the extra data now that it is there. Therefore, change your spreadsheet to be per location, and count everything as an average with an associated error instead of just a number. When you then do fitting etc. you just use the error as an associated weight.


I agree with @Buffy that the best option is to remain faithful to the sampling plan you made initially. For your own benefit, you can record that data and see how it might change the analysis, but it probably should not be part of the formal results.

The reason this is not snowball sampling is that it sounds like one of the companies volunteered information you weren't expecting. Something like snowball sampling requires you to attempt to get that extra information out of every company you call. Even if you were executing pre-planned snowball sampling, there can be serious bias in the data which you would avoid by using the simple random sampling.

Edit: The above answer is based on the assumption that these extra devices belonged to other companies that you could have contacted using other phone numbers on your list (i.e. you called company A and they offered information about company B). If the issue is that one company had many different devices (i.e. there are not other phone numbers on the list which correspond to those devices), then @nabla is correct to aggregate the devices for each phone number/company.


Snowball sampling involves the potential for arbitrarily long reference chains (I contact Alice, who gets me in touch with Bob, who gets me in touch with Chris...) so this isn't that.

If you've already started sampling, changing methodology mid-sample is probably a bad idea. File this under things to think about next time you design a sample, and please remember to talk to a statistician before running a sample.

If you haven't started sampling, then yes, you absolutely should collect this extra info, but you need to factor selection probabilities into how you analyse the data. Full discussion is beyond what can reasonably fit in a SE post, but the basic idea is that you need to know the selection probability for each unit (e.g. AED) and then weight accordingly to these selection probabilities.

For example, let's say my sampling frame encompasses 1000 small businesses which each own one AED, and 100 large businesses which each own 10 AEDs. (Total: 2000 AEDs, 1000 owned by small businesses and 1000 by large businesses.)

Let's assume that you're trying to estimate what proportion of AEDs have been used in the last year, and let's suppose that there's a systematic difference between small and large businesses: all of the large-business AEDs are used every year, and none of the small-business ones.

(Obviously this is unrealistic, but an exaggerated example makes the issues easier to see.)

Now suppose I run my survey by randomly selecting an AED from my list of 2000, ringing up the business that owns it, and asking about all their AEDs. Since 50% of the AEDs are owned by small businesses and 50% by large businesses, every random selection has a 50% chance of getting me info about one small-business AED, and a 50% chance of getting me info about ten large-business AEDs.

If I do this 100 times, I can expect to end up getting data on about 50 small-business AEDs and 500 large-business AEDs. Even though only 50% of the AEDs out there have been used in the last year, 500/550 in my sample have been used, so a naïve analysis will grossly overestimate usage of AEDs.

The solution to this is to weight by selection probability. In this case, each small-business AED has a 1/2000 chance of showing up on each 'draw', and each large-business AED has a 1/200 chance of showing up. Hence, when I'm analysing my data, I should weight the small-business AEDs 10x as heavily as the large-business AEDs.

If you're trying to estimate things like standard errors from your data, then it gets a bit more complex, and you really ought to talk to your friendly neighbourhood statistician.

Not the answer you're looking for? Browse other questions tagged or ask your own question.