Correct name for experiment aimed at confirming an alternative Hypothesis?

Question

I am running a series of surveys with single variant A/B tests aimed at gathering information about how different groups of users perceive a series of website designs with different distortions applied. Each participant will be able to rate the various design/distortion combinations using a single metric using a slider rating system (-100 to +100). The aim being to identify which elements/features of a design, when removed or distorted, have a greater or lesser effect on the different groups of users and what is the comparable level of effect on them.

The Null Hypothesis is that there is no perceived difference. The alternative is therefore that there is a perceived difference.

The second stage of my research will involve creating different designs based on this information which I will test using the same apparatus. The aim being to see if I can evoke a specific response among particular user groups to particular designs based on the information gleaned from the first series of experiments. My problem is that I don't know the correct terminology to describe this second stage.

What is the correct name for this type of 'confirmation' experiment?

I have been using variances of "alternative Hypothesis confirmation experiment" but that really does not seem right.

Any help is much appreciated.

Answer 1

If you are doing a bunch of hypothesis tests in the first phase, you are likely to detect false positives unless you correct for multiple comparisons.

In the second phase, you commit to a particular intervention so multiple comparisons is not an issue.

You could call the first phase exploratory and the second phase confirmatory. But, there is nothing special about the experiment. You are simply applying a intervention and comparing its affects vs a control group. Your motivations or hopes don't make any difference.

But maybe I've misunderstood your experimental design. You aren't very clear in your description. Specifically, I don't see how the second phase is "rigged".

Further, if you are being scientifically honest, the goal of the second experiment shouldn't be to confirm the alternative hypothesis, but rather to see whether the null hypothesis can be rejected. The experiment would be successful no matter what you learn.