Analyze an A/B Test for Statistical Significance

This prompt runs a full statistical analysis of an A/B test and ends with a clear ship-or-not call, plus the Python code to reproduce every number. It asks for conversion rates, relative lift, a two-proportion z-test, the p-value against your threshold, and a minimum-detectable-effect sanity check, which together prevent the common mistake of shipping on a lift that isn't actually significant. Use it when you have raw counts from an experiment and need a defensible, repeatable decision rather than a gut read.

What you can expect back

Results
- Control conversion: 496 / 12,400 = 4.00%
- Treatment conversion: 575 / 12,510 = 4.60%
- Relative lift: +14.9%
- Two-proportion z-test: z = 2.42, p = 0.0156
- Meets p < 0.05? Yes.

Recommendation: Ship the treatment. The 0.6 percentage-point absolute gain is statistically significant at your threshold, and the sample is large enough to detect an effect of this size.

Reproduce:
from statsmodels.stats.proportion import proportions_ztest
count = [575, 496]; nobs = [12510, 12400]
z, p = proportions_ztest(count, nobs)
print(z, p)

Illustrative example — your results will vary by tool and inputs.

Get sharper results

• 01Tell the model whether your hypothesis is one-sided ('treatment is better') or two-sided, because that halves or doubles the p-value and can flip a borderline result.
• 02Give it the planned sample size and minimum effect you cared about up front, so the MDE check tells you whether a non-significant result means 'no effect' or just 'underpowered'.
• 03Watch for the peeking problem: mention if you stopped the test early or checked results repeatedly, since that inflates false positives and the naive p-value will mislead you.
• 04Ask for a confidence interval on the lift, not just the p-value, because a wide interval that includes zero is a clearer 'don't ship' signal than a bare significance verdict.
• 05Run the Python it gives you rather than trusting the hand-computed numbers, since arithmetic done in prose can be off while the code is verifiable.

Adapt it for your case