A/B Test

From the book, Chapters 13 and 14. The book’s checkout test: 6.0% against 7.8% with 2,000 visitors each gives z = 2.25 and p = 0.025, a rise of 1.8 points (95% interval about 0.2 to 3.4). A p-value is the chance of a gap this big if the versions were equally good; it is not the chance that B is better. Decide the test length in advance and look once: checking every day inflates false alarms. A test of 2,000 per version has only a 33% chance of detecting a real rise from 6% to 7.2%.