Did that calculator really confirm your test was statistically significant? Running A/B tests can be extremely beneficial and also exciting. That’s why everyone who runs them likes to watch as the results stream in. This can be a huge problem, especially if you: End the test once it reaches statisitcal significance. Continue to run the test until it shows statisitcal significance. Look at every possible metric to see what changes. Both of these actions will show false positives.