Did that calculator really confirm your test was statistically significant? Running A/B tests can be extremely beneficial and also exciting. That’s why everyone who runs them likes to watch as the results stream in. This can be a huge problem, especially if you: End the test once it reaches statisitcal significance. Continue to run the test until it shows statisitcal significance. Look at every possible metric to see what changes. Both of these actions will show false positives.
Many programmers have conducted their random samplings and collected their anecdotal data and then begged the question: Why is it that so many programmers are also musicians? Cognitive Bias It’s very likely we are suffering from confirmation or ingroup bias. There are plenty of programmers who don’t play an instrument, perhaps we only remember the ones who do. According to https://www.statista.com around 11% of the US population plays a musical instrument.