Tuesday, February 2, 2010

Significance of Statistically Significant Results in A/B Testing

You are running an A/B test (or multivariate test) but are in a hurry to make a decision to pick the winning page. Should you pick a winner based on initial few days of the data before your tool has actually declared a clear winner? This question comes up quite often during the conversations with the clients.

I always warn against such an approach and advice to be patient and get statistically significant results before pulling a plug on an underperforming variation o declaring a winner. I wanted to share few graphs with you to illustrate my point.

The following example is from a test that I am currently running.
First Two Weeks: If you make a decision without completely getting through the test, you will pick �Yellow� as the winner while declaring �Blue� a loser.


Month Later: There does not seem to be a clear winner.


Few More Days Later: Seems like �Blue� is trending higher. Should you pick �Blue� now?


Finally: We don�t have a clear winner yet.


As you can see, picking a winner or dropping a loser in the early stages of test, without having a statistical significant result, would have been a wrong decision.

Comments? Questions?


Looking to fill your Web Analytics or Online Marketing position?

Post your open jobs on Web Analytics Job Board
--------------------------------------------------------------------------

0 comments:

Post a Comment

 
Copyright 2010 Camera Dashboard. All rights reserved.
Themes by Ex Templates Blogger Templates l Home Recordings l Studio Rekaman