Had a great conversation today with Aydin Ghajar; a guy who really knows his stuff when it comes to building viral loops for consumer services.  The issue of rates/conversions yield big differences in ultimate outcomes for viral systems.  One of the things I’ve been grappling with thinking about A/B testing in MVPs is the complexity this adds early on.  After all, Lean is about doing things efficiently to learn what you need to match a customer/market need.

What happens when the basics to install the ability to learn how well A or B (or C or D) is non trivial?  A/B testing that isn’t trivial website button color, text takes some effort (days, but still).  My friend made an astute observation (paraphrased):

At first, your A/B testing while you’re still validating early assumptions is NOT parallel A/B testing (50 visitors this week into TWO 25 visitor buckets) but rather sequential: A is done one week, evaluated.  B is done week after, evaluated.

Why?  First off, A/B testing is complicated and best deferred until a bit later in MVP.  Why else? At first, when you’re testing your early hypothesis your volumes are small enough that you’re decreasing the accuracy of and time to determine if A or B is true.

In our example, if we get 50 users per week and we split them we’ll get 25 users in A and 25 users in B per week.  Week one, we get 25 of each, and get an action or two in each.  The statistic relevance or confidence of 25 user sample is LOW; it’s probably not enough to validate or invalidate the effectiveness or value of A or B (whatever you’re testing).  So, next week you get another 25 in each.  At 50 users, you’re confidence increases and now you’re ready to make a real determination on if A or B worked.

What’s the problem?  When you’re early on, getting things figured FAST is important.  While not right for every thing, I’d much rather have tried A in one week (50 sample size), find the results, and move on to B.  Same learning (faster in some cases) and more efficient (aka cheaper) initially.

So, I’m in perfect agreement with my friend.  A/B testing happens sequentially at first, then in parallel once practical (implementation/sample size).