Check out this story from Wired about why most A/B testing is inadequate.
While A/B testing makes sense on big websites where you can run hundreds of tests per day and have hundreds of thousands of hits, only a few offers can be tested at one time in cases like direct mail. The variance that these tests reveal is often so low that any meaningful statistical analysis is impossible.
Worse, the results don’t identify which variables caused consumers to respond.
As a result, response rates for emails, catalogs, and other direct marketing campaign methods — still a staple of many businesses — are very low — usually less than 5% and often less than 0.5% — and they’re declining.
So what role should A/B testing play in the EdLab product development/iteration process?