This article by Noah Kagan once again makes the point that if you cannot fail you cannot learn, and gives up, as examples, some failed A/B tests that AppSumo conducted.
Only 1 out of 8 A/B tests have driven significant change.
(...)
All of these were a huge surprise and a disappointment for me.
How many times have you said, "this experience is 100x better, I can't wait to see how much it beats the original version?"
A/B tests are one of the most effective implementations of actionable metrics, but make sure the measurement boils down to a change in a key performance indicator. Noah finishes with some tips:
- Focus on one priority per week, with 1-3 tests focusing on that goal.
- Be patient - it may take a few thousand visits or 2 weeks for a test to be conclusive.
- Be persistent - most of your tests will fail, but that's ok.
- Focus on things which are likely to get you big results.
