swombat.com

daily articles for founders

Appsumo's failed A/B tests  

This article by Noah Kagan once again makes the point that if you cannot fail you cannot learn, and gives up, as examples, some failed A/B tests that AppSumo conducted.

Only 1 out of 8 A/B tests have driven significant change.

(...)

All of these were a huge surprise and a disappointment for me.

How many times have you said, “this experience is 100x better, I can’t wait to see how much it beats the original version?”

A/B tests are one of the most effective implementations of actionable metrics, but make sure the measurement boils down to a change in a key performance indicator. Noah finishes with some tips:

  1. Focus on one priority per week, with 1-3 tests focusing on that goal.
  2. Be patient - it may take a few thousand visits or 2 weeks for a test to be conclusive.
  3. Be persistent - most of your tests will fail, but that's ok.
  4. Focus on things which are likely to get you big results.
More from the library:
Startup gung-ho
Hand to wannabe "product guys"
Can you build a great tech firm outside of Silicon Valley?