Experimentation does not mean shipping something to see what happens. If we do that, we’re guaranteed to succeed—at seeing what happens. Something will always happen. You can always make up a good story about something you think you’ve learned, and no matter how bad things are going, you can always find at least one chart in Google Analytics that is up and to the right. And, as I mentioned before, it’s possible for your vanity metrics to be going up even while you’re ruining your product.
Science requires having a prediction, a hypothesis, about what will happen—based on theory, a set of assumptions. We need that prediction so that we can compare it to the actual results of our experiment. That’s one of the reasons why vision is so critical to startups. We need to be able to predict what customers are supposed to do when they encounter our product.
Which is what the Lean Startup approach boils down to - scientific product and business development, as opposed to the haphazard, gut-driven, happy-go-lucky and naive approach common to those new to the startup business. This makes me wonder why Eric chose a famous Zen painting to illustrate the cover of his book, when Zen Buddhism is about anything but science, but that doesn't detract from the fact that this is a good article (and probably a good book), and well worth a read if you're not familiar with Lean Startup principles.
If you read this far, you should follow my RSS feed here.