Lean startups don't optimize. At least, not in the traditional sense of trying to squeeze every tenth of a point out of a conversion metric or landing page. Instead, we try to accelerate with respect to validated learning about customers.

Often when people hear "Build-Measure-Learn" they think of split testing and everything that comes along with it. The good and the bad. The Lean Startup methodology advocates split testing, however this is a technique used to validate hypothisis about customer behavior, not a method for determining what a product should be. That's your job.


Step one is to have a strong vision. You must then test that vision against reality, and then decide whether to pivot or persevere. Each part of that answer is complicated, and I've written extensively on the details of how to do each. What I want to convey here is how to respond to the objections I mentioned at the start. Each of those objections is wise, in its own way, and the common reaction -- to just reject that thinking outright -- is a bad idea. Instead, the Lean Startup offers ways to incorporate those people into an overall feedback loop of learning and discovery.

So when should we split-test? There's nothing wrong with using split-testing, as part of the solution team, to do optimization. But that is not a substitute for testing big hypotheses. The right split-tests to run are ones that put big ideas to the test. For example, we could split-test what color to make the "Register Now" button. But how much do we learn from that? Let's say that customers prefer one color over another? Then what? Instead, how about a test where we completely change the value proposition on the landing page?

Price: $26 $14
Fall 2011
The Crown Publishing Group
A division of Random House, Inc.

Share this page?

You said: