How long to give an idea before you are allowed to conclude anything

You have an idea about where the next chunk of revenue comes from. A new segment, a partnership motion, an outbound push. The question is not whether it is a good idea — you cannot know that yet. The question is how long you are going to give it before you are entitled to an opinion.

The answer, prescribed to a founder deciding between four strategic options:

A week is not enough, six months is too long.

Jason Cohen

Roughly four weeks, with the rest of the business on a skeleton crew. Long enough to reach real people and hear real answers; short enough that you have not spent a season on it, and short enough that you will actually run it properly rather than letting it drift into the background.

What the box is really protecting

Not the calendar. The failure mode is much more specific, and everyone who has run one of these recognises it immediately:

You're gonna be tempted to do option A because it's easier. I can hide in Visual Studio and add a feature. That's easier than trying to get a customer on the phone and ask them questions. If you allow yourself to slip into option A, what you're doing is ensuring that you will not find out whether option D is a good idea. You will do a bad job of it, probably conclude option D can't work.

Jason Cohen

Read the last sentence twice. You do not abandon the experiment; you run a half-version of it, get a weak result, and then conclude something from that weak result. The retreat into comfortable work does not merely waste the four weeks — it manufactures a false negative you will carry for years.

And the same thing at the other end of the scale, where the conclusion arrives even faster:

That would be like the founder who makes a product and then makes four tweets about it to their 300 followers. Sales don't immediately appear, and so they go: that's stupid, I'm moving on to my next idea. And then they think they're smart because they did ten of these in a year. But in fact it's dumb, because what they did is plant a seed and never give it water.

Jason Cohen

Ten experiments a year, none of them run. It reads as decisiveness on a CV and it is the opposite: a person who has never once found out.

The sequence that puts the cheap work first

There is a good default for how much evidence to collect before building anything, and its virtue is the ordering:

• 2 hours of research. Does this survive contact with what is already known?
• 20 hours of conversations and a smoke test. Do the people who would buy it behave as though they would?
• 200 hours of building — and only then.

This term "validating", in air quotes, makes people think there is a way to get to 100% confidence. What validation is — or evidence, because I said don't build without evidence — is evidence that says if you build this, it might be something people want.

Rob Walling

The twenty hours are the ones that get skipped, always, because they are the uncomfortable ones. Two hours of research feels like work and is easy. Two hundred hours of building feels like work and is very easy. The twenty hours in between involve strangers saying no to you.

The related claim is blunter and worth pinning above a desk: building more does not de-risk the company. Every additional feature shipped before the twenty hours have been run increases the amount you will have to throw away if the answer is no — and it postpones the moment you find out, which is the only thing that was actually risky.

An initiative nobody owns does not happen

If the experiment runs inside a company that already has customers, there is a structural failure waiting that has nothing to do with effort:

What's bad is you say, here's this big strategic initiative, we're gonna go after BigCommerce. And then everyone's like — okay, well, who's doing that? I don't know, we'll just generally fold it into stuff that we're doing. That's not intentional enough. Every time something appears for the rest of the business, the other thing will take precedence, because that other thing is the existing business. There's existing customers tied to it. So it will trump the new thing.

Jason Cohen

The existing business always wins the argument, and it wins it honestly — there are real customers on the other end of every interruption. This is not a discipline problem that a shared commitment fixes. It is a queueing problem, and it has two mechanical answers:

• One named owner, whose week is the experiment and whose interruptions from the existing business have to be refused by someone other than them.
• Separate revenue and cost lines for the new motion, so you can see whether it is working at all and can decide deliberately when to fund it out of the old one.

Without the second, a new motion's early numbers are invisible inside the existing business's totals, and "is this working?" becomes a matter of opinion at exactly the moment you need it to be a matter of fact.

Deciding in advance what counts

The last piece is the one that makes the box worth having: write down, before you start, what result would make you continue and what result would make you stop.

Not a revenue target — four weeks does not produce revenue. Leading indicators: how many of the people you reach will take a call, how many describe the problem in the terms you predicted, how many ask what it costs. Pick the numbers while you are still neutral, because at the end of four weeks you will be tired, invested, and extremely good at explaining whatever happened.

The five rules

1. Fix the window — about four weeks — and put the end date in the calendar before day one.
2. Name one owner and protect their week from the existing business explicitly.
3. Run the twenty hours before the two hundred. Conversations first. They are the part that is skipped and the part that decides.
4. Write the stop/continue criteria in advance, as numbers you can count.
5. Notice the retreat when it starts. If you are shipping features in week two of a market experiment, the experiment has already ended and nobody announced it.