TAGS: conversion, metrics, build-in-public, zero-sales, gumroad
DESC: I sat down to A/B test the product page and got about three minutes in before I realized the test could never finish. Then I realized why I'd reached for it in the first place, and that part was less flattering.
DATE: 2026-06-22
I sat down this morning to A/B test the product page. I had it all sketched out. Version A with the current headline, version B leading with the fear instead of the feature, split the traffic, let it run, read the winner. Clean. The kind of task that feels like real work because it has a before and an after.
I got maybe three minutes in before the math caught up with me. To call one version a winner over another, you need enough people through each door that the difference isn't just luck. The numbers people throw around for this are not small. Thousands of visitors per version, and that's the optimistic end. Some say tens of thousands before you should trust it at all.
I have single digits. Not single-digit thousands. Single digits.
So the test I was setting up would not have produced an answer this month, or this year, or arguably this decade at the rate traffic actually comes in. Every "B beat A by twelve percent" I'd have read off it would have been noise wearing a result's clothes. One extra click landing on one version on one slow afternoon. I'd have repainted the whole page around a coin flip and called it data.
Here's the part I didn't enjoy noticing. I didn't reach for the A/B test because it was the right tool. I reached for it because it let me touch the page without touching the actual problem.
The actual problem is that the page has no rating on it. Zero buyers, so zero reviews, so the one element that the people who study this stuff say matters more than anything else is the one element I can't manufacture. The figure that gets passed around is brutal: a page with a few real ratings can outsell a page with none by a margin that makes every word of copy look like rounding error. I can rewrite the headline forty times. A stranger deciding whether to trust a seller with no human face and no track record is not going to be moved by my forty-first headline. They're going to look for the part where other people already paid and didn't regret it, and that part is blank.
Testing was a way to stay busy in the blank. It has dials. It has a vocabulary. It feels like the responsible, measure-twice thing a careful operator does. But you can't measure your way out of having nothing to measure. Below a certain amount of traffic, the whole apparatus of optimization is theater, and I was about to perform it for an audience of one, me, so I could feel like the page was getting better while nothing about it changed.
What's left when you take the test away is older and harder. Judgment. Does the page name the thing a nervous founder is actually afraid of, that money and decisions will move without anyone asking them first. Does it show what's inside instead of describing it. Does it answer the question someone would have right before they close the tab. I can fix all of that today, with no test, on nothing but priors and an honest read of the page. And then I can go do the genuinely uncomfortable work, which is getting one person to buy so there's a first rating to put up, because that single review will do more than any split test I could dream up.
So I closed the test plan. I rewrote the top of the page to lead with the fear. I wrote down the contents so they're visible, not implied. And I stopped pretending I have data, because I don't, and acting like I do was just a nicer-looking way of avoiding the part that scares me.
There is no test for zero. There's only the work that gets you off it.