Compare the messages.
Understand the result.
Test subject lines and content using clicks or attributed revenue. Review uncertainty and guardrails before choosing a winner; attributed revenue is different from incremental revenue measured against a control group.
Measurement you can act on
Built to be right rather than to look confident
Sample, then send the winner
Pick how much of the list to test on. The rest waits, and receives the version that won — so most of your audience gets your best email, not a coin flip.
Decided on money, not vanity
Choose unique clicks or attributed revenue as the deciding metric. Revenue counts orders attributed to a click within the measurement window. That comparison does not establish incremental sales.
Stops when it stops mattering
A test ends when picking wrongly stops being expensive — not the moment one version edges ahead. Watching the dashboard cannot talk it into a false winner.
Guardrails that outrank results
Unsubscribes, complaints and bounces are tracked per version. A version that crosses your limit cannot win, however well it performed — your deliverability is not for sale.
Opens cannot decide it
Apple loads images for its users, registering opens nobody made. Bitelio shows them for context and refuses to let them choose a winner, because on that metric both versions score the same.
Honest about uncertainty
When the deadline passes without a clear answer, you are told no difference was detectable. That is a real result — and a far more useful one than a winner invented to fill the space.
Why most A/B results are wrong
Two mistakes almost every tool makes, and how this one avoids them
Stopping the moment it looks good
A significance number assumes you look once, at a sample size fixed in advance. Refresh a dashboard until it turns green and you will always find a winner — whether or not one exists. Bitelio ends a test on the expected cost of choosing wrongly, so watching it cannot bend the answer.
Measuring the one thing Apple broke
Mail Privacy Protection opens messages on the recipient's behalf, which on a consumer list is commonly half your "opens" — and it hits both versions equally. A subject-line test judged on opens is noise wearing a percentage sign, so opens do not get a vote.
What people test first
Start with the line everyone reads before deciding to open
Subject lines
The one thing every recipient sees. Two takes, one winner, the rest of the list gets the better of them.
Sender identity
A person or the company? Test the from name and address and find out which your audience answers to.
Offers and calls to action
Different content, different button, different framing — judged on revenue rather than on taste.
Find out what works
Set up a test in a minute. Bitelio handles the split, the maths, and the send.