Measurement
Building holdout groups into every campaign, reading the difference honestly, and accepting that most reported marketing revenue was going to happen anyway.
Every marketing platform reports revenue it is responsible for, and every one of them overstates it. Not through dishonesty but through construction: they count the customers who saw the message and then ordered, and cannot count how many of those would have ordered anyway. In a category where people buy pizza every couple of weeks regardless, that is a very large number.
Randomly withhold a portion of the eligible audience from a campaign. Compare their order rate and revenue per customer against those who received it. The difference is the lift, and lift is the only figure you can actually spend against.
Randomly is the operative word. Holding out the customers who unsubscribed, or the ones with bad addresses, or the last 10% of an alphabetical list, produces a comparison group that differs systematically from the treated group, and that difference will masquerade as your result.
| Audience size | Holdout | Note |
|---|---|---|
| Under 1,000 | 20% | Small samples need a large holdout to detect anything. Expect wide error bars. |
| 1,000–10,000 | 10% | The standard case for most single locations. |
| 10,000–50,000 | 5% | Comfortable precision without much forgone revenue. |
| Over 50,000 | 2–5% | Precision is easy at this size; keep the holdout permanent. |
Two kinds, doing different jobs. A campaign holdout measures one send. A permanent holdout, a small group excluded from all marketing indefinitely, measures the whole programme.
The permanent holdout is the more uncomfortable and more valuable of the two, because it answers the question nobody asks: what would happen if we did none of this? Most operators are surprised by the answer in both directions. Some discover a programme doing far less than reported; others discover their baseline is much weaker than they assumed and marketing is holding the business up.
A permanent holdout costs you a little revenue and buys you the ability to say what your marketing is worth. Nothing else does that.
Three disciplines separate a real read from a comforting one.
Realistic incremental lift in restaurant marketing is more modest than platform reporting suggests, and that is fine. A win-back ladder producing 8–12 points of incremental return over a holdout is a strong programme. A broadcast promotion producing 2–3 points of incremental order rate is normal and can still be highly profitable given the low cost of email.
What you are looking for is not a big number. It is a real one, repeated, that you can multiply by your margin and use to decide next quarter’s budget.
The forgone revenue on a 10% holdout is real and small. The cost of running an entire marketing programme for years without knowing which parts work is much larger. Treat the holdout as the cheapest research you will ever buy.
Yes, geographically rather than individually. Withhold comparable carrier routes or comparable markets and compare sales. It is less precise than individual randomisation and it is the only honest option for channels without a click.
Indefinitely, refreshed occasionally so the group does not drift into a permanently under-marketed cohort that no longer resembles the rest of your file. Rotating a proportion of it annually keeps the comparison fair.
It happens, and it is genuine information: usually over-contact, wrong offer, or a campaign cannibalising full-price orders. Investigate rather than dismiss it; a negative result found early is worth more than a positive one found late.
Keep reading
Why last-click attribution systematically misvalues restaurant channels, and a practical model for splitting credit across email, SMS, search, social and mail.
MeasurementReading retention curves by acquisition month, spotting the second-order cliff, and why total revenue hides everything that matters.
Campaigns & offersA three-rung win-back ladder with escalating value and descending cost per contact, plus the honest maths on when a lapsed customer stops being worth chasing.
Campaigns & offersWhy issuing value forward, redeemable on the next order rather than this one, protects margin today, buys a second occasion, and self-selects the customers worth paying for.
marketing.pizza runs all of it, every night, across every store you have.