Originally posted as a guest post on Ari Murray's Go-To-Millions newsletter. I cannot recommend it enough if you haven't checked it yet.
If you’re reading this, you’re probably familiar with CRO & A/B testing:
Test a new landing page or product page change, measure conversion rate or ARPU, and implement the winner.
Pretty straightforward.
However, subscription businesses are trickier.
Consider this scenario
You run a test trying to bust subscriptions versus one-time purchases, making the latter more appealing:
Here are the results:
In a typical test, you’d immediately declare this a failure—you’re generating almost 6% less revenue per website visitor.
This means your ROAS takes a hit.
But with subscription eCommerce, what matters is the longer-term impact of acquiring more subscribers.
So you pause the experiment and monitor it for 2 months.
After 60 days, the variant’s LTV exceeds the control by +7%.
BINGO!
And the numbers will continue improving since subscribers generate higher LTV than one-time buyers.
When testing on subscription eCommerce, conversion rate and ARPU tell only part of the story.
The crucial metric is Average Revenue per User after 30/60/90 days, which factors in:
To test subscriptions effectively, you need:
1. Integrate your A/B testing platform with your eCommerce store
We use Intelligems and Shopify. It is very easy to use and if you have your COGS uploaded, you can evaluate profit too.
2. Tag customers based on which test version they encountered
With Flows and Intelligems, you can do it pretty easily. Just follow this simple tutorial and you are good to go.
3. Use an analytics tool to track each cohort’s LTV progression.
If you have a Data Analytics team on your brand, you can create a Looker Studio that pulls data from Shopify automatically and filters customers based on tags.
If you don’t have that luxury, you can always export the data into an Excel or Google Sheets and calculate each cohort’s total revenue.
4. Conduct statistical tests to compare variant and control performance
Since revenue is a continuous metric – contrary to conversion rate that is binary – the usual statistics calculator won’t suffice.
I use this one created by Andrea Corvi, which lets you analyze the test results in a few steps.
The only extra element you’ll need to calculate is each cohort’s standard deviation, which can be done through an Excel Formula or by uploading your data to ChatGPT and asking it to calculate it for you.
That’s it!
Hopefully, you are now armed to start experimenting with subscriptions and explore other venues than conversion rate and AOV.
And if you want to really take this to the next level, you can measure contribution margin per cohort by discounting COGS and CPA, but that should probably be a topic for another send.
In short, subscription experimentation is exciting but complex.
Get it wrong, and you risk implementing changes that harm your long-term bottom line.
Get it right, and watch your retention rate, LTV, and profit soar.
