Working the Odds In Your Favour Through Split Testing

Reducing Risk Through Structured Email List Management & Split Testing

Email is one of the most powerful forms of sale generation for eCommerce businesses because you are talking to people who already bought from you. They already know, like and trust you.

The key to exponential growth in any business is continual improvement and innovation.

Split Testing Odds

But many eCommerce businesses are stuck in sending out the same type of email, whether it is a image heavy, catalogue style, a standard subject or a standard call to action because they are afraid of risking the success of their email campaigns by changing anything.

The risk is too high but they could be missing out opportunities that a simple change to their email layout, style or subject line could bring.

In eCommerce businesses, the ability to increase the lifetime value of your customer by getting your existing customers to buy from you more often enables you to get even greater leverage on the costs of acquiring that customer for the first time.

By understanding the odds of split testing, you can both innovate and reduce the risk of innovation.

The Example of Bill & Pete – eCommerce Entrepreneurs

So lets take the example of 2 eCommerce entrepreneurs, Pete and Bill who sit next to each other at a conference on email conversions.

The conference provides them with the latest findings and ideas in email marketing. After the conference, fired up and motivated, they both decide to implement the ideas on their next email campaign.

Both Pete’s and Bill’s business are similar, they have email lists of 100,000 customers. Their standard email click-through rate is 5% and the conversion of email to a $100 sale is 5%. So for each month they send out 100,000 emails and that email campaign generates $25,000 in sales each month.

This represents the control, as shown below:


Pete’s Business

Pete prides himself on getting things done as quickly as possible. So he gets home and sends his entire list of 100,000, the new version of his email.

He hopes that it will be successful but it could fail.

A success would be an increase in the click through rate from 5% to 6%, but a failure would be an decrease in the click through rate from 5% to 4%.


So Pete has a 50% chance of making an additional $5,000 (20% increase) but also a 50% chance of losing $5,000 (20% decrease).

It becomes an each way bet.

Bill’s Business

Bill is more considered and likes to plan and measure all his leads and conversions. Bill applies a structured approach to his email campaigns.

Like Pete, he hopes the new email format will be successful and it could fail. A success is measured at 6% and a failure at 4%.

Bill sends 10% of his email list the standard email (control) and sends 10% of his email the new improved format. He measures which email converts better and sends the balance of his list the winner.

If the new email format is a success then:


He have achieved an increase in sales over the conversion of $4,500 or 18%. This is only $500 less than Pete’s successful campaign.

But if the new email style was unsuccessful:


Bill would have only lost $500 in sales in testing the new email campaign.

By using the structured approach Bill has significantly reduced the risk of a downside.

Bill had a 50% chance of making 18% more and a 50% chance of losing just $500. The odds are weighted significantly in Bill’s favour.

Another way of looking at it:

Bill has risked $500 for the chance of winning $4,500. That’s odds of 9/1.

Pete has risked $5,000 for the chance of winning $5,000. That’s odds of 1/1 – not very good odds for the risk.