The balance between consumers’ right to privacy, and marketers’ wish to target is a fine and sometimes fraught one. This balance changes over time. What is acceptable to consumers today would not have been tolerable 20 years ago. For example, consumers understanding of how companies such as Tesco use their personal information mean that marketing messages can now be much more targeted and to the point. “We know you like baked beans so here is a high value coupon”.

More recently Facebook’s attitude to privacy has met with criticism. “Every year, people are sharing twice as much information as the previous year,” says Facebook CEO Mark Zuckerberg. So it is no wonder the availability of personal data on the web is becoming more of an issue. Facebook, similar to many data driven businesses, has an inherent conflict in that it potentially makes more money by being more open with peoples’ personal data. Facebook’s experience is a case study in how respecting and protecting privacy is fundamental to trusting a brand.

The new cookie laws have been causing some consternation among digital marketers. In mid-December the Information Commissioner’s Office published some helpful new guidelines which seem sensible and pragmatic. It looks like web analytics and email marketing will be able to continue to operate without too much impact.

More worryingly are the proposed changes to the EU Data Protection Directive which seems to take a German approach to privacy. This new draft EU legislation on privacy and data protection was leaked by the civil liberties’ group, “Statewatch”. This unofficial copy of the draft legislative contains stronger protection of data subjects’ rights and a much stronger focus on enforcement. The new legislation is likely to take the form of a Regulation, rather than a Directive.This means, once approved, the legislation would take immediate effect on all 27 EU members. The draft is summarized below.

An opt-in only regime for all forms of direct marketing. The Regulation would prohibit any use of personal data for commercial direct marketing purposes without consent. Such a limitation could obviously affect the ability of businesses to market their products and services to existing and new customers.

The introduction of a new “right to be forgotten”. This means a right for individuals to have their data deleted and not even used for the purpose of suppression. In my experience consumers are understandably wound up when companies do not remember that they do not want to be contacted.

An emphasis on greater transparency by means of additional obligations. Businesses employing more than 250 permanent employees will need to designate a data protection officer. The draft regulation introduces data breach notification requirements. Depending on the circumstances, data controllers will have obligations to notify supervisory authorities and data subjects of a personal data breach. This is to be done within 24 hours of the breach being established.

Limits on profiling of individuals. In many cases this will mean the individual’s consent would be required to carry out such profiling. I’ve never understood the aversion of data protection authorities to profiling or the use of scorecards as they are accepted by most authorities as a better decision making method.

Greater limitations on data transfers outside the EU. This will make it more difficult for companies to operate globally and to use service providers outside the EU. The increased level of protection under the Regulation could make it difficult to provide adequate safeguards.

Thankfully the wasteful and pointless data protection registration requirements and costs may be abolished. The UK Government has made it clear, with the economy in a poor state now is not the time to implement legislation that will hurt business. However, it is difficult to see how the digital, direct and data industries would not be hurt by the new draft Regulations. With the digital industry alone being responsible for around 10% of the UK sales the economy is likely to be impacted. That fine balance between consumers’ right to privacy and marketers wish to target is once again becoming a fraught one.

I was reviewing results from a split creative test on a basket abandonment email recently (names removed to protect the successful) and it struck me how the methodology for measuring email results can, quite erroneously, determine how we use email marketing and develop marketing strategies.

So I thought I would combine the results here with my recommendations on how to measure email marketing.

The Past

Viewed from the vantage point of today, the past always looks more simple than today. And looking back on where email marketing has come from is no different. There was a time (although it concerns me that some marketers are still trapped in a time warp of costly proportions) when opens and clicks were the currency that email was measured in.

Apply this to my scenario:

Clearly, reviewing this campaign based on past criteria (known today as ‘old-school blinkered view on opens and clicks’) would have led the marketer to choose Template B as the most successful and therefore the one to proceed with. But hold that thought.

There is a common agreement that email open rates are going to decline, if they are not already. Direct mail watched a constant decline of response rates from about 1998 onwards. Email is going to suffer a similar gradual decline but not for reasons of disenchantment with the media, as was the case for direct mail, but instead because individuals, as they become more accepting of email in general, sign up to receive emails from more and more organisations and competition for attention in the inbox increases. But this does not mean that income from email will fall…

The Present

I would paint the current situation as 3D world superceding the 2D. Most email marketers discount opens and clicks as merely a 2D measure in a revenue focussed 3D world.

The 3D slant is effectively attribution of income. You cannot assess a channel without direct results for that channel and cannot invest effectively without accurate historical results. One could also develop this analysis to fully incorporate indirect media attribution analysis that provides marketers with not only the direct income results by channel but the overall effect of marketing investment one channel has on another… sometimes referred to as the ‘Halo Effect’.

But that is not for now. Instead, look at the revenue results for the different templates.

Therefore, the choice of Template B as defined by the criteria of the past is seen to have cost this business 20% of all orders. The previous methodology ignored click to order conversion, far higher from Template A.

The focus today is very much on income and conversion rates underpinned with a growing focus of loyalty and eCRM managed through the email channel. It still amazes me when we meet with an organisation that does not have proper attribution.

Strategic planning also has one more key measure: Cost per order. Email is a growing channel with, in general, static direct costs. As I argued in my previous blog, organisations should be searching for the tipping point at which these changes push email to the lowest CPO of all channels, below PPC and Affiliates, opening up the opportunity for disproportionate investment in email as the most cost effective channel.

The Future

In short, what I am arguing here is if used correctly, email can be the CRM tool of the future and not simply a tool for generating income. An opt-in list is a list of individuals who have said to you ‘yes, send me stuff and talk to me’. Think about how valuable that is. Never before have marketers had consent to sell to people!

So, as Email develops as a CRM tool, then so Lifetime Value (LTV) measures must rise as a primary measurement. In the example below, taken from a live analysis, similar brands are compared to show that, on an annually calculated basis, the LTV of one brands email marketing is clearly suboptimal and an opportunity of at least £1/2m presents itself

Before you all comment on the limited nature of the LTV in the analysis above, this is, in my experience, where it starts – looking at the value of a customer on your database. Once you start to do that, then the challenge is no longer to increase your open rates, it is to grow Lifetime Value. That, as they say, is a whole new ball game.

So, how do you manage that fine balance between short term revenue and longer term list value, in order to make the most of the sales potential now and protect the value of your list for the future? Considering the key elements outlined below will help you decide who to send to and how often, in order to maximise the success of your email campaigns.

Understand the risks: The recent introduction of email priority schemes by many of the webmail clients has changed the email landscape somewhat. These schemes are primarily focused on improving the user experience of the recipient, by helping them manage an increasing volume of opted in email. This email, (known as Bacn) differs from spam, in that it is (or was) wanted, but just not at the moment. Unfortunately this is the category that most marketing emails fall into and the challenge is not about getting into the inbox, but it’s what inbox we get into!

In a recent report by Return Path* they noted that 81% of the Gmail accounts they analysed had Priority Inbox enabled and only 17% of the inbox was deemed as “priority”. In creating these processes, the ISP’s are attempting to help the recipient, by looking at how they treat the mail, and use this information to decide how to prioritize it in the future. It doesn’t take too much of a leap to link someone ignoring your emails, with your emails eventually becoming de-prioritised.

Defining engagement: All emails addresses are created equal, but some become more equal than others. Email lists are full of different types of recipient, from the most avid opener and clicker, to the email that has been on the list several years, but is yet to see any activity. The split between “active” and “inactive” can at first be quite a scary metric. It’s not uncommon for a list (that has not previously segmented by engagement) to have half its volume showing no activity for the previous twelve months. So this is where you start, segment your list into groups of people, who have opened or clicked an email, and how recently they have done it, 30 days, 60 days, 90 days etc; and of course those who have done nothing, and how long they have been in the list.

Over the last three years Econsultancy and RedEye have surveyed thousands of client-side and agency digital marketers about conversion rate optimization (CRO). We have found the CRO market is rapidly maturing. Technology is no longer the major barrier preventing companies from improving their website conversion rates. The challenge is becoming one of people and processes.

Over the last three annual surveys, when answering the question “what are the biggest barriers preventing your organisation improving conversion rates” lack of resources and budget have consistently been cited as the two biggest barriers. Over the three years poor technology has moved down from third to seventh place, or 25% to 18%, and poor integration between systems has moved from fourth to eighth place, or 24% to 17%. This indicates technology is becoming less of a barrier to optimization.

However, while technology has declined in importance, people and process issues have increased. In the most recent survey when we analysed the variables most strongly correlated with improved website conversion we found the top four in order of importance were, perceived control over conversion rates, having a structured approach to CRO, having someone directly responsible for CRO and incentivising staff based on conversion rates.  This is all well and good, but the problem lies in the amount of companies actually implementing these activities. Only 31% had a structured approach to conversion and only 25% bothered to incentive staff. On the plus side 73% of companies did have one or more person directly responsible for conversion, although this did leave almost a third of companies with no one responsible for conversion. It could therefore be argued companies just aren’t using their staff or managing their processes as well as they could be, when it comes to improving conversion.

The digital industry lacks experienced people. A lot of this is down to companies not investing enough in people and wasting too much on tools they don’t use or media that does not work. Sadly, when the cost of acquiring a customer online is much better than offline there is little incentive to change this.

Not enough digital marketers have the skill sets to analyse the masses of online data. Many simply make superficial use of Google Analytics to track numbers of visitors without getting underneath the data to understand why numbers have gone up or down. Knowing why makes it possible to do something about it. Avinash Kaushik’s 90:10 rule (you should spend $90 on people for every $10 spent on analytics tools) is as relevant today as when he first said it in 2006.

A lot of this comes down to working on the analytics problems that exist rather than running off reports when requests come in. The people that best prove the value of analytics are those that can converse with senior managers and understand their problems, then provide solutions through analytics. Without this the senior managers would never have thought to ask the web analyst to help them.

A lack of structure and process is more a reflection of the digital market’s immaturity. There has simply not been enough time for the more rigorous process disciplines we have offline to have developed online. As results for online marketing inevitably converge with offline results the incentive for companies to be more disciplined with their online marketing will drive change.  After all, technology can only take you so far, it’s up to you how you use it. The next challenge perhaps is finding a successful way to make the most out of the people and processes available.

It seems most of the controversy is based on the typical ambiguity that seems to exist in many online rules and regulations, based, possibly, on the need to cover all the bases, in an environment of constant change and evolution. Because, let’s face it, the situation is meant to be led by public opinion, with the legislators supposedly following suit. Do the public want to be “protected” from evil online marketing spies, who are poised and ready to sell something at the first sign of interest? Are they really looking for this type of protection?

The study last year commissioned by the Universities of Pennsylvania and California seemed pretty damming for the US attitudes to behavioural marketing, with 66% of respondents thinking tailored ads were “not ok”. When told how tailored ads were achieved, even more objected!

I’m wondering how many would have put their hands up in support of non targeted, irrelevant and intrusive marketing, “Hell yes, give me some of that!”

Seriously though, this would suggest that people don’t want targeted advertising, which is interesting, as anyone in the Direct Marketing business knows, it’s very effective! So could this be another instance of looking at what people do, not what they “say” they do. I’m sure Amazon improves their customer experience by adding “those who brought this, also bought…” and I bet plenty of people click on the links too.

So, what does this mean to the marketer? Are legislators going to force people to opt in to cookie tracking before they can use a web site? I don’t think so; the European Parliament has already debated the subject (Oct 2009) and it seems that it “doesn’t” require “prior” consent (based on advisory documentation released at the time). But this obviously doesn’t suit the privacy evangelists, who are still desperately snapping at the heels of the behavioural marketing industry, in the name of consumer interest (they say).

However, the fun comes when you consider local member states still have to implement this directive till May 2011; and although the IAB stated at the time the directive was released an opt in prior to tracking was not required, it’s still not cast in stone.

On the face of it, the UK marketing industry might have dodged a bullet this time, but it will still be easy for legislation to be interpreted another way, if there is enough public outcry to justify it. So we can’t rest on our laurels. Action needs to be taken now to head off any need for far more restrictive legislation in the future.

The favoured view seems to be one of “prior consent” being the action of enabling (or not disabling) cookie tracking in your browser. For people who are concerned about cookies this means web browsers that are easily user configurable give you privacy options you can set BEFORE you go browsing; so when you enter a website you already have YOUR chosen privacy settings in place. And after all people visit websites because they choose to, not because they are being forced by the marketer.

Data protection and privacy are an important part of the development of the internet, which relies on mutually beneficial collaboration. Consumers should be confident to share data, and it should be seen as vital that consumers are educated to the benefits of behavioural tracking. Companies should be as transparent as possible about the data they collect, and the use to which they put it. It can’t be left to the privacy evangelists to educate the consumer in how behavioural marketing works.

The internet is fast, it’s convenient, it can be intuitive; buts to be all these things it also needs to be an information exchange, with transparent data and tracking policies. We must ensure internet users are sufficiently educated, to be confident to share the data that is required. This will not only power the way the internet works now, but also to fuel its development in the future.

Challenges

First customer analytics has become much more complex over the last ten years with many more degrees of freedom, for example channels. Digital and telephone channels have also created a need for real time analytics. Previously campaigns typically had their own discreet time slot, whereas now many campaigns will run simultaneously. Gone are the days when you could analyse that TV campaign in glorious isolation. Different channels have different data sets. Sometimes these data sets are incompatible, for example offline and online data. Lots of data exists in disjointed systems.

Computer processing power has led to an explosion of data volumes. But more data does not necessarily lead to better decisions; for example the credit crunch was a car crash with the banks seemingly being blind to the now obvious subprime mortgage lending problems in the US.

The recent confusion with the new cookie laws is a good example of the challenges created by privacy. Balancing the consumer’s right to privacy with the marketer’s wish to target is always a fine line.

Often there is no one simple objective or dependent variable. For example sales are important but customer engagement and online registrations also need to be taken into account. Last but not least it is always a challenge to sell it to the board. Return on investment is not about algorithms, it is about accounting.

How to address these challenges

First recognize that good analytics is both a science and an art. Improve your emotional intelligence when it comes to data analytics. You can do this using some very specific techniques. Develop multi-functional teams to get more pluralistic input. Use complimentary data and methods. At my own business we have found incorporating usability research techniques with web analytics has led to much better results when trying to improve website conversion. Most data analysts are left brain people who see the World as something that can be targeted, segmented and put neatly into boxes. We also need to engage right brain thinking for better creativity and big picture possibilities.

While it is important to use best of breed tools such as SPSS or SAS it’s even more important to worry about who will drive them. Avinash Kaushik has a very apposite rule of thumb for web analytics, that is for every $100 spent on web analytics spend 10% on the analytics tools and 90% on the people.

Use a scorecard approach to handle many different objectives at once. Identify a small number of measures and attach targets to them. Don’t try to eat the elephant all at once. Break complex analytics questions into their constituent parts. Marketing Directors don’t tend to like neural networks, not just because they are complex and unwieldy, but because they give a black box solution which cannot be debugged in a transparent way when something goes wrong. So make sure your analysis is transparent and robust.

Strive to become a more data driven business by making your data more actionable. Explain to senior management at every opportunity why data is an asset in accounting terms. To find meaning in data it’s now as much an art as it is a science. Having enough data is longer the issue; it’s what you do with it that counts.

1. High shipping costs – 44%
2. Not ready to purchase – 41%
3. Price checking – 27%
4. Price too high – 25%
5. Wanted to save products for later – 24%

Abandonment due to high shipping costs may certainly require a real time response. But number two, not ready to purchase, definitely does not. A real time response could well be unwelcome. The customer needs to be left alone to consider their purchase and, depending on the product, pricing etc, should be contacted later. Equally, price checkers will not want to be hurried. If it is a holiday, testing longer timescales will be informative. And finally, ‘wanted to save the product for later’… well, an immediate email screaming BUY NOW! could easily have a detrimental affect! So, what is the solution? Here are some straight-forward suggestions.

• Research why your customers abandon. Surveys are simple and cheap, so set up a trigger to email a short survey to basket abandoners.
• Segment your booking process. Individual online behaviour can be used to define segments. When you have your survey results (or indeed without any survey results) review you booking/purchase process.If you have a five stage purchase process it is likely the motivation for abandoning at each different stage will be different. For instance, if you have a pre-registration stage, drop-outs at this stage are likely to object to this. But if the abandonment is at final payment, it may be price or shipping fees.
• Build separate basket abandonment emails for each stage based on likely motivation.
• If you are using only one treatment, then address the top three reasons the research tells you people abandon.
• Always pull in images and information on the product/basket that was abandoned. This always increases performance.
• Test the timing of the basket abandonment email. Use your own judgement initially but then let the data guide you.
• Finally, consider using a two or three stage process, say one hour/three days/seven days.

And why is this worth this investment? Well, basket abandonment emails are your second most profitable behavioural email, usually driving 20% or more of your income from email. The number one most effective? Well, sorry, but we save that information for clients! Good luck!

Why the relationship is so important?

There are few who would dispute the importance of testing new website optimisation ideas (hypotheses) in a live environment with real users using tools such as AB/Multivariate Testing software, before final implementation on the website. But the optimisation ideas have to come from somewhere…  And who is better qualified and has a greater understanding of user behaviour than the User Experience or Usability consultant. Hundreds of hours spent watching and listening to users, sharing their pain and delight during website interaction naturally results in a good understanding of what makes users tick.  And as optimising a website is all about influencing the behaviour of users in line with business objectives, their role in the orchestration of AB/MVT experiments is so important.

Difficult relationship

A Multivariate Testing (MVT) salesman a few months ago at a London conference said to me “We enjoy proving usability consultants wrong!”

Whilst this is quite an extreme viewpoint, it’s clear that not all AB/MVT professionals are sold on the idea of usability. And in a similar way, many of the Usability consultants I have worked with over the years have been just as sceptical about AB/Multivariate Testing.

In a strange way it’s as if both parties see MVT as a substitute for usability and as a result are a quite defensive about the whole subject.  In fact these two disciplines are totally complimentary.

So in an attempt to heal the wounds like any relationship counselling session, the most important thing is to air each other’s issues and concerns. So here are the types of comments MVT companies say about Usability:

• “Usability testing is not statistically significant due to the small sample sizes.”
• “Participants do not behave naturally in a usability lab when there are cameras pointing at them and a moderator asking them questions.”
• “It’s so expensive as you need to compensate participants.”
• “It’s just a few individuals’ opinion – what do they know.”

And here are the types of Usability and User Experience comments about AB/Multivariate Testing I have heard over the past few years:

• “Where do the MVT companies get their optimisation ideas (hypotheses)? They don’t seem to be based on evidence or insights?”
• “How can they create the design variations (recipes) for optimising the website if they don’t have User Experience Professionals, Interaction Designer or User Interface Designers? And they don’t do Usability?”
• “Everything just seems very random; it’s as if they’re just throwing things against a wall and seeing what sticks.”
• “How could a change to the colour of a button have resulted in huge conversion rate increase unless there was an underlying issue with the button in the first place?”

(I realise the usability list is quite short at the moment so feel free to comment to extend it.)

It’s just a question of language

One of the reasons why Usability and MVT experts struggle to understand each other as well as they should is their use of language.  They often talk about the same things in a completely different way. And this language barrier as well as their different viewpoints can again make it a little more difficult to get along.

The following Wordle Word Clouds are snapshots of the Wikipedia page content for Usability and Multivariate Testing. Try finding the word “user” in the MVT word cloud or “visitor” in the Usability wordcloud!  Also “Design” is so prominent in the  Usability wordcloud, and yet so hard to see in the MVT wordcloud.

The future

UX/Usability guys are wising up to the idea of AB/MVT as a very useful tool for evaluating and fine-tuning all the great recommendations which come out of usability testing sessions and other qualitative research such as surveys, remote usability testing etc.

Websites such as whichtestwon.com have definitely helped raise the profile of AB and MVT in the user experience arena.  And the fact that so many of the winning recipes are due to enhancements to the user experience just as often as they are due to persuasive design patterns and messaging, again reinforces everything the UX and usability consultants believe in, and makes them want to use the tool.

And the fact that the tools are so readily available to use increases their exposure to the wonders of MVT.  Tools such as Google Website Optimizer and Visual Website Optimizer are so accessible, there is no excuse not to have a go and learn all about them, even if you don’t know your “recipes” from your “hypotheses”!  Check out the Econsultancy MVT buyers guide.

MVT companies are now showing a greater interest in usability, or at least pretending to do usability. This suggests they are realising its importance, or at least have realised it’s a good idea to talk about usability to prospective clients to allay their clients’ fears that the MVT house isn’t a user centric company.

Conclusion

Usability and AB/MVT specialists can and should work together. The disciplines are totally complimentary when the individuals from either camp except the limitations of their own tools and learn to recognise the strengths of the others’.

Finally working in silos is not the way forward. After all, we’re all working towards the same goal; to make your online business more successful. Please let me know your thoughts and feel free to disagree with anything I have said.  Would love to hear from MVT companies who are usability testing fans?  And usability people who like nothing better than split testing?

Businesses are fully aware of the benefits of operating online. As well as the advantages of significantly streamlined overheads, growth in online spending – especially in the UK – continues apace:

• UK shoppers spent £4.9bn online in February 2011, up 20% on February 2010 (IMRG March 2011)
• £5.3bn was spent online during May 2011 (IMRG Capgemini, June 2011)
• 40% of UK consumers shop online at least once a week (Reevoo, June 2011)
• Annual e-retail spend has grown from £0.8bn in 2000 to £58.8bn in 2011 (IMRG May 2011)

So, why then is it that so many businesses can be reluctant to invest in conversion rate optimisation programmes that will help secure them genuine competitive advantage and with that, larger shares of this lucrative pie?

Before my current role, I spent many years working in offline marketing with a particular focus on retail. Without exception, the successful retailers were those that continually assessed and improved the conversion potential of their bricks and mortar stores by reviewing every aspect of the customer experience;  how and where merchandise was displayed, what products were offered, seasonal requirements, the flow from one store area to another, the look and feel, location of payment points, checkout pass-through times, customer service levels, and so the list goes on. And, they used every tool at their disposal to deliver rich data to inform their store improvement decisions, from qualitative and quantitative customer research at one end of the scale, to mystery shopper exercises at the other.

Sadly, not only is such diligence the exception rather than the rule in online commerce, many of the very businesses that trail blaze offline are, by comparison, shamefully neglectful of their online operations.  And shockingly, numerous pure play operators are just as bad!

In the early days of ecommerce, when for many an online presence was an added extra rather than a primary revenue source, this lack of focus might just about have been understood. (Though why any business would not want to steal a march on competitors is beyond me.)  But in today’s climate, when online business could be the route to a profitable future, businesses will ignore CRO at their peril.

So, for all those CRO believers who need a little help to persuade the less enlightened in their organisations, here are just a few reasons for investing in website conversion rate optimisation.

Maintaining a strong and positive brand image
Your website is a key representation of your brand. Providing an excellent user experience that out-performs the competition will be instrumental in establishing positive brand perceptions. On the other hand, an underperforming website can seriously undermine brand equity.

Staying one step ahead of the competition
Your customers’ online needs are constantly evolving as both the channel and they mature. CRO techniques will ensure you stay abreast of these developments and how best to meet them. However, if you lose track of changing customer requirements, don’t be surprised when you lose business to more innovative competitors who are optimising their sites.

Getting to the root of problems and solving them
When web analytics identify behaviours that are limiting sales – such as consistent pre-purchase exit points, abandoned baskets, high bounce rates – there’s no need to file away yet another disappointing report. CRO techniques will find out why this is happening so that you can fix the problems.

Eliminating risk and costly guess-work
When there are several potential solutions to an online problem or user need, eliminate the risks and the guess-work by using CRO techniques such as A/B or multivariate tests. These are easy to implement, extremely cost-effective and will provide you with the definitive answer.

Enjoying the biggest possible share of a major sales opportunity
When shopping for something that can be bought both online and offline, 34% of UK shoppers would prefer to buy online (Econsultancy Habits & Motivations of Consumers Report, July 2010). In any sector, the sites optimised for conversion are the ones that will benefit most from this substantial opportunity.

And finally……having won the argument and secured your CRO budget……….

Ensure you optimise your site for mobile
As well as the viewing area / rendering issues presented by mobile, RedEye’s usability testing experience has shown that mobile journeys are often different. Currently, there are 15 million Smartphones in use in the UK (IAB 2010); their use for online transactions is growing. Mobile payments increased by 5% during 2010 (Cybersource January 2011); and it’s predicted mobile devices will be the dominant tool for web access by 2015 (Morgan Stanley 2010).

Testing is easy and can give quick and big wins. Most things are easy to test. You can get results in real time. Technology such as MVT (multi-variate testing) is designed to help speed up testing and Google’s Website Optimizer is free to use. However, remember it is not so much the technology, but what you do with it that really counts.

Without properly constructed tests no learning is reliable. So, test one thing at a time and compare like with like. In your test, make sure you’re comparing a similar size of a similar demographic at a similar time for a similar offer. So for example, a test at the weekend might give a different result to a weekday. Where the campaign or traffic value is high use a control. The control group gets what you have always used, while test groups get something new and different. For lower value tests you can use a simple champion/challenger approach where the test is run against the existing format.

Set a goal, for example increased sales, more conversions or a higher average order value. Make sure you can measure your goal. Design tests for the things that can and will have the greatest impact on your goal results. Test the big things, those things that could really move the dial. Don’t tinker around the edges.

Benchmark competitors for ideas and brainstorm what tests to run with four to six stakeholders from different parts of business. Pick out those tests that could make the greatest difference. Prioritise tests by potential value and ease of implementation. If all your tests work then you are not taking enough risks. Before you start to test, consider what you want to achieve. Design tests carefully, consider how you will know whether the test has worked. Do not waste time testing things you could not implement.

Each test group has to be large enough to give statistically valid results. This depends on the responses rather than the number of people who see the test. A good benchmark is election polls. For 1000 market research interviews, 95% of the time a repeated poll would be within 3% of the same result.

The higher the confidence limit required, the bigger the sample (80% confidence requires a much lower sample than 95%). If a test is expected to give 1% to 2% improvement, the required sample size is much higher than if it’s expected to give 20% to 30% improvement. There are freely available online sample size calculators to check.

Testing is the best way to improve results and a structured approach means you can confidently use those results. Make sure your investment in testing is not wasted by doing properly structured tests.