{"id":1276,"date":"2014-07-21T09:52:19","date_gmt":"2014-07-21T08:52:19","guid":{"rendered":"https:\/\/www.bronco.co.uk\/our-ideas\/?p=1276"},"modified":"2014-07-21T09:52:19","modified_gmt":"2014-07-21T08:52:19","slug":"what-to-do-when-your-split-test-is-failing","status":"publish","type":"post","link":"https:\/\/www.bronco.co.uk\/our-ideas\/what-to-do-when-your-split-test-is-failing\/","title":{"rendered":"What to do when your split test is failing?"},"content":{"rendered":"

Every fibre of your being tells you that your split test results are wrong and the variation you\u2019ve been running should be converting better than the original.<\/p>\n

You\u2019ve not just changed some small bits of copy or reorganised pre-existing content, you\u2019ve built something brand new or included some new feature to help your users. But your variation is failing; hugely. But what can you do next?<\/p>\n

Is giving up an option?<\/h2>\n

Your test has failed, give up and more on, right? That\u2019s fine if you\u2019re testing something simple like a new Add to Basket button but it\u2019s not really an option is you\u2019re testing something more complex like a brand new feature.<\/p>\n

There are times we can get our research wrong, misjudge our audience and give them something they do not use or want. We could walk away at this point and accept the loss but what happens on your next split test?<\/p>\n

In having a test fail you\u2019ve proved that your research might be lacking and if you just give up and move on to the next test you could be doomed to make the same mistakes again based on the same ineffective research.<\/p>\n

Instead you need to review the failed variation and try to understand what caused the test to fail, only then can you learn from the test and allow that to inform your future tests, which might include re-testing your failed attempt.<\/p>\n

Finding out what went wrong<\/h2>\n

Excluding internal conversions<\/h3>\n

The way conversion rate is calculated is as a percentage of conversions over the number of unique visits. So 1 conversion in every 100 visits would return a conversion rate of 1%. Simple.<\/p>\n

If either of these figures is inflated in any way by you or a client then the results of the split test will be not be accurate. Visiting the website might only have a small effect on the final results of a split test but if the website is utilised for internal ordering, such as handling telephone orders then these extra conversions on a single variation could have a large impact.<\/p>\n

Fixing the problem might be as simple as not including the code when the website is viewed from a specific IP address, but this requires these users to have a fixed IP which isn\u2019t always the case. Alternative methods could include constructing some form of URL that identifies an internal user or use an existing member account to the same effect.<\/p>\n

Rough and ready usability testing<\/h3>\n

Sometimes sourcing a few more opinions might be all it takes to shed some light onto the problem. As website owners and creators we can become a little blinded to websites we\u2019ve worked on for some time. While experience gives us knowledge and insight that\u2019s invaluable it can be difficult to have the same perspective as the consumer.<\/p>\n

Asking people with less experience of the website for help might uncover some issue and getting opinions this way is always worth doing before shelling out on paid for services. You might just find the answer is something simple and getting opinions in this way might identify a problem much quicker.<\/p>\n

Heatmaps & Analytics<\/h3>\n

It\u2019s obvious but looking at how your users are interacting with the page could reveal the answer to the problem. This sort of data is often gathered in the research phases before a split test is created but if your test is set up right you may be able to look at the data specific to the new variation rather than the original website.<\/p>\n

Understanding analytics information especially the referral information will give insight into why people came to the page, while bounce rates and time on page show if they stuck around to digest the content and then page speed will show if something as simple as slow page load might be holding you back.<\/p>\n

\"heatmap\"<\/a>
Example Heatmap Overlay<\/figcaption><\/figure>\n

Heatmaps and scrollmaps will show you how they\u2019ve moved around the page and what elements they click. You might find that an element you want them to click isn\u2019t getting any attention at all and instead the user is navigating away from this page and towards the home page or search bar which can be an indication that the user is lost.<\/p>\n

Usability Tests<\/h3>\n

Analytics and heatmaps produce data, and lots of it. But its quantitative data which is all numbers and measurements with a human needed to interpret the data, observe trends and make judgements as to what it means in a given situation.<\/p>\n

But usability tests provide qualitative data, which gives observations that have more meaning. Often this comes in lower quantities so it\u2019s difficult to plot any trends in the data but does give an insight into the thoughts of the consumer. There are a few different types of usability tests but mostly we\u2019d use the 5 Second Test and User Testing Videos.<\/p>\n

With the 5 Second Test you show an image for 5 seconds and the tester answers some follow up questions. It usually allows you to get more results for your money but the testers do not interact with your website. If your split test involves more than testing a new look this form of usability testing isn\u2019t likely to offer much insight.<\/p>\n

User Testing Videos often means sitting for a few hours watching people navigate through a website tripping up on things you take for granted. It\u2019s a great way of testing a website and is something to consider doing regularly as the website evolves. It\u2019s a fascinating experience to watch how people use your website but you have to consider that if one person has a problem it may not mean all your audience experiences the same issue. You have to use your own expertise to interpret what\u2019s happening and prioritise any issues.<\/p>\n

As a sidenote getting the brief right for these kinds of tests is important. Make it short and don\u2019t try and lead the user too much, let them do their own thing so long as you make sure they visit any page you\u2019re wanting them to view. Remember people have a habit of ignoring instructions so make it simple.<\/p>\n

Change and re-test or give up<\/h2>\n

Now you\u2019ve had chance to research how your users are experiencing your tested variation, and you\u2019ve made sure you and your clients aren\u2019t skewing the results you\u2019ll have a very easy decision to make on what to do next.<\/p>\n

Either you\u2019ve identified a problem, fixed it and can re-run the split test or you\u2019ve found that the test wasn\u2019t giving the users what they wanted and there\u2019s no advantage in continuing with the original hypothesis your split test was based on. Either way you\u2019ve learnt more about the website and its users and so can move on to other split tests with greater insight.<\/p>\n

But in some rare occasions you might find that the test failed and none of the extra research provided a good reason as to why. You might want to try and run the test again, and for longer to be really, really sure but if that doesn\u2019t give a positive result it\u2019s time to move on and see if any of the new data you\u2019ve gathered helps you come up with new split tests in other areas.<\/p>\n

Other things to consider when split testing<\/h2>\n

But make sure you give it time<\/h3>\n

Don\u2019t be quick to rush into a decision about whether a variation has won or lost. Many split testing services will let you know when a variation has won and the numbers are large enough to back this up.<\/p>\n

If your split testing software does show a winner within the first couple of weeks of the test don\u2019t listen to it. Often tests results can vary wildly when not much data has been collected and two weeks is rarely long enough unless you\u2019re split testing a high traffic website.<\/p>\n

Test something meaningful<\/h3>\n

I\u2019ve already mentioned about testing a new feature other than remixing or redesigning something that already exists. If you\u2019re not seeing positive results or just small increases then be bolder in what you\u2019re testing.<\/p>\n

It\u2019s true that in certain cases there are big uplifts in conversion to be had from small alterations in design or copy and small increases in conversion rate can have big benefits for websites with big turnovers. But in some cases, especially with smaller websites, adding a new page or feature that provides users more choice or enhances the user experience can be a better way to approach split testing.<\/p>\n","protected":false},"excerpt":{"rendered":"

Every fibre of your being tells you that your split test results are wrong and the variation you\u2019ve been running should be converting better than the original. You\u2019ve not just changed some small bits of copy or reorganised pre-existing content, you\u2019ve built something brand new or included some new feature to help your users. But […]<\/p>\n","protected":false},"author":2,"featured_media":1278,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"class_list":["post-1276","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/posts\/1276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/comments?post=1276"}],"version-history":[{"count":0,"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/posts\/1276\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/media\/1278"}],"wp:attachment":[{"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/media?parent=1276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bronco.co.uk\/our-ideas\/wp-json\/wp\/v2\/categories?post=1276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}