{"id":13425,"date":"2017-11-07T11:19:19","date_gmt":"2017-11-07T11:19:19","guid":{"rendered":"https:\/\/marvel7077.wpengine.com\/?p=13425"},"modified":"2018-05-15T12:11:33","modified_gmt":"2018-05-15T11:11:33","slug":"ab-testing-youre-wrong","status":"publish","type":"post","link":"https:\/\/marvelapp.com\/blog\/ab-testing-youre-wrong\/","title":{"rendered":"A\/B Testing – You\u2019re Doing It\u00a0Wrong"},"content":{"rendered":"

So, your company wants to increase revenue and adoption by making some marketing site tweaks. They want more conversions, more clicks, more shares, and more users. What do they tell you to do first? Well, A\/B test<\/a>! Compare two versions of a page, define a key goal (ex. clicks), and see if you get more clicks. But, does this actually work? Is it really the approach you should take? Let\u2019s look at the data.<\/p>\n

\"By nature, an A\/B test is an experiment that assesses multiple (often 2) versions of a feature or page relative to a defined\u00a0metric.\"<\/span>
\nThis article focuses on superficial A\/B tests \u2014 the testing of cosmetic changes that distract teams from delivering meaningful customer\u00a0value.<\/em><\/p>\n

A\/B Testing\u00a0Band-aid<\/h2>\n

For apps with millions of users, small cosmetic changes to your app, like color, layout, and language could net you some noticeable increases in your key metrics, like more clicks and user engagement. The real question is this: for companies with smaller user bases, should you try to focus on different \u201cSign Up!\u201d button colors or focus on actually making your product better?<\/p>\n

\"\"<\/a><\/p>\n

A\/B testing for many companies becomes a band-aid for poor value propositions. If your content is not being shared, maybe your content is actually not share-worthy, regardless of how amazing you make your \u201cShare Now!\u201d button.<\/p>\n

Moreover, maybe your goal should not be getting a prospect to click on a button. Maybe your primary goal should be to build trust, provide a sandbox demo, or empower the prospect to make a decision.<\/p>\n

Survey Data<\/h2>\n

AppSumo estimated<\/a> that only 25% of A\/B tests actually produced significant results. The question is why? Well, let\u2019s define significant results. For many of these tests, the primary metric orgs were trying to move was conversation rate<\/em>. So, if the rate wasn\u2019t increased, then the test failed.<\/p>\n

<\/div>
<\/svg><\/div>

\"Less than 25% of A\/B tests produced significant positive results.\" -AppSumo<\/p><\/span><\/blockquote>\n

However, we could look at it a different way. If changing your header slogan, hero image, or CTA didn\u2019t move metrics, then maybe that is indicative of a larger issue. A failed test should be an indicator that:<\/p>\n

    \n
  1. Your visitors are not ready to convert<\/li>\n
  2. Your visitors are seeking something else besides signing up<\/li>\n
  3. Your core product is fundamentally unattractive<\/li>\n
  4. You need to drive more qualified<\/strong> leads to your product<\/li>\n<\/ol>\n

    This next dataset is from a qualitative and quantitative A\/B testing survey of 26 A\/B testing practitioners conducted from May 1 to May 30 in 2016 (Northwestern, IDS \u2014 Justin Baker, 2016). While this is not the end-all be-all of surveys, it could still give us some meaningful insights.<\/p>\n

    \"\"<\/a><\/p>\n

    <\/p>\n

    \"\"<\/a><\/p>\n

    <\/p>\n

    \"\"<\/a><\/p>\n

    <\/p>\n

    \"\"<\/a><\/p>\n

    <\/p>\n

    \"\"<\/a><\/p>\n

    <\/p>\n

    Key Points<\/strong><\/p>\n