A/B Testing: The Devil’s in the Details

business2community.com

A/B testing has been around, although not always under the same name, for over a century now. In the early 2000s Google data scientists used it to establish the optimal amount of search results to display. It’s therefore safe to say A/B testing is kind of a big deal. But what exactly is A/B testing and how do you do it right so it will benefit your app?

What is A/B testing?

In marketing, A/B testing is performing an experiment with two variants (A and B) to see which one gives the highest yield. A crucial part in this is that the result should be scientifically significant. A/B testing is widely applied in web development, for example for optimizing the purchase funnel of ecommerce websites. Two subtly different versions of a landing page are shown to two groups of randomly selected visitors. Subsequently, differences in bounce rates, CTR (click through rates), conversions, etc. are measured to see which variant performs best. By continuously performing these tests the optimal combination of copy, graphics and calls-to-action is achieved and the funnel is optimized. In app development, A/B testing has only relatively recently getting significant attention. However, thanks to the emergence of convenient app A/B testing software it is gaining a lot of traction and by now every developer should know about it and use it to their advantage!

What is there to test?

Although there are some parallels with web development, app development is a completely different ballgame. The goals app developers usually pursue are very different which asks for a different approach to A/B testing as well. In any given app there are a bazillion things you can A/B test.

Visuals

As with websites, an app’s visuals are extremely important and often have enormous effects on user behavior. Think about fonts, icons, buttons and of course what colors you use. Make sure your visuals for your call to action are striking.

Copy and call to action

The wording you use in your app is also an aspect that is crucial for the look and feel of your app and has significant effects on your bottom line. Especially the formulation of your call to action (register, get started, share, etc.) determines your conversion rates.

In-app and push messaging

By sending your users messages you can convince them to do the things that are valuable to you. Sending the right type of messages at the right time to the right people will boost engagement and conversions.

Screen flow and layout

The structure of your app makes a big difference in your app’s UX. What you think is a logical layout may be incomprehensible by your users. Be creative and see what works best for your users.

How do you do it right?

1. Formulate the right hypotheses.

What is the objective you are trying to accomplish? A right hypothesis is one that you can confirm (or debunk) with hard data.

2. Keep variations limited

Avoid testing two completely different variations. By keeping tests small and focused you can pinpoint exactly what item produces an effect on outcomes. Large variations can be suitable to get general directions for the development of a new app however.

3. Segment your users

Different users behave differently. By segmenting your users into meaningful categories you can achieve optimal results for each of them.

4. Honor the scientific method

Test results can only be statistically significant with a large enough sample. Follow the rules when it comes to data collection, sample size and random selection.

5. Test. Analyze. Repeat

If you think you can ever be done with A/B testing, you’re gonna have a bad time. Every iteration and update brings forth new things to be tested. A/B testing shold be part of your routine in order to adapt to new trends and changing behaviors.

One last thing

An important thing to remember is that A/B testing is a game of extreme subtleties, both in variations and results. That is why you shouldn’t expect to see your revenues double overnight and give it patience. A  thought-through and disciplined approach will drastically improve your bottom line, all in time. Finally, take into account that there is a limit to where A/B testing is profitable. Testing all bazillion combinations of  screen flow, messaging and visuals is a time consuming activity and it might drive you to madness. Leave testing 41 different colors of blue to the Google’s of this planet.

PS. here’s a list of A/B testing software you can use to make your life a little easier.