Thanks for signing up! Here's your first lesson. You'll also be receiving it via email shortly.

What is A/B testing?

Lesson #2

In the first lesson we went through the basics of Mobile Performance Marketing. That should have provided you with some motivation to keep going and give you a few key things to think about. Hopefully, you're coming into this lesson ready to be be results-driven.

This lesson is all about giving you a rundown of what A/B testing is. One of the core pieces of Mobile Performance Marketing is A/B testing in all of its forms, so we want to set you up with some background information right away.

Ideally this lesson will dispell any confusion around A/B testing, teach you how you can use A/B testing for your benefit, and address some of the specific challenges mobile presents for A/B testing.

What is A/B testing?

An A/B test can be thought of as an experiment, and just like any experiment there is a control and a treatment group, a standard version and a changed version, an A and a B (see what we did there?). For A/B tests in our context, these experiments are run in a mobile app with the objective of determining which version (the A or the B) maximizes the goal or outcome of interest.

Mobile A/B testing is A vs B

Specifically I may run an A/B test on the checkout page of my shopping cart. The two versions of my test will be the version that I originally built into my mobile app with the second version including some change whether to the layout, flow, design or calls to action. The goal of this type of experiment will clearly be more people making it through the checkout funnel resulting in a greater number of purchases.

One popular misconception about A/B testing is that it is limited to small changes, like the color of a button, the call to action, or the position of some element. These are just simple examples, when A/B tests can actually be quite extensive, testing whole new features, flows or any combination of things in a mobile app.

Why is A/B testing important?

Some people ask themselves at this point, “is A/B testing actually important? Apple never A/B tested the iPhone!” To this I say two things; 1) we can’t all be Apple and 2) Apple built and failed with the Lisa before it built the Mac and succeeded. Bam, A/B testing!

Really that’s all A/B testing is, it is trying something new, gathering data about it and making a decision as to whether or not it's an improvement and then going with the best option. What we are encouraging more than anything is just being informed about how the changes you make affect your users, how they interact with your app and how their interactions change over time. We think this is just good practice and A/B testing tools help you to do this really easily.

What is different about A/B testing on mobile?

Mobile platforms create some very unique challenges. We know, we’ve been developing for iOS since the App Store originally opened. The biggest challenge with mobile apps is the review process. The review process creates a delay from when you create your change to when your users actually get to see it. This delay, and the resulting delay in information around a change, generally makes mobile development harder than web development. This means that it’s also harder to create multiple versions for parts of your app. So inevitably most people don’t A/B test on mobile.

With the new tools that are available on the market, like Taplytics, mobile A/B testing is now extremely easy to run. Just like on the web, there is now one-line SDK integration, and you can create experiments, set goals, define segments and declare winners all without ever touching code and without any App Store updates.

This takes away a major pain point in mobile A/B testing for all but the most complex tests. It also enables the entire team to be a part of building experiments. Whether they are designers, marketers or product managers. If they have ownership over the performance, or results of a specific part of the app, they can now manage the A/B tests that they need to.

What you should test

This is a difficult question to answer because every situation is different and requires different tests. Luckily this will be covered in some depth in future lessons on how to reliably come up wit testing ideas. Until then, you should take some time to read up on A/B testing from other sources, such as:

Always be Testing

You Should Test That