Thanks for signing up! Here's your first lesson. You'll also be receiving it via email shortly.

How to reliably develop experiment hypotheses

Lesson #4

Now that you have an understanding of what it means to be results-driven vs. data-driven you are ready for our lesson on how to develop experiment hypotheses.

Here at Taplytics we generally find that people are extremely excited about A/B testing. The desire to get going with testing projects is strong and only getting stronger. However, we also noticed that teams can find it challenging to come up with ideas for A/B tests, particularly in the early days.

Because of our experience helping so many teams A/B test, we've come up with a framework for helping to generate ideas, both methodically and consistently. It is a great way to ensure that no matter what A/B testing tool you are using, that you're using it to the fullest extent.

A/B Testing Idea Generation Framework


The point of this framework is to get you looking at your metrics critically and picking out the places in your funnels where there is a clear friction point. When you can do that methodically, you'll be able to come up with relevant A/B tests every month, every week, or every day, whatever cadence works for you.

#1 Set up tracking for your most important metrics

The first step to coming up with great A/B testing ideas is to track all of your metrics properly, and more specifically, to track the metrics that determine the success or failure of your app. This idea of which metrics to track was covered in our last lesson.

The important thing to note is that you should probably stay away from acquisition rates and look at things like retention (how often a user comes back to your app) or sales. Acquisition is easy to fake and will be affected more by things you do outside of your app than things you do in your app. So don't look at acquisition rates if you want good A/B testing ideas.

#2 Set up logical funnels

Once you are properly tracking your most important business metrics, the next step is to set up logical funnels for those metrics. This type of funnel is meant to show you at every important section of your app, if people are progressing toward your key goals or if they are dropping away.

For retail apps, this is easy. You make your key metric sales, and you track the steps in the app a customer has to take toward a completed sale.

For other apps this might be a bit tougher, but don't worry, you can always modify your funnels later. Just pick what you think an important action is and the key steps to get there and make sure you are getting data about how many people make it from one step to the next.

#3 Observe your metrics within the context of the funnels

Now that you have funnels all setup and ready to go, all you have to do is observe them and the data they give you. Without too much time needed, you'll start to see some trends. The most important trend you will see, is where you are losing people.

#4 Isolate your problem areas

Now is a good time to take out your whiteboard. If you put your funnel information out on a big whiteboard for the whole team to see, it will be clear where the problem areas are. You will be able to know right away where people are dropping off and where they are breezing through toward your goal.

The important thing to note here is that there are natural drop-off points in any funnel. For instance, if you need to collect any user data at all, this will be a natural drop-off point. No matter what you do, fewer people will put their credit card information in then will click the 'add to cart' button in your store.

This is just a fact of life, so this whiteboard funnel will serve as a new benchmark. You'll know where your problem areas are, but you will not know the optimal state of every step yet, just a starting point.

#5 Design and execute A/B tests that target those problem areas

Take the worst performing areas of your funnel and take a deep look at them alongside all of the great metrics you set up. Maybe the worst part of your funnel is your credit card entry form. Now, this will always be a tough place to get people through, but I am sure that you can come up with ideas for how to make people's lives easier on this screen.

Maybe it is running an A/B test using Apple Pay for checkout, or maybe it is changing when in the process you ask for credit card information. Alternatively, maybe you just haven't told people enough about what you will be doing with their credit card information.

These are three great A/B tests that I came up within about thirty seconds after determining the hypothetical problem area. I am positive that once you have the problem areas defined, and you hold them up against your important metrics there will never be a shortage of ideas for how to make them better.

#6 Rinse and repeat

Now that you are running your awesome new experiments go back to Step 3 and observe your funnels and metrics again. The A/B test will tell you clearly, if you were able to increase your important metrics, and your funnels will show you if there is a new candidate for the major problem area.

As I said before some parts of your app will always be problem areas, so don't put all of your efforts on one screen or one step of your process. Make sure you are trying different areas and track your progress in how you are improving each piece of your funnel. If you see large gains in one area, and small gains in another you have probably found your "low hanging fruit". It might be a good idea to keep tackling the areas that are giving you the biggest gains until you start to see smaller improvements.

Wrap Up

This is not a one-size fits all solution it is just a framework for helping you to look at your app in a way that will make problem areas far more obvious. When problem areas are clear as day, it becomes much easier to come up with ideas for how to fix them. Do not let yourself get caught in the trap of trying to fix things just for the sake of fixing them. Stay focused on the metrics and the true problem areas, and you will be successful in your A/B testing endeavours.