How to do A / B- split testing: tools and popular hypothesis

Hypothesis is what sets A / B-testing from a trial and error approach. I laid out the whole process and identified 22 hypotheses that often impacts outcome. Certainly I tried to keep it simple.
What sets A / B-testing apart from a song?

1. The main principle of A / B-testing (split-test) is to produce a different version of the main page. For example, if page A is the original page then page B is a test version that differs from A in a parameter. Visitors that visit page B never reach A, except when they switch computers or devices. When the numbers of visitors reaches a certain threshold, e.g. 10,000 visitors, they compare conversion rate of the two pages and choose the most efficient.
2. A/B-testing includes a hypothesis that is tested in real life settings. If you do a test without a hypothesis, just out of curiosity, you waste your time - you can't draw any conclusions about your target visitors, even if your test is a success.
3. You can make a single test at a time; otherwise it's hard to determine what impacts outcome.

A / B test preparation
It is necessary to determine the main parameters prior to the test. As a rule, the following parameters are monitored:

• income;
• transactions;
• purpose;
• session duration;
• failures;
• page views
• Review seasonality - do not run the A / B testing during high or low season, it can significantly skew the results of the test.
• The last step in preparation - definition of the sample size. This helps to accurately calculate the number of visitors who visit the site in order to determine the difference in the main measurable indicators. You can use a special calculator - Driverback.
• For example, the conversion rate of the current web site is 4%. The minimum number of visitors to calculate 25 % difference in conversion is 6,238.

How to make A / B-testing with Google Analytics

1. In Google Analytics, go to "Behavior" then "Experiments", "Create an experiment."
   
2. Name the experiment, the purpose, the percentage of traffic in the experiment.
   
3. Turn off the even distribution of traffic between all the options (read Google Help for further information), set "minimum time" - two weeks and reliability threshold of 95%. It should work for most tests.
   
   It's important to set time frame for 14 days but the test should continue until you reach the required sample size. Even if you think that the outcome is obvious.
4. Specify the source and test page
   

How to come up with a hypothesis
There are two options to come up with a hypothesis - you can take one that comes to mind or draw it from a real life problem. For example, visitors don't go to the end of the page or small number of them use Order button. The hypothesis can be like this - if you move the button to another place, the conversion rate increases:
Formulation of hypotheses is not the goal but rather finding a problem or idea that would become the foundation for the hypothesis
Options:

• Analysis of key performance metrics of the site through analytics systems. Find the page with lowest duration of the sessions, a large percentage of failures, low conversion rate.
• The analysis of heat map and scrolling maps.
• Visitor surveys and review of frequently asked questions from consultants and technical support
• The analysis of competitors' fancy features.

What to look for in the analysis? You can test everything, but should you? There is no point in wasting time on changing the text that nobody reads or the appearance of the footer that only 10% of the visitors ever reach.
The following options are frequently tested:

1. CTA-button (text, appearance, location on the page)
2. Titles and descriptions of products.
3. Form (appearance, number of fields)
4. Images, video
5. Layout and page design
6. Prices

Which hypothesis to test?
Heat map hypothesis
A heat map is a tool to measure and display statistics of clicks, there are a lot of tools online that provide this service, e.g. heatmap.me.
The heat map is shaded in different colors, depending on the click frequency. The more frequent the clicks the warmer the color, the less the colder.

Main hypothesis

1. If users click on an element of the site and do not receive the expected result - it is necessary to make changes to meet their expectations. For example, you can find the images that visitors click to go to another page. You can create image data links and reduce the bounce rate.
2. Any elements that visitors click should be improved.
3. Elements, which account for greater number of clicks and are popular with users should be move up and to the left.

Scroll map hypothesis

Scroll map hypothesis help to analyze how users' attention is distributed in different areas of the page. It also shows the average time and the number of views a certain area of the page - there are lots of resources on the web.

If you scroll the map shows that users do not scroll down the page then you need to:

1. Move important elements (buttons, or form) to the top of the page
2. Zoom page length.

Buttons hypothesis

Problem - not enough clicks on the buttons. What to do?

1. Change the text on the button
   
   
2. Change position.
   
   
3. Increase the size of a button.
   
   
   
4. Change the color of a button.
   
   
   
5. Change the text color.
   
   
   
6. Add an icon next to the text of a button.
   
   
7. Add hover-effect when you hover over a button.
   
   
8. Add a pointer to a button.
   
   
9. Increase the free space around a button.

All of these actions are intended to highlight the button on the background of the rest of the page. Make sure that it is not out of the general color scheme of the site.
In order to verify the quality of selection, you can use a bookmark Grayscale CSS" . It helps to determine which elements are visible immediately and which aren't. Just add this code in the bookmarks bar and click it, open the page.

javascriptfunction(){var e=document.body;e.style.filter="progidXImageTran sform.Microsoft.BasicImage(grayscale=1)",e.style.f ilter||(e.style["-webkit-filter"]="grayscale(1)",e.style.filter="grayscale(1)")} ())

Titles hypothesis

If a title does not attract attention, try the following:

1. Change the title and optimize the text structure. Excellent templates for a title in English are here [LINK HERE].
2. Change the length of the header. Moz experts recommend to limit it to 60 characters, despite the expansion of Google's search results column.

Image hypotheses

1. Large and high-quality images increase the number of clicks on the page of a card catalog.
2. It is better to use your own photos and not stocks. It is also desirable that the people in the pictures were about the same age as the target audience.
3. The images without any extra parts on a light background allow focusing directly on the product.

Forms hypothesis

• The smaller fields in the form the greater the percentage of people to fill them. There may be exceptions in specific subject areas.
• If the form contains a large number of fields, and none of them can be removed, it's better to split the form in several steps.
• The field with a promo code can be removed (or decrease its size).

How to determine results

When the test is complete, you need to review the significance of the results. If the result is not statistically significant, the changes do not affect the estimated rate of success.

If the test is successful:

1. Apply the changes to the site
2. It might be possible to implement similar changes to other pages - you can make another test to test a new hypothesis.
3. Go to the next A / B testing

If the test is a failure:

1. Analyze the data used in the preparation of the hypothesis and with your new experience try to improve the hypothesis
2. Develop a new hypothesis
3. Run a new test.

Conclusions

1. The main difference between A / B-testing and a trial and error approach is a hypothesis.
2. You need to determine key metrics before your test and take seasonality into account.
3. In order to properly formulate a hypothesis, it is necessary to analyze key indicators of the site, such as heat map, scroll map and review fancy features of your competitors or conduct a user survey.
4. The first is to test the hypothesis based on the analysis of the heat map, scroll map, the hypothesis for buttons, headlines, images and shapes.
5. After the test is complete, review the statistical significance of results and introduce a better version of the page. Or move to a new A / B-test