A/B Testing UX Changes at Clearlink

At Clearlink, our product organization worked closely not only with the broader organization, but also worked closely with marketing partners in order to build new products, test ideas, and drive better business outcomes.

As part of this, we were often running A/B tests of marketing changes. We worked with our partners to create new marketing copy and then test out the changes over certain periods of time to see what images or phrases would work best for their sites.

But as I helped build out the product and UX organization, we wanted to expand from just running A/B tests on copy and images and use the same methodologies to experiment with user flows and the actual experience of the users on the sites.

This was a bigger deal, both for our partners and for our teams because it involved more work to create the changes, as well as more buy-in to allow us to run experiments not just with language or images, but with the actual experience of users. 

But we were adamant we could improve the user experience of their sites, helping to improve traffic and drive additional conversions. So we designed and created a new user flow for one of the sites that we targeted first. This was an online retail site that gathered information and then moved people to either sign up online or call a number to sign up.

Our hypothesis was that we could simplify the user flow, eliminating over two-thirds of the steps to get the user from first action to a purchasing decision, and increase conversions by requiring less of the potential customers.

I worked closely with our partner to overcome a number of obstacles and concerns, and we created prototypes, reviewed designs, and prepared stakeholders for the changes we were creating.

​(This may seem like overkill for some good UX changes, but we were dealing with a large telecommunications company that didn't like change and didn't like to move fast or try new things, hence the need to really get everyone on board)

Launching the Test

We ran a 2 week test of our changes versus the unchanged site. We randomly assigned visitors to one of the two sites, but also tracked unique users and attempted to give them the same experience if left and returned.

This specific site traditionally got thousands of visitors over the timeframe we were targeting, so we were confident in getting enough traffic for a meaningful result.

We used many of the same targets for other A/B tests that we ran for other marketing purposes, including a full testing period and appropriate confidence intervals to test for statistical significance. 

It's important to run a test for a full length of time, because stopping a test short can lead to noisy results or incorrect conclusions. This was certainly a temptation, especially with a partner who had reservations about so many changes and also would want to stop using one of the sites as quickly as possible if the other was showing significant good results comparatively.

​As you can see from this image, there is noise in data, and stopping an A/B test early can lead to making the wrong decision, which is something we reminded everyone of as we tested.

The Results

The results of our UX changes were even better than we expected. We had set a p-value of 0.05, but got a p-value of 0.01, suggesting statistical significance beyond random chance as we saw conversions almost double over the time period on our site with the UX enhancements versus the original site.

This was a validation not only of the changes we made, the designs we created, and the work we did to get the partner on board with the test, but also more broadly validated the work I was doing within the product and UX organization. UX changes--designing better experiences and not just nicer pictures or better wording--has a significant impact on the overall user experience and ultimately the business value. 

I continued to use this as an example of the value of product and UX as we worked internally as well as with other partners, to help everyone understand how and why we can focus on the users and why taking time to design the whole experience adds significant value.