I want to walk through a particular type of effort I recently put forth. The study had the following characteristics

• It tested the priming effect and the rejection then retreat principle
• It was an a/b test with a/b elements on multiple pages
• The expectation was that the conversion rate would increase
• The expectation was that the average revenue per transaction would increase

Google Website Optimizer

Normally an a/b test works by having one page with one element that is being changed. For example, if Google were doing an a/b test they might change one element on their home page. In this imaginary case the wording on the search button is changed.

However, Google Website Optimizer doesn’t make it easy to perform an a/b test that spans multiple pages. One easy way to do it is to copy and paste the pages and on the base page, the a/b code sends the user one way or another.

I have a few concerns with that approach:

• I take the URL to be a relevant part of the test
• It’s less maintainable
• Your users can share the test page with their friends

Instead, I prefer to have A/B elements on multiple pages. Now, you might be thinking that this should be a multivariate test, but it’s not. For instance, you might want to test the color of an element through a purchase process. That’s an A/B test, but not supported by Website Optimizer’s standard A/B tests.

To accomplish this, I did the following. First, I put this in the utmx_section for the experiment pages:

<!-- Don't do anything-->

You’ll also need to put the control script on all experiment pages. Then, in Website Optimizer I added the following as the variation code:

<script>
var row = document.getElementById('controlGroup');
row.style.display = 'none';
var testRow = document.getElementById('testGroup');
testRow.style.display = '';
</script>


I have the controlGroup and the testGroup elements in both pages, where the controlGroup is displayed by default and the testGroup hidden. So, when Google replaces my “Don’t do anything’ comment with the script above, it toggles the display of the two elements. The reason I implemented this strategy is that, while the elements are conceptually the same on both page, they aren’t identical. Thus, I cannot allow Website Optimizer to replace the elements in both pages with the same code. That is, leveraging this technique allows me to have different content on both pages in the flow.

One shortfall of Website Optimizer is that it only allows you to measure the conversion rate. I also need to measure the average revenue per transaction as well as segment my audience.

Google Analytics for A/B Tests

With Google Analytics’ custom variables, you can get a lot of data for your experiment. All you need to do is store one custom variable with one value for the control group and one value for the test group. Here’s how I implemented it:

<script type="text/javascript">
var combination = 'Control';
if (utmx('combination') == 1) combination = 'Test';

_gaq.push(['_trackPageview']);
</script>


This code replaces your normal tracking code for Google Analytics. I haven’t played with the ordering too much, but I believe the code needs to be after the tracking script for Website Optimizer. Using this code, you will have one new custom variable in Analytics named Combination with two values: Control and Test. (Click Visitors, then Custom Variables in the menu)

Combine this with your other segments, for example, custom country segments and you can track revenue, products purchased, transaction data, and much more.