There is much to be said for testing user experience design in the wild – it has a number of advantages if used properly, but using it properly is something of a delicate matter – but for the moment, I’d like to focus on the various kinds of tests that are run in a production environment, as there seems to be a great deal of confusion among them.
Aside of being persnickety about calling things by the proper name (many people use “A:B Test” when what they really mean is a multivariate or champion-challenger test), it’s also important to make the distinction so that you can plan the right test to get the information you need and relate the results in a meaningful and accurate way, using each method to its best advantage.
A champion-challenger test is the most commonly used variety. It compares an as-is design to a proposed to-be design to confirm that the desired effect on user behavior is effected. Consider these two examples:
The “challenger” version of the element has many differences to the “champion” version: the headline is different text, font, and color; the call-to-action button is a different color, a different phrase, a different size, and in a different position; the image is different; and the copy is different. As such, if you should notice a difference in their performance, you have no idea which of these changes caused the difference to occur.
That’s not necessarily a bad thing – since the ultimate goal is to increase click-through, you likely do not care which of these elements is to credit, so a champion-challenger test effects a one-time improvement but does not “teach” you anything that could be used to improve any future design.
The A:B test is the one most often claimed, but often mistakenly so. It is misunderstood because the parameters of an A:B test are very stringent: only one thing, and one very specific thing, can be tested. Consider these two examples:
The only difference between them is the color of the button. If you change anything else about the layout, or even another aspect of the button (the words, the size, or the position), you are no longer running an A:B test but a champion-challenger test, with the combined impact of all changes commingled and inseparable.
The greatest benefit of an A:B test is that it supports not merely a one-time decision (this layout is better than that one) but a general observation (green buttons perform better than blue ones) that is very likely to be applicable across all such layouts on your site, and possibly across multiple sites.
A multivariate test combines the best of both worlds: it enables you to make multiple changes to a layout, but test them in such a way that you can identify the degree of influence each change has on the outcome. Consider these examples:
These four variations test two elements: the wording and color of the button. By doing so, you can be informed not only that one does better than the rest, but also the influence of each factor. That is to say that A:B testing might suggest a green button is better than a blue one and “get info” is better than “learn more,” leading you to conclude that a green “get info” button is the best of all – but when you actually test every possible combination, you may discover that the blue “learn more” button outperforms is.
This arises due to covariance, a statistical concept that is best explained as being the combined effect of many different factors, and it is ultimately what is significant to the user, who does not mentally dissect your layout into its component elements but experiences it as a whole, whose impact may be greater or lesser than the sum of its parts.
Multivariate testing yields the same general result as a champion-challenger test, identifying the combination of elements that has the greatest impact for a very specific instance. And because the influence of each factor is examined, it also has the benefit of providing the general insight that would be yielded by multiple A:B tests of each factor.
Informed Decisions, Informed Interpretations
It is important to make the distinction between types of tests in order to decide which kind of test is appropriate to the information you wish to learn, as well as in knowing whether you can rely on test results to guide future decisions.
For example, if you run a champion-challenger test in which the champion has a blue button and the challenger (which wins) has a green button, you cannot accurately or reliably state that green buttons outperform blue because you do not really know which of several changes effected the difference in outcome. Sadly, claims of this sort are very frequently made, and the result is proceeding boldly on bad information.
Of importance: consider (and report upon) each test for what it’s actually worth:
- A champion-challenger test enables you to make an accurate one-time decision but yields no reusable observations
- An A:B test enables you to make a precise observation that may be too limited or granular to make a practical decision
- A multivariate test supports both one-time decisions and general observations, plus it tests for covariance among the factors
Thus considered, the multivariate test would seem to be the best of the three in terms of outcomes – but depending on the decision with which you are faced, one of the other two may be more relevant, and so long as you are aware of what the results actually mean, may be entirely acceptable to a given decision-making scenario.