Don't Even Test That

I've lately been engrossed in optimization testing, and have noticed that some design choices that seemed awkward have had highly positive results.  For example, a phrase that seemed a bit stilted and technical was suggested for a button to begin a purchasing flow.  When first it was suggested, I thought "that probably won't work" but in spite of my reservations I argued to include it in an A:B test if for no other reason than to get actual evidence that it wasn't a particularly good idea.   Problem is, that didn't happen.

This clumsy phrase that I would normally have rejected effected a double-digit increase in conversion rates.   And while I am delighted with the results (and happy to gain a halo as the person who fought to keep it in the test), it have given me some angst.   It hasn't undermined my confidence or left me paralyzed with self-doubt, but it has given me the sense that I should be a little reluctant to trust my intuition and be more open to testing things that seem a bit awkward or unusual.

But at the same time, I am cautious of going to the opposite extreme and signing off of spaghetti tests - where you exercise no discretion and test everything that comes to mind to see what happens.   Most of the time, theory bears out in practice and professional instincts refined by decades of experience are reliable - so it's good to trust your gut.   But where do you draw the line between standing in the way of boldfaced stupidity and being too obstinate to try something that seems awkward but just might work?

It's likely I will be meditating on this for quite some time, and while I am likely to reconsider some of my hesitation there are other instances in which I am likely to remain resolute that some ideas are so entirely awful that they should not even be experimented with.

Consider the illustration above, in which a red octagonal sign is used to communicate a speed limit.   That is a truly awful idea and should not even be tested.   People will come to a dead stop on a busy street because they think it is a stop sign, and over time they will start speeding past actual stop-signs if the misuse of that color and shape loses its association of the "stop" imperative.

That example may seem utterly inane, but I get something similar at least half a dozen times every year: "Let's put this information in bold red text so that people will pay attention to it."   Bold-red messages are used to call attention to critical errors on most sites everywhere on the Internet - they are in effect stop signs.  And using bold, red text to highlight anything that is not a critical error may get attention for a short time time, but over the long run will break the connection, such that people stop paying attention to truly critical errors.   So don't even test that.

Another example, from personal experience, was a clothing store that placed a button beneath the image of a dress shirt that read "select your size" - clicking on that button added the shirt to a shopping cart, where I had to click "edit item" to be able to indicate the size (and it would have shipped a "medium" by default.

In this instance, it was just rotten design likely caused by a bad ecommerce infrastructure that was not built to enable the shopper to select a size before adding an item to the shopping cart.   But it is entirely conceivable that someone might think it a good idea to force customers to add an item to the cart before selecting their size to increase the likelihood of their actually purchasing it.  And it might even work.

Even so, it would be a truly rotten idea that doesn't even merit testing.   There will be many customers who are jarred by he experience, feel a sense of doubt and anxiety, and leave the site immediately.  Worse still there will be other customers who are not upset, but don't fiddle about to discover the awkward way they select size, and end up getting shipped the wrong size and have to return the item to the vendor, which is a significant expense and a horrible customer experience.

There are likely other examples of ideas so awful that they should not even be tested - but I don't think it merits the time to come up with a lengthier list, as it would likely not be comprehensive or entirely accurate.  Instead, I think it's more productive simply to adopt the general principle of "trust your instincts but do not be ruled by them."   That is, be willing to try new ideas that seem a bit awkward, but be wary of those that seem thoroughly bad - to an experienced person, the distinction should not be that difficult to make.