The Best-Laid Predictions

Forecasting and future planning, in all their myriad forms forms, represents a struggle against an invariable truth: that the future is unknown.   At best, our most careful research and attentive modeling is based on the assumption that what is true in the recent past will continue to gold true in the future.  The most horrific blunders in history are laid upon that foundational assumption – but then, so have the greatest success.   To fail, to succeed, or to act at all requires some sense of what will happen in the future, particularly in reaction to something that does not exist at all in the present.

This can be difficult to observe in the aggregate: when financial plans fail, they do so with a shrug.  Something went awry “in the market” or “in the international economy” and there is not even the ghost of an attempt to recognize the flaw in the analysis – generally because in spite of the catastrophic failure, there are firm plans to apply the exact same analytical model to the very next decision.   The model was perfect – the world went wrong, and it’s assumed that the world will follow the rules next time and the model will work.

But where predictive models can be observed in a more granular manner, the inherent flaw of the present-future assumption becomes clear.    Ask an individual person what he plans to do in the future, and you can plainly see how he will speculate, confabulate, and make up plausible stories about a hypothetical situation in which he plainly has no clue how he will actually behave.    Even the most sincere respondent, reacting to the most unbiased instrument, doesn’t really know what he is going to do in a real situation that has not yet arisen.

When investigating behavior, you are implicitly making people think about things they do not normally think about – and often hectoring and harassing them until they lie in a manner that suits your preferences.    As a subject why he chooses to purchase a given brand of mustard, and he will shrug and say, perhaps, that “I like it.”   Press upon him, asking the reason he should like this brand over another that is virtually identical in all regards, and he will struggle to respond.   If you don’t like the lie he tells you, press him further until he tells a lie that you like – and then write that down and include it in a study where you declare, with pompous certainty, that you have identified “the real” reason consumers prefer one brand over another.       Use that as a basis of making your strategic decisions … and see what happens.

Consider that the very act of studious observation takes the customer out of the context in which a decision is made, if they ever are in such a context to begin with.   Consumer taste-tests of products rest on the assumption that customers consider the taste of a product to be the critical factor in making a buying decision – and do not consider for a moment that it will be any other factor.   In rare instances in which stores offer samples at the point of purchasing decision so that a customer can compare the flavor of two products they have not yet purchased, this may hold true – but this seldom ever happens.   And this is why Pepsi consistently beats Coke in blindfolded taste-tests, yet consistently sells less across virtually all markets where both are offered.

In one sense, asking the customers for their opinion seems a better course that relying on insider opinions – and it may be a more democratic method of deciding among options, but it is not necessarily accurate.   Customers do not know how they will decide in real-life purchasing situation and are speculating.   Observation studies can be very useful in exploring the present and recent past and, certain enough, the patterns of the past will tend to perpetuate if nothing else is changed – but they do so in a very superficial manner: one can easily observe and measure what is being done, but this does not accurately indicate the reason it is done – and it is the reason and the conditions, more so than the superficial act, that is most useful in the prediction of future behavior.

Design as Conversation

I’m doing rather a lot of remote user testing these days – people use a site to view information about a proposed product and respond verbally to what they are shown and asked.   Most of the respondents follow a predictable pattern, reading a question, giving a brief answer, and moving on – but one of them took an unusual and rather interesting approach: she had a conversation with the product page.

“You say you are a [type of company] and that you want to talk to me about a new product called [product name].    I understand the kind of company you are and I do not know anything about the product.” 

 “You say that this product is supposed to [description of benefit].  I understand what you mean, and I think that what you’re offering is interesting to me because [her reason for being interested] but I am also suspicious about [certain aspects of the agreement].”

 “You say that [key selling point] and I’m not sure I believe you.   It sounds good, but right now it’s just a claim you’re making without proof.  I’ll be listening to hear if you provide any support for that claim later.”

She carried on throughout the test like this, “you say” the various information we presented on the product information page.  “I think” about what her reactions were.   It was very thorough, detailed, and useful.

I’d like to be able to coach others to do this, but my sense is it would be weird and artificial for them, and it might taint the results.   But I do think that this conversational model can be flipped to talk about the product and page design.   “We are telling the user [this], and we expect them to think [that]” – or for more interactive models, to say “We are asking the user [this] and we expect them to say [that]”

I’ll mediate on this a bit more later – for now I wanted to document the experience and its potential for use as a design method.

Research for Innovation

While I am general a strong proponent of research-based approaches to innovation, I often see research being misused in a way that prevents research.   Either the wrong kind of research is conducted, or its results are used in the wrong way, and the result is that an innovation effort becomes an efficiency improvement: no significant changes are made, but the results are marginally improved.

This is because research, by its very nature, is focused on present practices and present results.    The researcher seeks to discover what people are doing right now (or more accurately, he seeks to discover what people have done in the recent past) and those who consume the research use it to propose methods by which existing customers can keep doing what they are already doing and new customers can simply imitate the behaviors of existing ones.   It’s more of the same, perhaps slightly different.

To be innovative, one must disregard what people are presently doing and define a new way to achieve the ultimate result toward which their behavior is intended.  That is, rather than considering the things that people are presently doing, an innovator must seek to discover why they are doing those things – what goal they are attempting to achieve by their actions.

It could well be (and often is) that people are doing the wrong things to achieve their goals, or at least things that are not as efficient or effective as they could be.     Even if people claim to be happy with what they are doing and the results they are getting, this is not to be taken at face value.  There is the psychological tendency to justify past decisions and defend current practices, as well as the tendency to satisfice and accept a margin of frustration or disappointment.

And if the outcome of an “innovation” effort is to keep doing the same thing with minor adjustments, then the customer is being supported in the existing behavior without a substantial improvement.   It will be easier to “sell” the new idea because it isn’t really new at all, and the customer will experience a slightly lesser degree of frustration and disappointment than he presently does, but with less need to change.

In this sense, most of the existing secondary research is not suited to innovation: it tells us what people are doing, but not why they are doing it.   And that information is difficult to collect and even more difficult to quantify.   It requires probing deeper than a multiple-choice questionnaire, and applying greater effort to the analysis of the data collected – which may be one reason that so few sources are conducting the proper kind of research for innovation, and cling to the methods that gain information that justifies a continuation of business as usual rather than disruptive change.

Research and Innovation

While I tend to be an advocate of research-based design, I also recognize that it is not a panacea.  In fact, there are some instances where too much devotion to market research can be harmful to design, and ultimately to the abdication of competitive advantage.

Research can be detrimental to innovation because, by its very nature, it is focused on “what is” and is blind to “what could be.”   It is not possible to research what does not exist, hence researchers focus on the known (the known needs of customers and the known needs by which they solve them) and is incapable of exploring the unknown (unidentified needs and undiscovered solutions).  Gaps and anomalies in research can indicate areas where the unknown might exist, but that is all.

Research can also be detrimental because it assumes that its subjects are knowledgeable.   Where there is any inefficient or ineffective mechanism, research asks the customer to identify a better solution – as if the customer is an expert.  An airline passenger may be acutely aware of a deficiency in his experience of air travel, but he does not have the domain knowledge to design a better airplane – and when asked to define one he will struggle to speculate and confabulate, and neither speculation nor confabulation is likely to yield a reliable solution.

And this is where innovation and research come into conflict: research seeks to understand what customers currently do whereas design seeks to provide capabilities to do something they are not already doing, and this may be something of which they are unaware and completely unable to conceive.   To innovate, one must set aside existing products and existing patterns of behavior and investigate what might be possible.

And again, this is not to advocate switching entirely from research-based design to innovative design, but to suggest that each has their purpose and should be used appropriately.   When innovating, a research-based approach is toxic; and when optimizing, a innovation-based approach can bear little fruit: it is a matter of having both tools at the ready, and choosing the right one for the job.

Three Sources of Innovation

One of the most common questions asked of a person who has presented an innovation is “where did you get that idea?”    It is, perhaps, an obvious question to ask because every act of innovation is based upon the application of an unusual idea, as usual ideas can be applied by the usual methods, and generally effect no substantial changes or improvements to business as usual.   And it is precisely because the ideas are unusual that makes innovation so difficult to comprehend and so difficult to practice.

There is no defined process that can be reliably applied to produce innovative ideas, so studying innovation often involves inspecting instances in which someone has been innovative and attempting to reverse-engineer the process to determine the source.    Having studied the topic for some time, I don’t have a clear answer – but I do think I have gained a perspective worth sharing, which points to where the answer may be found, or perhaps at least divined.

Specifically, I have come to the sense that innovation may be classified according to its source – i.e., the domain of knowledge in which an innovative idea takes root – and while information about the creative process is scant and often secretive, I’ve come to recognize that innovations tend to be rooted in one of three basic sources: engineering, marketing, and design.

Engineering Innovation

In the present age of scientific progress, many of the innovative ideas that impact the life of the average person are based on engineering.   I say “engineering” rather than “technology” because there are few instances in which a company invents a new technology and far more instances in which a company applies a technology that someone else has invented.  It is only when technology is applied through a process of engineering that it renders an innovation.

Engineering innovation considers the capabilities of technology and then seeks to identify needs to which these capabilities might be applied.   It is fairly easy to do, because the capabilities can be objectively observed: one can see that a thing has certain properties and merely needs to make a list of those properties and consider the way in which each might render some useful benefit.   This is not to say that engineering innovation is of little value and does not require hard work – because it is of considerable value and requires a lot of meticulous and deliberate analysis to arrive at an innovative conclusion.

However, it can also be observed that engineering innovation very rarely leads to radical or substantial innovations.  Instead, it tends to lead to technical or efficiency improvements that make existing solutions faster, cheaper, more efficient, more durable, etc.    Engineering generally does not result in a new device, but an improvement to a device that is already in use.   Its product is an innovative mousetrap, not an innovative solution to the problem of mice.  Engineering innovation is the most common form of innovation in the present day.  Witness that new devices are very rare, but better devices are very common.

Another distinctive quality of engineering innovation is that it is an entirely internal process, which can and must take place in a laboratory where specialists analyze solutions to problems (or discover problems for solutions).   This is not something that the average person can do because highly specialized domain knowledge is required to do it.   A person may recognize the desire for a more fuel-efficient automobile, but his desire does not give him knowledge of physics necessary to conceive of an efficient engine.

The drawback to engineering innovation is that it does not produce a unique or sustainable competitive advantage.  The qualities of a thing can be easily observed by anyone, and the “innovator” is simply the first to recognize the application of those qualities and bring them to market.   Any other firm that investigates the same phenomena will inevitably stumble across the same solution.

Marketing Innovation

An alternate approach to innovation begins with the customer rather than the product, on the plausible premise that people know what they want and would gladly embrace it if it were offered to them.   And so, customers are observed and studied, sometimes even included in product development exercises, so that the firm can find unique ways to provide the value that customers want of them.

The obvious problem is that the premise, while plausible, is quite flawed: customers do not know what they want – they know what problem they are trying to solve, but do not know what would solve them, and often speak in terms of their experience with existing products.  So the outcome is often an improvement to one of those familiar and existing products rather than a radically innovative product that is much better at solving their problems.

Take for example the phrase “If I had asked people what they wanted, they would have said ‘faster horses.’” While Henry Ford never actually said this, there is some truth to that: people are rooted in the known and can’t conceive of the unknown. While no-one believes that customers have the expertise to know how to breed a faster horse, it seems to be accepted that they have the problem-solving skills to know that a faster horses is what they need, when they clearly do not tend to consider other possible solutions to the problem of speed in travel.

This is not to say that market research is at all a bad thing, simply that the results of market research are too often taken at face value, with too little thinking about what desires or frustrations cause customers to say the things they do about existing solutions.   So a company that considers what the customer says he wants rather than trying to understand what he really needs will not be radically innovative as a result of this approach – they will simply uncover opportunities for minor improvements and adjustments.

Oh, and the distinctive quality of a firm that engages in marketing innovation is the involvement of the customer, through market research and participatory exercises – though attention must be paid to whether the customer was actually involved.   There are many who speak of “what customers want” while never actually having listened to the customer, and others who outright lie about having done market research when they are speaking from their own imaginations.

Design Innovation

A third approach to innovation employs a process of design, and as the word “design” is often misused, it’s worth pausing a moment to define it:  design is the process of creating or improving a practice by which a goal can be achieved. It is often misunderstood that design is about an object (a good) rather than the practice that is facilitated by the thing, and recognizing that the practice is the aim of design enables design thinking to be applied to all products (services as well as goods).

It is generally recognized that design innovations results in the most radical and substantial improvements to products because the process of design is parthenogenic and idiosyncratic.  Two engineers who look at the same technology are likely to come to the same conclusion as to how it can be applied, two marketers who use the same survey to study the same market segment are also likely to come to the same conclusion, but two designers who look at the same problem very seldom have identical ideas about how it can most efficiently and effectively be solved – hence their conceptions are often unique.

The distinctive characteristic of design innovation is the designer himself, a person who has the insight to formulate a wild and unusual idea and the charisma or power to get others to collaborate with him to bring it to  fruition.   Firms that leverage design innovation are often led by visionary figures such as Thomas Edison or Steve Jobs, who champion unpopular ideas and score dramatic victories when they succeed where critics were adamant that they would fail.

Unfortunately, that is also the chief drawback to design innovation, in that a “wild and unusual idea” is not necessarily a very good idea, and a designer who completely ignores the culture of his market, the capabilities of technology, the necessity of sustainability, and often the basic principles of reality, comes up with very impractical solutions that result in disaster rather than triumph.

Design innovation is extremely rare in the present culture, which clings to the tried-and-true and fears the new-and-unproven ideas that a design innovator suggests.  Skepticism is healthy when it causes wild ideas to be considered carefully, but quite deadly to innovation when it causes wild ideas to be dismissed without consideration – and the latter describes the function of most organizations: the larger the firm, the more defenses it has against radical innovation.

Classifying and Adjusting Innovation Efforts

People tend to be creatures of habit: they find that something produces good results and repeat that activity to get the same results, even in instances where another approach might be more fruitful.   This is what causes firms to settle into a routine and to fail to be innovative – but it is also the same quality that causes firms to gravitate toward a specific kind of innovation.   To determine how a firm innovates, consider how it approaches innovation:

• Technical – New ideas are discovered in a research laboratory that focuses on the capabilities of the product itself.
• Marketing – New ideas are discovered by surveying, interviewing, observing, or involving the customer, focusing on their preferences.
• Design – New ideas come from insiders, whether subject-matter experts or executives, by reconsidering how the customers’ needs can be met.

Then, consider what kind of innovations are the most advantageous.   Technical and marketing innovations tend to result in minor changes to the product that create a temporary advantage – but in some instances that is quite enough to sustain a firm.   Design innovations require major changes, which are not always desirable: the existing product may be selling profitably, the firm may not be able to absorb the effort and expense to make major changes, or the design innovation may be rejected by the market.  

With this in mind, the firm’s usual, familiar, and comfortable method of innovation may be entirely inappropriate (and ultimately ineffective) in a given situation, and the best approach is to keep all three tools handy so that the right approach can be applied to a given problem.   Often, multiple stratagems must be employed in tandem to achieve the best possible outcome.

Propaganda and Customer Education

It’s generally held that the amount of time that a customer will spend educating himself about a purchase is proportional to the risk involved in the purchase – i.e., a high probability of disappointment, particularly where there is such a significant investment of cost or emotion that making a bad decision will be damaging.    This seems to make perfect sense, but there is still the notion that customers make even major decisions by gut-feel.  I’ve researched the subject a bit, and have unconverted some interesting details:

First of all, there is a great deal of disagreement on what qualifies a customer as being “educated” about a product.   The general sense is that the customer has gathered enough information and analyzed it to the point where he makes the correct decision.   But the “correct” decision seems to be highly subjective: it is the decision with which the assessor agrees.   Ask any salesman or marketer, and the only correct decision is for the customer to purchase their product.

The best test of the correctness of a decision is the outcome – the solution solved the problem that motivated the customer to make the purchase.  Or said another way, that the customer is satisfied with the purchase.   This, too, is difficult to assess because customers who respond to satisfaction surveys most often give positive responses to justify their past decisions: admitting they were wrong or foolish, that they decided to purchase a product that turned out to be useless, is a shame that many people have difficulty admitting.

Ironically, this suggestion that customers overstate their satisfaction is followed immediately by the suggestion that they are often dissatisfied: that most customers make uneducated decisions and end up being disappointed by a brand, but place the failure of their decision on the brand itself: they claim the product is bad whereas it is really the wrong product because they made the bad decision.   

So it seems that a customer who didn’t buy their brand is uneducated, but so is the customer who bought their brand and is dissatisfied.   The only customer a firm will consider to be educated is the one that purchase their brand and expressed satisfaction with the decision.   It’s difficult to tell whether the sellers or the buyers are in a state of utter denial.

A second point of interest is the degree to which information sources are reliable.   One author cited a survey conducted by Ogilvy One Worldwide that suggested around 70% of the sources that customers use are “deceptive and useless” and provide “the wrong kind of information.”   But again, there is the subjectivity defect, as this was a survey among salesmen whose reaction is skewed to regard any source that does not support his brand as being a bad source – and further, that the salesmen are unlikely to know what sources a customer actually consulted.

There once was the sense that the Internet is full of mostly bad information that is misleading to the public – but this is a distortion.   There are indeed many bad sources of information, but they are not considered reliable by customers: they are quite capable of assessing the value of a source, recognize that someone’s opinion on Twitter is not as valid as the information on an academic or industry organization’s site.

The greater problem with sources is that they are simply not consulted: people make gut-feel decisions and do not bother doing research.   But it remains unclear whether they are doing so for purchases in which there is sufficient risk to merit that level of rigor.   Again, industry insiders feel that their products or brands are very important and customers ought to invest significant time in researching them – but this is not the way that actual customers feel about their products and brands.

I was only able to devote a few days to this research, but during that time I did not find a single source that seemed objective and reliable – so while I’m left without an answer, the experience has left me very suspicious about the entire topic of customer education.  Whether a given source believes customers to be educated or uneducated probably has more to do with the bias of the source than the behavior of the customer, and should be viewed with a jaundiced eye.