The technology of one-to-one marketing | 
 |
By Jesse Quatse (VP Product Operations at 7th Street Software)
Published by The Wise Marketer in August 2005.
There's an awful lot of technology behind true one-to-one marketing, and it's all aimed at refined targeting. But what are those technologies, and what does the future hold? Jesse Quatse explains...
If you search Google.com for the phrase "1 to 1 marketing" today, you're likely to get many thousands of results. According to Marketswitch Marketing Technical Overview, "1-1 Marketing is big business, accounting for approximately 50% of the marketing dollars spent in the U.S. each year." No doubt the phrase is rapidly gaining in importance and popularity. What it means is not so clear. Traditionally the term applies to e-commerce but is rapidly extending to bricks-and-mortar superstores. The common thread is the communication with a mass market on an individualistic basis, for whatever business benefits that may yield.
As Don Peppers and Martha Rogers put it in their book, The One to One Future: "Customers are different, and that's the whole point. To become a 1-1 marketer, you first have to recognize this basic reality." The problem is that a mass market cannot be understood individually without some very sophisticated technological support. No one person can know all of the customers of a mass market, especially without ever meeting them.
So it follows that the meaning of the phrase "1-1 Marketing" will depend upon the technology that supports it. Four major technological approaches and their variants attempt to address the problem of individual communications with a massive audience, each in its own different way and its own reasons. A technology that works well for one kind of 1-1 marketing may not in another. An overview of the differences and reasoning behind the different technological approaches can shed light on the meaning of 1-1 marketing and its potential. The mathematics of the technologies are not appropriate here, but a useful understanding of the algorithms can be had without the mathematics.
Development of Recommender and Targeting technologies
The phrase "1-1 marketing" implies some kind of individualized business communication from the seller to the customer to further some business objective of the seller. To do that in a mass market, mom and pop's direct familiarity with the best customers must be replaced by some form of synthetic familiarity displayable on a screen to prompt the sales clerk or operator, or sent directly to the customer by land mail or electronic channels. The communication could consist of any notice delivered by any 1-1 communication medium, but the prevalent technology was developed specifically to match an offer to sell with a customer most likely to buy. Software applications that match items to customers are generally referred to as Recommender Systems because they recommend likely items to a specific customer. Targeting Systems do the reverse by finding likely customers for a specific item. E-commerce stores like Amazon.com need to recommend likely items to a specific online customer, and physical stores like Safeway have a relatively small list of weekly offers to target to the most likely customers.
Recommender Systems began gaining importance in 1996 with the publication of the first world-wide trial at the "GroupLens" project of the University of Minnesota. Since then, ever improving Recommender Systems have been used to reduce the enormous number of choices available to a shopper on an e-commerce website. Notable examples are Amazon.com, E-bay, Moviefinder.com, and Reel.com. The growing popularity of internet is providing a rapidly expanding audience for 1-1 communications by the "bricks-and-mortar" physical stores as well as e-commerce. According to InternetWorldStats.com, there has been an average increase of twenty million more US internet users every year for the past five years .years. For the bricks-and-mortar store, that means significant growth of website and e-mail exposure to the customer base. These electronic channels can replace individualized postal communications at far less cost. In high volume, companies like MailWorkz offer e-mail delivery at around half-a-cent per message compared to the traditional land mail costs of US$0.70 or more per postcard . For whatever the business payoff, physical stores are turning to electronic 1-1 communications. According to Newspapers & Technology, online grocery coupon distribution is growing at almost 500% per year.
Successful targeting technology can payoff for the physical store by inducing the shopper to visit the store directly and therefore to spend enough to warrant the markdown cost. This in turn stimulates increased customer loyalty in terms of spending per visit, number of visits, category sales, and customer life time value. Thus 1-1 marketing that began with e-commerce now has two different purposes: to sell more from a virtual store online and to attract customers to a physical store where increased loyalty can justify discounted offers. Whether recommender or targeter, the essential figure of merit is the redemption rate, usually stated as the percentage of offers distributed that are actually acted upon by the customer. In principle, a coupon or offer is of no value if not redeemed. The reasoning is that an unredeemed offer is indistinguishable from an offer that was never made. Some advertising value may accrue from unredeemed offers but the main purpose of an offer is to be accepted.
The issue of human judgment
Recommender and Targeting Systems face an almost impossible task for a computer: that of imitating human judgment. Computers don't think, they execute algorithms. So when Amazon.com sells a copy of The Da Vinci Code, the reader knows why she bought it but Amazon.com's computers simply cannot. Yet the computer program must anticipate the judgments of the buyer in order to recommend other books that might appeal to her. The best that Amazon.com and other e-commerce sites can do is to find a computational way to rely upon the judgment of other humans who seem to be similar to the given customer. All successful e-commerce systems now base their recommendations on matching customer behaviours, sometimes supplemented by ratings or preferences explicitly stated by the customer. For example, the Amazon.com Recommendation Wizard prompts users to rate books, in their judgment, or to state their own specific interests or choose which Amazon stores they prefer. The customer's explicit ratings are not required, but when given they can improve the accuracy of the Recommender System developed by Amazon.com for their Your Recommendation service.
Four critical software processes
The four most important implicit software processes are summarized here as Clustering, Collaborative Filtering, CRM Queries, and the Statistical Inference technology (such as that found in 7th Street Software's own product, LoyaltySuite). Their common technological struggle is to somehow weave human judgment into a computational process, with as little explicit human effort as possible. Each process faces the issue of human judgment in an entirely different way.
- Clustering
Cluster Analysis is a collection of well known mathematical processes dating back to 1939, before modern computers. The purpose has been to identify groups of things having similar attributes when the attributes are unknown. These groupings are abstract in the sense that the software has no way of understanding human attributes. Clustering might simply treat each customer as a database record with demographic information such as address, age, income, and other more subtle factors made available such as rent or own, house vs. condo, average home value, children or not, white collar vs. blue collar, and others. The clustering software might then group customers by their individual attribute values so that all customers in one cluster appear to be more similar to each other than they are to any customer of any other cluster. The theory is that people who have similar demographics or life styles should like many of the same items offered for sale. The clusters are found by any of a large number of well known algorithms and variants that appear voluminously in technical publications.
These computationally intense methods were not widely used commercially until computers became plentiful, powerful, and cheap in the mid to late 1990's. Now complete application packages can be obtained for a wide variety of uses from major vendors like Claritas (PRIZM) and a very recent release from Fair Isaac (PEACOCK), and even an Open Source version (OSCAR). More generally, Oracle (Darwin) and Microsoft (SQL Server 2000) offer Clustering Analysis algorithms as data mining services. By one estimate, clustering is used by over 20,000 corporations in the US and Canada.
Clustering began to appear in 1-1 marketing after the year 2000. The answer to the human judgment issue was the old adage, "Birds of a feather flock together." The logic is that people of the same cluster appear to have similar lifestyles and so should have similar preferences. The mathematics assigns some abstract measure of "distance" between attributes. The algorithm groups people in such a way that the average "distances" between people in the same cluster are "shorter" than the "distances" to any person of another cluster. The process does not have very high accuracy, tending to form relatively few groups, each of relatively large size. For example, the clustering system called PSYTE separated 110 million Americans into only 65 clusters averaging 170 thousand people each. Even with this relatively low degree of individualization, clustering has some use in 1-1 marketing where the intended promotional targeting is not amenable to detailed customer individualization. For example, Mellon Bank says that Claritas P$YCLE clustering software made the direct mail promotions of financial products more efficient and cost-effective . Among other things, GMAC used Claritas PRIZM clustering to target messages and to clarify insurance cross sell and co-branding opportunities. One important use in 1-1 marketing is to trade-off individualization against the processing time of Collaborative Filtering by clustering in advance.
- Collaborative filtering
By far the most successful e-commerce technology for Recommender Systems is called Collaborative Filtering, or just "CF". The technology was introduced in 1994 and subsequently became the subject of most of the technical papers on computerized recommendation. The human judgment issue of CF is resolved by computationally imitating "word of mouth" recommendations between customers. The software identifies a few customers who seem to be making similar purchasing choices as you. It then recommends to you some of the purchases that they made but you didn't. Where explicit rating is included, each customer is asked to assign a figure of merit, say from 1 to 5, for each book read. The rating is then used by the software to limit recommendations to those most highly rated. In effect, CF tries to model a composite customer that is the most like you so that the human judgment represented by the past purchases of the model customer can be used for judging which books to recommend to you.
The software may be based upon any of several different mathematical methods. CF algorithms represent each customer as a very long list of items, typically the catalogue of all items on sale by the retailer. Some value is associated with each item on the list, for example the frequency of past purchases of each item by the given customer. The lists are represented mathematically as vectors and the so called "distance" between customers is computed as a function of the angle between their two vectors. The word "distance" is in quotes because it is an abstraction that can be visualized as a distance, but which is not a true distance measurable in feet and inches. The so called "nearest-neighbour" is then the customer having the smallest average distance from the given customer. As might be expected, the computational cost is high with hundreds of thousands of items for each of a million customers. A typical CF algorithm will identify only a few "nearest-neighbours" and recommend items that those customers rate highly but that the given customer has never purchased. Although the "distances" depend upon the items purchased, CF is matching the people, not the items they have purchased.
For example, Amazon.com improves the computational efficiency of CF with their Item-to-Item Collaborative Filtering. The method is described by Linden et al. in "Amazon.com Recommendations" as matching items rather than customers. All of the items that customers tend to purchase together are used to directly predict which items will appeal to the given customer.. An abstract "distance" between two items can be computed by counting how many customer bought both of the two items. The more customers in common, the shorter the distance. The past purchases and ratings of the given customer can then be used to recommend the most popular or nearest items. The algorithm is much more rapid in execution than the typical CF algorithms. Speed is important to Amazon.com because the customer is online, staring at the screen. The recommendations must encompass the most recent events and purchases in the real time of the waiting customer. Bricks-and-mortar stores do not require that rapidity because all of the recommendations are made before the customer enters the store. Thus a wider range of technological solutions are available.
- CRM queries
Cornell University defines customer relationship management (CRM) as activities that enable retailers to "better understand their existing customers," in order to "develop targeted promotions and services." That definition comes in two parts. The first is the understanding of existing customers. It is hard to imagine how that could happen without a CRM system or other data mining equivalent. Most large retailers use them, and the effect of loyalty programs are almost impossible to analyze and understand without them. Besides generalized products from Oracle, and Siebel, several systems specifically featured for retail are available, for example MarketEXPERT of Valassis, Blue Martini Relationship Marketing, and Epiphany Marketing.
The second part of Cornell's CRM definition is targeting. It is only natural that such a popular technology for analyzing customer behaviour should be adopted for influencing it. But 1-1 marketing targeting is not an easy thing with CRM. The human judgment issue is resolved by a human operator, typically an IT professional or specialist at the task, who passes judgment on which group of customers might be the most likely targets for each promotional offer. For each offer, the operator must try to envision a typical customer who would want to redeem the offer. The profile of that customer is stated by the operator in terms of customer attributes recorded in a customer data base. The attributes are cast as a database query by the CRM (or data mining, OLAP, etc.) targeting software, the results of which constitute a targeted customer list for the particular promotional offer.
The time spent on each offer is a major drawback. If a pool of 100 or 200 offers are to be individually targeted, the human operator must envision hundreds of distinct customer profiles and declare each on a database query form provided by the software. If the profiles are too rigid and restricting, too small a percentage of customers will appear on the resulting lists. For example only 10 percent of the customer base may match any of the profile queries. They could be the only ones to receive any offer of any kind. On the other hand, if the profiles are too broad, the individualization will suffer. For example, most of the offers could be targeted to everyone thereby negating the objective of targeting.
Another drawback is the "lumping" of offers that is hard to control. As an exaggerated example, 10% of the customers could receive most of the offers, 20% a few, and the rest none. These disadvantages can be partially relieved by saving trustworthy queries built up over time. Each stored query could identify customers who are to be targeted for one kind of item. The human operator could then use these previously developed queries rather than to continuously develop new ones. For example, a time-tested query named "diapers" could be used to target all diaper offers, but even then the reliability of the query would depend upon what other offers are being targeted at the same time. In an extreme case, queries that adequately separate out offers for ice cream from offers of canned vegetables may not work well when the other offers are all for frozen food and none for canned vegetables.
Rules based... The term "Rule Based" derives from artificial intelligence research but is basically just a variant of CRM Queries when applied to a recommender or targeter. In effect, a set of trustworthy queries could be viewed as business rules for targeting. Examples are Sagarmatha from Israel, and the newer net services company called Loyalty Lab. Such systems use queries of a data base and are therefore similar to CRM systems with especially strong support of preserved queries that they call rules. Many CRM solution companies provide such support whether or not they refer to them as rules.
In some systems, the rules must be constructed by the software provider. They are expensive and inflexible without bringing in the provider to make changes. In other systems, the rules can be declared and named by the human operator. These systems are very much indistinguishable from CRM query systems where the software provides convenient support for saved queries.
- Statistical inference
Several of the differences between e-commerce and bricks-and-mortar retailing can be exploited technologically to the significant benefit of the retailer or CPG. An entirely new approach, Statistical Inference, has been developed specifically for that purpose. As the name implies, a model of each customer is based upon the statistics of previous purchases and the probability of a future purchase is inferred from the model. The approach would not be effective in e-commerce. It depends upon several distinctive qualities of bricks-and-mortar
Physical stores typically target from a fixed number of offers, in the hundreds, as contrasted to the hundreds of thousands of items from which each e-commerce recommendation must be made. Those few offers are renewed typically on a weekly cycle as contrasted to the e-commerce constraint of interacting with an online customer in seconds. Together the difference admits much more complexity as a trade-off against available computation time. More complex algorithms can provide much greater individuality and automation.
A second difference is as significant as complexity. Online stores normally sell a book or a CD only once to each customer. They cannot succeed by repeatedly recommending the same items already purchased. By contrast, people buy the same consumables, like bananas and cold medicine, over and over throughout their lives. It follows that the recommendation process for the physical store can relate the recommendation directly, and therefore more accurately, to the previous purchases of the same customer. It need not rely upon projected purchases by some theoretical customer constructed by the software.
The third important difference resolves the human judgment issue without additional human interactions. Purchased items are identified computationally by an abstract code that distinguishes the item from all others. Large retailers like drugstores, grocery chains, and beverage chains create nested categories for tens of thousands of items, referred to as the Product Structure. For example, Safeway lists 10 different items in the category of frozen beans, which is a category of vegetables which is a category of frozen foods which is a grocery. The Product Structure associates the meaningless product identification code with meaningful categories. That association obviates the need to match customers as is required by Clustering and CF as well as the item by item judgments of the CRM Queries user. It is possible for the software to statistically understand the intention of the buyer because the retailer built in the human judgment when originally creating the Product Structure. In that sense, the human judgment comes free.
In 7th Street Software's own LoyaltySuite Statistical Inference product, the automation process is based upon a mathematically sophisticated model of the purchasing behaviour of each individual customer for each item being offered. The mathematical methodology, known as "Empirical Bayes", fills in for rarely purchased items and significantly improves the accuracy of the predictions in general. The mathematical model is used to rank each item to be offered in order of individual preference of each customer. Only the top ranked offers for each customer are then sent through web sites or any of the electronic or paper communication channels. In that way, the physical store can avoid spamming or other annoying practices by imposing very few highly relevant offers on each customer.
The technological future of 1-1 marketing
The history of 1-1 marketing has been very closely tied to the technological developments. Technological advances will continue to shape 1-1 marketing in the future. The 1-1 recommender systems were first developed to meet the growing popularity of e-commerce. Other technological progress extended the benefits of 1-1 marketing to bricks-and-mortar stores. Initially traditional data mining technology was applied to the problems of the physical store. In the immediate future, the new Statistical Inference technology is available specifically for 1-1 marketing at the physical store. The future promises greater precision and more hands-free automation as elements of CRM and CF merge with statistical inference technology.
Loyalty marketing... for real facts, figures, research, case studies, best practices, practical how-to's,
technologies & examples, The Loyalty Guide 4 is the world's most complete report (1,000+ pages) that
covers it all. Costing less than a conference pass, details of this electronic report's contents, chapter samples,
pricing, and ordering details are online now at
www.TheLoyaltyGuide.com.
About the author...
Jesse Quatse is the vice president of product operations for 7th Street Software (recently acquired by Pay By Touch). Jesse is the principal inventor on a pending patent for a new technology in the 1-1 marketing space, has a Ph.D. from Carnegie-Mellon, and has been on faculty at UC Berkeley.
7th Street Software markets its LoyaltySuite loyalty marketing products that automatically target one-to-one promotional offers to individual customers. For more information, or to contact Jesse Quatse directly, visit the Pay By Touch web site at http://www.paybytouch.com/business/pdf/PBTPersonalizedMarketingSolutionSheet.pdf.
Looking for permission to reproduce this article?
We are usually pleased to permit the reproduction of our
Feature Articles in other trade journals (but not 'blogs').
If you would like to reproduce this article, you will need to
obtain written (i.e. e-mailed or faxed) permission from our
publisher (Wise Research Ltd) as well as the original author.
Contact the publisher by clicking here
and contact the author by clicking here
Please explain your intended usage in your requests.
|
