Personalization, Recommendation, Predictive Analytics & Serendipity

Continuing on from my last two posts — the theme of which was gathering internet and location data for behavorial and predictive analysis as well as the internet-as-collective-consciousness — I offer up an LATimes article that describes what has been going on for some time as we all have collectively been allowing organizations to “watch” what we’re doing online, gather our clickstream and transactional data, try to figure out who we are and what we want and to then monetize the knowledge they have gleaned from their observations.

Predictive analytics is the Holy Grail of the information technology category called business intelligence (BI). It’s purpose is to allow people to use patterns of data from thousands of people or transactions in order to make better and more accurate decisions. The point is to be more adept at building trend-right products and services while penetrating new markets. The goal is to drive top line revenue, decrease costs, gain competitive advantage, and all the other things that make a business grow and be profitable.

When at Vignette during the dotcom adventure, we OEM’ed NetPerceptions’ Recommendation Engine. It was, as this article points out, a “‘Preference engine’ (to) track consumers’ choices online and suggest other things to try.” It was a great technology and the Web site personalization (which Vignette offered) and recommendation (from NetP) was a powerful combination that delivered a highly customized and individualized experience for those companies who implemented our software and the users who visited their sites. Amazon was an early NetP customer and they’ve gone far beyond what was, at that time, fairly rudimentary categorization and simple business rules (more on that later).

But will automagic preference engines, personalization and predictive delivery work?

I think about what we used to offer in the good old days and how personalization, recommendation and predictive analysis could still come together and help all of us make better, more informed choices. Done right, we could actually deliver on much of what is described as being in the long tail. With these three coming together and my control over what’s publically known about me, extremely narrow, individualized content or products could be recommended based on my consumption patterns (web pages, blogs, podcasts, products), my public identity, how I ‘teach’ the system to learn, etc..

The LATimes article went on, “Preference engines emerged in the earliest days of e-commerce to boost sales — the Internet equivalent of “Would you like a belt to go with that?” — but they have improved with technology and incorporated human feedback to more precisely predict what someone might like.

Their spread worries some who fear that preference engines can extract a social price. As consumers are exposed only to the types of things they’re interested in, there’s a danger that their tastes can narrow and that society may balkanize into groups with obscure interests.”

I agree with “…there’s a danger that their tastes can narrow…”. I thrive on serendipity and the constant connections I make and it fuels my creativity and innovation.  Seeing things that others don’t has been something I could do since the first grade. Connecting the dots (hence the name of this blog) is vitally important to me so that I’m NOT a passive audience member that allows influencers to guide me or to tell me what’s important and worthy of my attention. This is one reason I’m bored with most television, radio and other things considered mainstream.

Who are these influencers? The people behind-the-scenes making assumptions, creating a framework for content delivery, building in the business rules for the preference engine, and/or analyzing data to make predictions. As we’ve seen by the extremely strong demand for desktop search (in Mac OS X Tiger, with Windows 3rd parties and maybe within Windows Vista at some point), humans are really bad at presumptively figuring out what a taxonomy should be, actually using it, figuring out what people ‘might’ do and attempt to set up a framework that will use business rules for guidance. The secret sauce seems to be some structure with a dash of serendipity thrown in.

Advertisers and marketers want desperately to have a way to deliver influencing and informational content to people that want it or would even be open to considering a product or service. Even I like some ads related to technology, cars, health, family, housing, and media. I just don’t want to be exposed to crap I wouldn’t dream of buying (which is 90% of the ads I’m exposed to).

People are already drowning in content and availability of choices. Most of us will just shut down if we’re drowning in possibilities (and media) since just figuring out what should have our attention is already extraordinarily difficult. Whoever figures out how to create a methodology or system to address this problem of giving me what I want, when I want it (attention.xml is a start) with some serendipity thrown in, will get rich.