Web 2.0 Conference: The dirty little secret

Lots of buzz in the blogosphere (and here, of course) about being very lean-n-mean on building your startup and building cool web services (Flickr, Zimbra, Rollyo, Tribe, Socialtext, Joyent, zvents, et al) and getting stuff shipped so to speak. At the same time, Yahoo, Google, AOL and IAC were showing numerous offerings that are consumer targeted, web services. Very cool stuff introduced by both startups and big guys.

But the dirty little secret is that a lot of plumbing stuff isn’t being discussed on blogs or at this conference…and I gained some interesting perspective over the last few days as well as at breakfast this morning. I’m a glass-is-half-full guy, but can’t ignore the holes and what’s necessary to make Web 2.0 a reality. Things discussed — which are enormous holes needing to be filled include:

• Identity management. Dick Hardt’s sxip is an interesting offering and will be successful in many ways. They see the problems, “get” the scale and scope of the technical, political and funding friction that exists, but are a for-profit organization that may-or-may-not get the support of the community (look how Microsoft Passport was crushed by the lack of adoption by the community!). Without “trust but verify” directory/identity management services, true consumable web services will be low level only. Healthcare consumer services won’t fly, commerce will suffer, unless there are trusted networks and directory services that providers and customers can share.

• All Your Data is Mine. There was a fair amount of debate here about the collection of personal clickstream data by Amazon, Google, Yahoo and others…and that that data should be “free” and manageable by the owner. Sitting by Microsoft CTO Ray Ozzie during a panel discussion that included an Amazon guy expected to answer “yes” to free’ing customer data, I leaned over to him and said, “Maybe I’m not getting it, but why would Amazon willingly give up an enormous competitive advantage and make it free?” “They couldn’t” he responded. I agree obviously since what would Amazon’s incentive be to do that?

Investing in the infrastructure and software necessary to offer premium and high value services means that one return on investment for all companies (like Amazon) playing will need to include be the harvesting and use of this data (retail stores and malls have done customer observation, traffic flow and purchase analysis — and matching it to credit portfolio analysis — for YEARS so this isn’t new…just more transparent). Admittedly balance will be required so that Big Brother doesn’t emerge from the corporate world vs. the government — but educating the consumer of the threat *and* providing them with an alternative will be the only way this will fly.

• Latency. This is a technical problem. Imagine you have a portal that is “consuming” web services from a bunch of different sites. You’ve undoubtedly experienced ONE web service in the past (DoubleClick ads) where the web page “hung” (didn’t parse) waiting for the DoubleClick service to deliver the ad. Now imagine that your blog or web page is grabbing photos, catalog items, maybe audio or video, blogrolls, calendars ALL from different web services, and you end up with one incredibly horrible user experience!

BitTorrent is an amazing technology for moving huge files around with minimal cost and impact with internet transfers. With huge momentum around digital use leveraging all these cool web services, audio (e.g., podcasting), video (e.g., Brightcove) which will accelerate internet use and massive downloading, exacerbates the latency problem mentioned above.

Latency and Open Infrastructure: One plugged-in visionary I follow (Marc Canter, founder of MacroMind, leader in the open infrastructure movement) has a blog that is, ironically, a poster child for latency. Besides my news aggregator, I visit 50 or blogs a day and — because Marc connects up to a wide variety of sites (i.e., web services sort of), I often find myself waiting and waiting while his blog loads all these piece-parts snagged from all around the Web. I find myself pissed off often with the slowness of his blog but tapping in to his knowledge and connectedness has, so far, made it worth the wait.

• Exposing data and functionality as web services. This is a big one since getting as many independent software vendors (ISV’s) to proffer up appropriate data so as to be consumed by web services is key. The ISV I work for has “punch out” partners so our customers can seamlessly use our procurement application, set up an account with a supplier, and punch out from within the application, buy stuff and automatically populate the purchase within our system. All these partnerships are one-off business relationships…effort and cost for us and our customers which could be obviated with a trust, secure, global web services infrastructure which we could leverage.

Lots to do. Lots to think about. Collective energy and leadership will hopefully drive standards and problems out of the way so we can deliver more innovation and functionality to users and customers.