Monday, June 22, 2009

Search evolves: Google, Alpha, Hunch (& Bing?)

Search has become to the World Wide Web what word processing was (and largely is) for desktop computing: the single most important application. Before the advent of the Web (and the proliferation of networking) the most common use for personal computers was word processing; the purchase of a PC was justified on the basis of just this one application (and, perhaps, spreadsheets, among the bean-counting set). Now nobody I know justifies buying a PC in order to do searches on the internet; rather, people are interested in email and in surfing the web. But you can't just 'go surf the web': you need a place to start. And that place invariably is the search box of a Search Engine.


While it may not be the First Internet Search Engine Ever (FISEE?), Yahoo! was the first successful and iconic one, which made its founders very wealthy and caused venture capitalists to open up their wallets and purses to hordes of prospective Web 1.0 success stories. As I recall, back in 1993, when yahoo was still a graduate student thesis in Stanford's computer science department, it started off as merely a list of all existing websites (of which there were fewer than 100 then). As it evolved, it became a combination of a search box and a list of potential links that surfers could visit by directly clicking on them. Yahoo! both categorized websites and also allowed free form search.

Yahoo! was followed by up to a dozen other search engine efforts that used different algorithms but presented the same paradigm of a search box and a list of categorized links.

Google, another Stanford computer science production, went the other way, simplifying the interface down to its bare essentials: all that the user saw was a single search box into which a search term was typed and in response to pressing the enter key, the search engine presented a massive list of hits ordered using proprietary, and frequently tweaked 'Page Rank' algorithms. A later player in the game, Microsoft, copied the same approach, but used different algorithms. The Big Three of Search: Google, Yahoo and Microsoft, have dominated the search space and have used largely the same approach of presenting the user with a long list sites ordered by some criteria. This situation has persisted for nearly a decade.

Come 2009, two new upstarts have surged onto the stage employing radically different approaches: Wolfram Alpha and Hunch. (There is a fourth entrant, Bing, but more about them later.) Alpha and Hunch represent completely new approaches that break away from the dominant paradigm represented by Google and Yahoo (and even Bing, for that matter). Alpha and Hunch embody the first entirely new innovations in search in nearly a decade. Alpha, in fact, is not search at all, strictly speaking: it is a "computational knowledge engine". Alpha computes answers from a vast, and rapidly growing database maintained and curated by sophisticated professionals. Alpha focuses on facts, and systematically organized knowledge. From a blurb on its website:
Wolfram|Alpha's long-term goal is to make all systematic knowledge immediately computable and accessible to everyone. We aim to collect and curate all objective data; implement every known model, method, and algorithm; and make it possible to compute whatever can be computed about anything. Our goal is to build on the achievements of science and other systematizations of knowledge to provide a single source that can be relied on by everyone for definitive answers to factual queries.
Alpha tries to answer any question that requires computation of some sort. Sample topics include:
  • Mathematics
  • Engineering
  • Dates & Times
  • Physics
  • Money & Finance
  • Units & Measures
  • Chemistry
  • Places & Geography
Hunch, goes in pretty much the other direction. It relies on crowdsourcing to gather knowledge and then answer questions input by users. Again, from the blurb:
Hunch is a new way to help people make all kinds of decisions, such as:
  • Where should I go on vacation?
  • What's the best US college for me?
  • What kind of smartphone is right for me?
  • Which museum should I visit in the Netherlands?
  • What blog should I read?
Results are based on the collective knowledge of Hunch's users. Hunch already has more than 2,500 possible topics, and Hunch users add new topics every day.
Hunch works off the same paradigm as Wikipedia, while Alpha's approach is more Encyclopedia Britannica, which is ironic, since the former seems to have beaten the latter in the encyclopedia game. But both likely will have their own constituencies and will complement rather than supplement each other. Google doesn't work to create knowledge bases: instead, it indexes every page that comes online and is publicly accessible and performs lexical searches in its search index in order to deliver responses to search terms. Note that Google is search term oriented -- queries are not asked, but kind of implied. Alpha and Hunch, on the other hand, are query oriented. They explicitly assemble knowledge and attempt to answer only that subset of queries that match the data available in their knowledge bases. Only Alpha systematically organizes and curates its knowledge.

UPDATE June 24, 2009: To (over)simplify,

Alpha says,
As long as it is something computable, if I understand your question, I can figure out the answer. And only I. I'm not going to let the riff-raff mess with my pure and elegantly computed knowledge. And any question I cannot answer isn't worth answering.
Hunch says,
I'll ask everybody what the answer is and maybe they'll know. Please ask everybody to join in the fun and together we'll be able to answer all of everybody's questions. Maybe. Eventually.
And Google says,
Let me parachute you into my vast junkyard, somewhere in the vicinity of where I think you might find the answer to your question. Or not. And if you don't find it today, come in tomorrow. Or the day after. Keep coming, you might find something you like. Or look at everything differently, and maybe you'll find your answer in there someplace.
Both Alpha and Hunch were founded by well-known (and colorful) personalities from the world of computing and the internet: Alpha by Stephen Wolfram, the creator of Mathematica, and Hunch by Caterina Fake, who founded Flickr, later acquired by Yahoo! The core computational engine used by Alpha is Mathematica, while the Web 2.0 ideas that were first seeded in Flickr form the basis for Hunch. It appears that powerful ideas cast long shadows.

I don't see either Hunch or Alpha replacing Google. Given that vastness of the web universe, there is far more out there than can be captured within finite (if growing) knowledge banks such as those of Hunch and Alpha. But clearly, if Hunch and Alpha can answer 80% of the questions people ask everyday (prices, schedules, feature comparisons, etc.) then they can capture significant chunks of the search market, and still leave a lot of room for Google to sweep up all the rest. It is also possible that Google (or Bing, or Yahoo) will buy out Hunch and integrate it into their search engines. It's unlikely that Wolfram will sell Alpha since it is founded on Mathematica.

The good news, then, is that innovation is far from dead in the search segment. Perhaps there is a lot to come in this area in the coming years.

And oh, about Bing -- it seems stuck in the old paradigm for an engine being presented as something new in 2009; unlike Hunch and Alpha, it's trying to out-Google Google. I'm not sure Microsoft's old trick will keep working over and over again.

No comments: