Predicting what clicks with consumers

Predicting what consumers want is as elusive as creating a Hollywood blockbuster. You just don’t know what will click, until it does.  

For marketers and advertisers, the challenge to get to the hearts and minds of consumers has become even more daunting, not only because of constantly changing demands, but also the volume and velocity of data and the vast channels of data, from social media, online and mobile.

Today, the industry is increasingly adopting new technologies that marry science and big data to optimise the content that is presented when users search or browse the web. Leading companies such as Facebook, Twitter, Linked In and Ebay have already supported an open source technology called Hadoop, which Yahoo initiated and helped bring to the mainstream market. 

What Hadoop does is connect thousands of servers to process and analyse data at the same time, at supercomputing speeds. What this means is that Hadoop makes it possible for companies to rapidly personalise content and advertising, and deliver highly relevant experiences to consumers.

The power of Hadoop can be illustrated by the fact that, with the support of an extensive network of 40,000 Hadoop nodes, the daily throughput routinely reaches more than 10 petabytes of data (10 billion megabytes) processed. Hadoop also cleans up and compresses the equivalent of two million iPod music files every day.

This enormous power is at the epicenter of big data, which social media firms are leveraging.
Facebook, for example, uses Hadoop as a data warehouse, to archive status updates, to deliver event feeds describing what friends are doing, as well as for data processing. Likewise, Hadoop helps Twitter stay agile by analysing user information from tweets. For Linked In, Hadoop helps produce user-facing data, helping recommend 'people you may know' and viewers of profiles.

E-commerce companies like Ebay use Hadoop as a primary data processing system for its search-related functionality, producing search index, analytics, reporting and modeling -therefore enhancing search relevance and extending catalog coverage.

Today, Yahoo runs the biggest implementation of Hadoop, connecting more than 35,000 of our servers around the world and crunching 16 terabytes of data per day.  Hadoop started as a project by a group of Yahoo developers in 2005, and by 2009 the technology was adopted across the company's global operations and because it is open source, by more and more industry players. 

As the industry embraces Hadoop and other cloud computing infrastructure, we can expect more innovative consumer and advertiser products. Hollywood may still not have an answer to what makes a blockbuster, but for consumers, we are finally able to deliver the perfect user experience and just the information they need, always respecting their preferences, of course. 

For marketers and advertisers, this could be the closest we could ever be to predicting what consumers really want.