Understanding the nature and potential role of ‘big data’

Big data opportunities emerged when several advances in different IT categories aligned in a short period at the end of the last decade, creating a dramatic increase in computing technology capacity. This new capacity, coupled with latent demands for analysis of “dark data,” social networks data and operational technology (or machine data), created an environment highly conducive to rapid innovation. Gartner Says Big Data Will Drive $28 Billion of IT Spending in 2012

…. through 2018, big data requirements will gradually evolve from differentiation to ‘table stakes’ in information management practices and technology. By 2020, big data features and functionality will be non-differentiating and routinely expected…”
Gartner Says Big Data Will Drive $28 Billion of IT Spending in 2012

In 2012 Gartner assessed that Big Data was approaching the “Peak of Inflated Expectations” on its Hype Cycle. Gartner on Big Data

The possibilities arising from (the) evolving ecosystem makes it clear that big data is not like your father’s business intelligence (BI) tools. For Big Data Analytics There’s No Such Thing as Too Big

Ralph Kimball, founder of the data warehouse consulting company Kimball Group, says use cases ….. are motivating a search for architectural similarities and differences across all of them, and more. What they seek in this fluid environment is a system architecture that addresses big data analytics generally. For Big Data Analytics There’s No Such Thing as Too Big

…. the story of big data is still being written; new methods and tools continue to be developed to solve new problems. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Big data is driving innovative approaches like these that change as frequently as the problems do, says Adrian. He advises remaining open to new ideas, and frequent experimentation.
For Big Data Analytics There’s No Such Thing as Too Big

At the moment there are not many big data best-practices. In fact, as Michael Chui, one of the authors of the well-known report Big Data: The Next Frontier for Innovation, Competition, and Productivity by the McKinsey Global Institute told at the MIT Sloan CIO Symposium in 2012, there are “‘no [Big Data] best practices. I’d say there are emerging next practices”. bigdata-startups.com/best-practices

Mark Graham has leveled broad critiques at Chris Anderson’s assertion that big data will spell the end of theory: focusing in particular on the notion that big data will always need to be contextualized in their social, economic and political contexts. wikipedia/Big_data

[capturing value from bid data] requires highly specific customization to an organization’s context, strategy, and capabilities. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

With data becoming a key competitive asset, leaders must understand the assets that they hold or to which they could have access.Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

In “Steamrolled by Big Data.”, The New Yorker’s Gary Marcus declares that “Big Data isn’t nearly the boundless miracle that many people seem to think it is.” He concedes that “Big Data can be especially helpful in systems that are consistent over time, with straightforward and well-characterized properties, little unpredictable variation, and relatively little underlying complexity.” But Marcus warns that “not every problem fits those criteria; unpredictability, complexity, and abrupt shifts over time can lead even the largest data astray. Big Data is a powerful tool for inferring correlations, not a magic wand for inferring causality.” Calling for “a sensitivity to when humans should and should not remain in the loop,” Marcus quotes Alexei Efros, “one of the leaders in applying Big Data to machine vision,” who described big data as “a fickle, coy mistress.” The Big Data Debate: Correlation vs. Causation

Big data can highlight correlations with the data, but this does not imply causation, not does it mean that correlations are necessarily strategically relevant or significant. Matti Keltanen at The Guardian agrees, explaining “Why ‘lean data’ beats big data.” Writes Keltanen: “…the lightest, simplest way to achieve your data analysis goals is the best one…The dirty secret of big data is that no algorithm can tell you what’s significant, or what it means. Data then becomes another problem for you to solve. A lean data approach suggests starting with questions relevant to your business and finding ways to answer them through data, rather than sifting through countless data sets. Furthermore, purely algorithmic extraction of rules from data is prone to creating spurious connections, such as false correlations… today’s big data hype seems more concerned with indiscriminate hoarding than helping businesses make the right decisions.” The Big Data Debate: Correlation vs. Causation

Organizations will need to have the right people and processes to capture value from the use of big data. On people, MGI research indicates that the key sets of talent that will be in increasingly short supply are deep analytical talent to execute big data analyses; managers and analysts who know how to request and consume big data analyses; and supporting technology personnel to implement big data. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

A significant constraint on realizing value from big data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from big data. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

When setting up a big data analysis system, your biggest hurdle will be finding the right talent who knows how to work the tools to analyze the data, according to Forrester Research analyst James Kobielus.

Big data relies on solid data modeling. Organizations will have to focus on data science, Kobielus said. They have to hire statistical modelers, text mining professionals, people who specialize in sentiment analysis. This may not be the same skill set that todays analysts versed in business intelligence tools may readily know.

Such people may be in short supply. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions, McKinsey and Company estimated. 5 Things CIOs Should Know About Big Data

Moreover, enterprises signing on to open source solutions should keep in mind the need to have enough specialized staff on hand to engage with them, potentially adding to their costs. For Big Data Analytics There’s No Such Thing as Too Big

As a new tool, nearly every NoSQL developer is in learning mode. This learning curve can make it difficult to find experienced NoSQL programmers or administrators. For Big Data Analytics There’s No Such Thing as Too Big

In 2012, Gartner updated its definition as follows: “Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.” wikipedia/Big_data

A wide variety of techniques and technologies has been developed and adapted to aggregate, manipulate, analyze, and visualize big data. These techniques and technologies draw from several fields including statistics, computer science, applied mathematics, and economics. This means that an organization that intends to derive value from big data has to adopt a flexible, multidisciplinary approach. [A brief overview of many of the techniques and technologies can be found in the McKinsey report.] Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

An article by SearchCIO provides short summaries of the following 10 big data examples:

1. Macy’s Inc. and real-time pricing.
2. Tipp24 AG, a platform for placing bets on European lotteries, and prediction.
3. Wal-Mart Stores Inc. and search.
4. Fast food and video.
5. Morton’s The Steakhouse and brand recognition.
6. PredPol Inc. and repurposing.
7. Tesco PLC and performance efficiency.
8. American Express Co. and business intelligence.
9. Express Scripts Holding Co. and product generation.
10. Infinity Property & Casualty Corp. and dark data.

Ten big data case studies in a nutshell – SearchCIO.com

bigdata-startups.com provides over 40 examples of the application of big data including:

1. The Los Angeles Police Department Predicts and Fights Crime With Big Data
2. AirBnB Matches Apartments, Castles And Igloos With Guests Using Big Data
3. How P&G Uses Big Data To Turn Diapers Into Insights
4. MasterCard Applies Big Data To Help Retailers Achieve Better Results
5. Kayak Uses Big Data To Predict The Best Day To Book Your Travel Journey
6. Union Pacific Railroad Turned To The Industrial Internet To Stay On Track
7. How Nestlé Understands Brand Sentiment Of 2.000 Brands In Real-time
8. How Heineken Interacts With Customers Using Big Data

Addressing privacy and security issues will become paramount as more data increasingly travel across boundaries for various purposes. Privacy, in particular, not only requires attention to compliance with laws and regulations, but also is fundamental to an organization’s trust relationships with its customers, business partners, employees, and other stakeholders. Certainly, organizations will need policies that comply with privacy laws and any government privacy regulations. But in developing a privacy policy, organizations will need to thoughtfully consider what kind of legal agreements, and, more importantly, trust expectations, it wants to establish with its stakeholders. And it will need to communicate its policies clearly to its stakeholders, especially customers, as they become increasingly savvy and concerned about what is known about them and how that information can potentially be used. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Marketers are increasingly looking to Big Data to provide the insight they need to create richer, more personalized and more targeted messaging for consumers. But collecting that data without making consumers leery of your brand requires data management that respects consumer privacy while creating value for your customers.

Many companies are pursuing Big Data with the ultimate aim of better understanding and selling to their customers. After all, it is well-understood at this point that even basic personalization, like using customers’ individual names, can substantially increase open and response rates for physical and digital mail. And Big Data promises a far greater degree of personalization and targeting. But collecting that data can be like handling a live wire: You can wind up having a real bad day if you don’t treat it with respect. Big Data for Marketing: Respect Consumer Privacy or Get Burned

A recent Bain & Company study [found that] early adopters of Big Data analytics have gained a significant lead over the rest of the corporate world. Examining more than 400 large companies, we found that those with the most advanced analytics capabilities are outperforming competitors by wide margins. The leaders are:

• Twice as likely to be in the top quartile of financial performance within their industries
• Five times as likely to make decisions much faster than market peers
• Three times as likely to execute decisions as intended
• Twice as likely to use data very frequently when making decisions

Big Data: The organizational challenge

… big data techniques allow you to sift through data to look for patterns at a much lower cost and in much less time than traditional BI systems,” says the PWC report. For Big Data Analytics There’s No Such Thing as Too Big

We have identified five broadly applicable ways to leverage big data that offer transformational potential to create value and have implications for how organizations will have to be designed, organized and managed.

• Creating transparency
• Enabling experimentation to discover needs, expose variability and improve performance
• Segmenting populations to customize actions
• Replacing/supporting human decision making with automated algorithms
• Innovating new business models, products and services

Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

In 2012 Gartner assessed that Big Data was approaching the “Peak of Inflated Expectations” on its Hype Cycle. Gartner on Big Data

The six sectors that achieved a leap in productivity [from Internet and Web 1.0 investments] shared three broad characteristics in their approach to IT. First, they tailored their IT investment to sector-specific business processes and linked it to key performance levers. Second, they deployed IT sequentially, building capabilities over time. Third, IT investment evolved simultaneously with managerial and technical innovation. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Big data has reached a critical stage. The market is poised to grow to more than $50 billion by 2017, but more than 55 percent of big data projects fail.

With so much opportunity coupled with hype and misinformation, we are in the midst of the big data Wild West. There is a standoff coming between those that understand what big data is — the ones making investments to collect, store and harvest it — and those that are buying snake oil and don’t understand how big data can impact their business. [interestingly, the 55% failure rate is approximately the same as that for any broad based organizational change] The big data Wild West: The good, the bad and the ugly

In order to validate these big data opportunities, leading organizations have often discovered that adopting a process of purposeful experimentation (a meta-application of the big data lever of experimentation) can be the most powerful path toward becoming an organization that fully leverages big data, rather that specifying a complete plan for the enterprise prior to doing any implementation. … we find that most organizations follow a journey that builds capabilities over time.

Our research identified four levels of maturity or sophistication to categorize actions that can be taken.

1. Most basic is digitizing and structuring the data, …
2. The second level of sophistication requires making the data available, e.g., through networks, …..
3. The third level of sophistication is applying basic analytics, which essentially covers a range of methodologies, such as basic data comparisons, and relatively standardized quantitative analyses, e.g., those that do not require customized analyses to be designed by people with deep analytical skills.
4. The fourth and highest level is applying advanced analytics, such as the automated algorithms and real-time data analysis that often can create radical new business insight and models. They allow new levels of experimentation to develop optimal approaches to targeting customers and operations, and opening new big data opportunities with third parties.

Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Speed is the key to success in the early stages of big data implementation. The faster you can complete projects and build organizational expertise in using data in this new way, the sooner you can create value and move to a more sophisticated stage of adoption. The big data Wild West: The good, the bad and the ugly

“Furthermore, big data analysis is usually iterative: you ask one question or examine one data set, then think of more questions or decide to look at more data. That’s different from the ‘single source of truth’ approach to standard BI and data warehousing,” the PWC report says. For Big Data Analytics There’s No Such Thing as Too Big

Jeff Hammerbacher, cofounder of Cloudera and a Hadoop expert, says organizations need first to see data as a competitive advantage before building a big data function. The next step is to build out a low-cost, reliable infrastructure for data collection and storage for whichever line of business they perceive to be most critical. If organizations don’t have that digital asset, then they’re not even in the game. Once there, they can start layering on the complex analytics. Most companies go wrong when they start with the complex analytics. For Big Data Analytics There’s No Such Thing as Too Big

The lack of a customer-centric view severely limits the organization’s ability to use any of the powerful big data levers to create new value. An effective enterprise data strategy must include interoperable data models, transactional data architecture, integration architecture, analytical architecture, security and compliance, and frontline services. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Best practice big data companies have built sufficient scale in a core group of deep analytical talent, upon which the rest of their organization can draw.
….. But having a core set of deep analytical talent is not enough to transform an organization, especially if the key business leaders and analysts do not know how to take advantage of this big data capability. All of the business leaders in an organization will have to develop a baseline understanding of analytical techniques in order to become effective users of these types of analyses. Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.