Unearthing previously unimaginable insights from massive data sets is the premise of all the big data hype. Over the past few years as more and more stories come out about how companies are finding competitive advantages in their data, big data has moved beyond the buzz. Enterprises are deploying big data projects at a faster rate every year, and even more plan to do so within the next 2 years.
The extent to which a company can take advantage of big data analysis is determined by the amount of resources and infrastructure it has available. The good news is that now the barriers to entry have been lowered, making it possible for more organizations to meet their goals to transform operations with insights gained from big data. Here are three approaches that companies of any size can take based on their particular situation.
One thing to note is that these are underlying infrastructure approaches, and that you’ll still need an analytic engine like arcplan on top in order to interact with, visualize and distribute your insights.
Lots of resources and lots of infrastructure
Before big data was “big data,” Teradata was the only game in town. They’ve been at it for so long and their functionality is so robust – some of their capabilities are second to none. Now other vendors like SAP (with HANA) and Kognitio have their own massively parallel analytic databases. They offer robust processing and querying power on multiple machines simultaneously, enable near real-time MDX (Multidimensional Expressions, for OLAP querying) and SQL (Structured Query Language, the standard way to ask a database a question) queries, and in the case of SAP HANA and Kognitio, are fully in-memory. Not surprisingly, Teradata and SAP HANA come at a high price, but for that high price, the insights you achieve can be very near the speed of thought.
Musings on the challenges of big data in a year of serious hype
There’s a reason you haven’t heard more than a handful of big data success stories in 2012. Handling big data correctly is hard, requires huge infrastructure and resource investments, and may not be worth it…yet. According to one survey in November 2012, 60% of businesses said it’s too early to tell if their big data project was successful and produced a proper ROI. It seems that so much of the hype around big data is focused on the technologies you need to buy and the talent you need to acquire (data scientist is the latest fad title), and not on what’s most important: what you can do with the data, what value you can extract, what business decisions you can speed up or improve with all that data.
With companies jumping on the big data bandwagon to the tune of $28 billion this year, it’s time to discuss why it might be best to ignore the hype for now and focus on reaping insight from the data you have already. Here’s why I’m not impressed with your big data:
You don’t actually have big data.
The marketing hype can lead you to believe you have a “big data problem” when you really don’t. Using the terminology incorrectly has the potential to harm your business, causing you to invest in unnecessary infrastructure when you may be able to leverage what you already have in place. Even Microsoft and Yahoo! have made this mistake…
According to industry forecasts, the world’s volume of data doubles every 18 months, and all forms of enterprise data will grow 650% over the next five years. The talk around big data is more than marveling at the mass of information we’re creating. As analysts and data scientists, we’re trying to find the good stuff – the trends, the data that allows us to make better decisions now and in the future, to predict the moves that will make our business more successful down the line.
Big data (explained in our previous article here) might be new to you, but I’ve seen some analyst reports referencing big data ideas as far back as 2001. However, the BI world is talking about it more and more as data volumes grow and we begin to see the potential knowledge to be gained in these data sets.
So maybe you’re thinking, how can big data benefit my company? It’s hard to think conceptually about it, so let’s take a look at some concrete examples of how companies are using big data today. We’ll start with the retail industry. Keep in mind that many of these ideas can be used on a smaller scale for retailers of any size.
Wal-Mart sifts through massive amounts of unstructured social media and search data to find out what products consumers are talking about. They use that information to set their ad buying strategy on sites like Google, with the goal of competing for e-commerce sales – currently dominated by Amazon.com.