Sixty-four percent of organizations are already investing in or plan to invest in big data soon according to a recent Gartner report.* That equates to a huge number of individuals who now have to research how to embark on a big data deployment. The prevalence and benefits of big data analytics are undeniable, but there are some considerations to keep in mind before jumping in:
1) Identify a specific business need
Big data projects reap the most benefits when they address specific business needs. Having a use case in mind will help determine what data you need to analyze – social, machine or transactional data. Gartner recommends researching use cases and success stories in other industries; why not get inspired by what’s worked for others? Gartner analyst Doug Laney recently shared examples of big data at work in various industries: using big data analytics, the department store Macy’s was able to adjust prices in near real time for 73 million items based on demand and inventory; Wal-Mart was able to optimize search results and increase web checkouts by 10 – 15%; and American Express used sophisticated predictive models to analyze historical transactions and forecast potential churn. Once you’ve identified the analytic need not met by “small” data analysis, you have the first green light for considering big data technology.
Unearthing previously unimaginable insights from massive data sets is the premise of all the big data hype. Over the past few years as more and more stories come out about how companies are finding competitive advantages in their data, big data has moved beyond the buzz. Enterprises are deploying big data projects at a faster rate every year, and even more plan to do so within the next 2 years.
The extent to which a company can take advantage of big data analysis is determined by the amount of resources and infrastructure it has available. The good news is that now the barriers to entry have been lowered, making it possible for more organizations to meet their goals to transform operations with insights gained from big data. Here are three approaches that companies of any size can take based on their particular situation.
One thing to note is that these are underlying infrastructure approaches, and that you’ll still need an analytic engine like arcplan on top in order to interact with, visualize and distribute your insights.
Lots of resources and lots of infrastructure
Before big data was “big data,” Teradata was the only game in town. They’ve been at it for so long and their functionality is so robust – some of their capabilities are second to none. Now other vendors like SAP (with HANA) and Kognitio have their own massively parallel analytic databases. They offer robust processing and querying power on multiple machines simultaneously, enable near real-time MDX (Multidimensional Expressions, for OLAP querying) and SQL (Structured Query Language, the standard way to ask a database a question) queries, and in the case of SAP HANA and Kognitio, are fully in-memory. Not surprisingly, Teradata and SAP HANA come at a high price, but for that high price, the insights you achieve can be very near the speed of thought.
Acquiring thorough insight into your data and tapping into the needs and buying patterns of customers are growing needs for businesses striving to increase operational efficiency and gain competitive advantage. Throughout 2011, I noticed a heightened interest in ‘big data’ and ‘big data analytics’ and the implications they have for businesses. In August, Gartner placed big data and extreme information processing on the initial rising slope of their Hype Cycle for Emerging Technologies, so we’re just at the beginning of the big data trend. A recent TDWI survey reports that 34% of organizations are tapping into large data sets using advanced analytics tools with the goal of providing better business insight. The promise of big data analytics is that harnessing the wealth (and volume) of information within your business can significantly boost efficiency and increase your bottom line.
The term ‘big data’ is an all-inclusive term used to describe vast amounts of information. In contrast to traditional data which is typically stored in a relational database, big data varies in terms of volume, frequency, variety and value. Big data is characteristically generated in large volumes – on the order of terabytes or exabytes of data (one exabyte starts with 1 and has 18 zeros after it) per individual data set. Big data is also generated in high frequency, meaning that information is collected at frequent intervals. Additionally, big data is usually not nicely packaged in a spreadsheet or even a multidimensional database and often takes unstructured, qualitative information into account as well.
So where does all this data come from?