Why It’s a Bad Idea to Build a Business Intelligence Platform From Scratch
A friend of mine is a Python developer for a billion-dollar corporation. His team is building a custom call center reporting app that connects to the company’s cloud data storage via APIs. I’ve seen some of the application and while it’s impressive for a custom system, it’s mostly tables of numbers with the occasional pie chart. This is after 8 months of work, and the only people accessing the system are a select few big data scientists.
Believe it or not, a number of companies are doing this kind of in-house development of analytical platforms. All the hype surrounding big data has them convinced that they’re missing out on the action. Consequently, companies large and small are devoting huge amounts of time, money and human resources to developing custom business intelligence systems for big data (Google BigQuery, Hadoop, etc.) reporting rather than simply choosing a platform that already exists and is proven to work in a similar environment.
At the heart of this trend is a desire for big data to have a greater impact in the organization. Since it’s usually small teams of data scientists who are dealing with big data, their impact and effectiveness is equivalent to a small drop in a much larger body of water – their ripple effect throughout the organization is often minimal and short-lived. To extend the reach of big data in the company and get important insights out to a greater number of decision makers, a BI platform is a necessary next step – one that leverages big data insights in easily-digestible executive reports and dashboards.
Some companies are going down the road of custom BI platform development, but their efforts are no match for solutions like arcplan that are already available. Below is a list of what you’d need to do to build a BI platform from scratch. You’ll quickly see why the effort and expense aren’t worthwhile.