Seven Must-Have Capabilities For Any Big Data Application
There are many definitions of Big Data. Most of them focus on the infrastructure and tools required to handle and manage extremely large sets of data. Gartner has a more multi-faceted view of Big Data that goes beyond infrastructure.
When we think about Big Data, we come at it from the perspective of an individual using an application built on the Big Data infrastructure. Big Data is what you can do with it.
What people look for from Big Data is insight. For us, Data + Analytics + Visualization = Insight. We also want to accelerate and amplify the process of insight and subsequent action. This accelerated and amplified process of insight and action, we call Big Insight.
Before you can do anything useful, there is a lot of work to be done to collect, organize and manage big data sets. A lot of what you can get out of the data can depend on how you organize the data. Once the data is organized, here’s what one should be able to do in order to get Big Insight:
- Explore. You should be able to explore the breadth and depth of the data in multiple, easy ways. You should be able to search for anything in the data set, like you do on Google. You should also be able to discover connections between items in the data set – like you discover connections with friends and colleagues on Facebook and LinkedIn. You should be able to explore the data set from the highest level down to the lowest, be able to see everything together, be able to make your way to a single item, and then go even deeper to view the attributes of that item – like you do at Amazon. Furthermore, you must be able to traverse the breadth of the data set if needed – again like on Amazon. All of this depends on how effectively and efficiently the data has been organized with appropriate indexing, classification, categorization and meta information.
- Analyze. While you gain insights from data just by exploring it in new and different ways, you can get so much more value by letting algorithms loose on the data. Building a wide range of algorithms to work on the data is always essential, but given there are so many ways to analyze the same data set, one application will never cover all of the possibilities. The trick is to make it easy for a broader community of individuals to develop algorithms by providing programmatic access to the data.
- Visualize. Visualization is key to extracting more value out of Big Data. In order to get Big Insight, you need great (big?) visualization. Visualization done right presents data in such a way that it stimulates insight – like at Information is Beautiful. It’s also important to provide multiple ways to visualize the data in order to accelerate the process of insight and action. There is no “one size fits all” to visualization, so it’s necessary to provide built-in visualizations and to enable additional visualizations through custom or third-party tools.
- Personalize. To make Big Data productive for people to use daily, it’s necessary to help them personalize their view into the system. This way they can specify the data set, algorithms or visualizations that work best for them or get to those they use most frequently. This is like having all the music in the world available to you and then being able to personalize it to suit your tastes and your moods and find what you might like quickly and easily. Personalization is an accelerator of value.
- Collaborate. Being able to share data and insights with one’s colleagues and peers adds to the breadth and depth of insight that can be gained from the system – like with Reddit. It also allows the system to learn from the network of collaborators and raises the baseline intelligence and value of the system. Collaboration, including simple sharing, ‘social’ networking and workflow, accelerates and amplifies the value one can get out of any Big Data application.
- Integrate. Every Big Dataset can become more valuable when it’s connected to other data sets. Integrating with new data sets and making connections across data sets is hard, but once the connections are made you can get new and different valuable insights. Here again, a programmatic approach is needed to enable lots of people to do the work of integrating and connecting additional data sets, applications and tools. IFTTT provides an interesting example of the value of connecting disparate personal services.
- Act. All of Netflix’s sophisticated algorithms that come up with suggestions for what you might like to watch are for naught if you don’t finally click to watch a movie. In a Big Insight application, all of the preceding effort is meaningless unless you can make decisions and act to improve your business. For this there must be some knowledge of what is important to the individual and the business. The nature and priorities of the business drive the way individuals act. Being able to capture the strategy and priorities of the business using rules or goals is an important part of optimizing the application to best meet the needs of the business and individual.
Interested in how we’ve addressed these seven capabilities? Sign up for a demo by sending an email to firstname.lastname@example.org.