Eight Things Every Market Intelligence System Must Do
Market Intelligence systems are developing quickly as more and more content comes online, as big data, analytics and visualization technologies develop and as businesses realize that there is a great deal of value that remains to be extracted from both public and private data.
We have been building a market intelligence (MI) system that discovers, organizes, analyzes and visualizes the web from the perspective of product managers. Over the past year, we have distilled eight requirements for modern MI systems.
- Provide data as close to real-time as possible. The days of quarterly or even monthly updates of market intelligence are over. Today, the Consumer Price Index, a key measure of inflation, is updated monthly. Setting politics aside, this system should be and will eventually be real-time or close to real-time, and in turn that information will be useful in so many new ways. In our case we have seen prices of products changing at an accelerated pace and as a function of time of day, day of the week, ZIP code and more. Given the huge volume of data online — at least 4.7 billion indexed pages on the web as of early June 2013 — getting a complete picture or even a partial view in real-time remains a challenge. But the closer to real-time an MI system becomes, the closer it will reflect what is going on in the market, and the more useful it will prove.
- Provide real-time and dynamic analytics. It’s not enough to have real-time data alone. Given the vast amount of data that is available, the MI system must allow for different views and perspectives of the data, at multiple levels and using lots of different algorithms and analyses, all in real time. If one variable or set of variables changes, the system should quickly reflect and allow for reaction to that change.
- Be extensible. New sets of data are becoming available all the time. For example, the Pinterest stream is emerging as a significant source of intelligence for commerce. Or a business may want to add data from its internal systems. Being able to add new data streams quickly and easily and then being able to do the analytics with the new data is an important feature of MI systems.
- Provide simple, visual representations of complex data. No longer should MI be relegated to the few people in a company patient or skilled enough to pound on spreadsheets or databases to extract basic insights. The system should provide easily customizable and multiple visual representations of insights that meet the interests and needs of a broad set of people.
- Provide actionable insights. The potential for useful insights increases with the amount of data at the system’s disposal and the sophistication of the algorithms available to the system. There is also the potential that the system might produce hundreds or even thousands of insights, but that as humans we can be overwhelmed by such volume. The ideal MI system will prioritize insights and possible actions to reflect what customers have told it is most important to their business.
- Get the system to generate as many insights as possible. The ideal MI system will cope with ‘known knowns’ and ‘known unknowns’ — the problems that businesses can describe and characterize. If you can characterize the problems you want to solve, the system should be able to automatically process it and provide insights into those problems. A large majority of the insights that will help a company be more efficient and competitive fall into this category.
- Allow for people to augment the system’s intelligence. The trickier part for MI systems is how to deal with the ‘unknown unknowns’ — the things you don’t know you don’t know. These are patterns, facts, angles that were not apparent and perhaps could not even be detected by the system but are nonetheless real and important. It’s also simple things like teaching the system about false positives and negatives and improving the machine learning system by teaching it to recognize new patterns. A modern MI system should amplify human intelligence and use human input to improve its machine learning.
- Combine internal company data with public data from the Internet, to yield the richest insights. This requirement brings in a whole host of additional issues, such as securing a company’s private data, expanding the data and analysis models to accommodate private data and integrating with the business’s existing systems. To increase the usefulness and usage of the MI system, it would be nice for users of the system to project the data in a way that’s familiar to them, such as product category names that they use internally. And not have to change their processes and workflows too much.
We believe that a market intelligence system that meets all of these requirements will provide any business with the power to compete boldly and skillfully.