In the world of commerce, “product feeds” are exchanged quite often. Retailers send them to price comparison sites such as Google Shopping. Affiliate networks aggregate feeds to give their customers a single place to shop. As omnichannel commerce becomes more prevalent, the application for these feeds is virtually endless. In short, feeds are a great way to exchange large amounts of structured product data.
Product feeds are used to enrich existing product records with additional data or to add new products to an assortment. In short, a product feed is a bulk dump of structured product information based on specific criteria.
This criteria can be as broad as pulling the entire current catalog of a specific retailer or as narrow as pulling a list of 1,000 products from a certain brand, within a specific category, with a valid GTIN (UPC), and containing at least 8 descriptive attributes. Below I’ll cover some of the most common kinds of feeds our customers and prospects at Indix use as well as the feed’s application to the business problem.
In the world of commerce, it’s extremely common for a “publisher” to work within affiliate networks to earn revenue. The basic model is that the publisher acts as an intermediary between the end consumer and products that consumer might buy. If the consumer decides to make a purchase, they are directed to an online shopping cart on the retailer’s website. The publisher then gets a percentage of the sale.
The publisher will typically have a theme such as the outdoors and might recommend the best products for a weekend camping trip. These products can live across several retailers, brands and categories. The way these publishers obtain product information today is primarily through flat files (csv, excel, etc.) from hundreds of retailers or established affiliate networks. These flat files vary in format, are often outdated, and missing key bits of information. Now imagine if the publisher could get feeds across any number of retailers, all in the same format and with complete information. Sign me up!
Imagine you’re an entrepreneur and you’re living in a geography that doesn’t have the Amazon.com access we take for granted in the U.S. Imagine the opportunity in front of you to set up a new e-commerce shop. Similarly, imagine if you’re an established retailer: You’re very strong in a few product categories, but you see an opportunity to grow your catalog. What are your options?
Well, the only real option is to have your in-house buyers manually research and source product lists from manufacturers, distributors, and suppliers. If you’re setting up a sizable shop or adding several categories to your assortment, you’d be dealing with thousands of data sources. And, like the retailers, each source gives you varying formats of incomplete data. If only there was a cloud catalog of structured product data covering tens of thousands of brands…. Sign me up!
Hopefully by now you have a decent grasp on the concept of a product feed. Because the business applications of product feeds are virtually endless, the ability to customize feeds can add a lot of value. Take for example the growing world of online marketplaces and their sellers. In the early stages of these marketplaces, the goal was to get as many products online as possible. Quantity over quality. There weren’t many constraints, if any, on what attributes were required to list a product.
Furthermore, required attributes weren’t checked for accuracy. For example, UPCs didn’t have to be all numbers (note: UPCs are always all numbers). There are two common non-Indix ways to solve this. The first way is to manually fix records through research and data entry or to pay someone else way too much money per record to manually do it for you. This cost and manual effort might be worth it for the fastest moving, most popular products. But what about the millions of longer tail products, many with zero product attributes that aren’t worth $5+ per record to manually fix? If only there was an organization that uses machine learning and AI to provide millions of structured product records at a fraction of the costs…. Sign me up!
Both product feeds and the process of creating them come in many forms, but here is how we do it at Indix:
Indix hosts the world’s largest Cloud Catalog of structured product information. Click here to read about how we gather, structure, and store a catalog of this magnitude.
Because we have hundreds of millions of structured product records stored in the cloud, customers begin with introductory conversations with the Indix Sales and Technical teams. These typically start at the high level to understand the general business problem. We can also typically discern which of the three feed types mentioned above are the best fit.
Next comes the Proof of Concept (POC) where we’ll begin to define the technical specs and criteria around what a successful feed needs to look like. A sample feed is usually a part of the POC. Indix has the ability to apply a combination of SQL and User Defined Functions to the data before it’s egressed as a feed. This allows for extreme flexibility and control with regards to data quality and format of the feed. For example, we can exclude products that don’t have a 14 digit GTIN, or we can derive new fields such as the median or mode of price, or we can partition feeds by brand, category, or store.
After a successful POC, it’s time to go into production. A format (.csv, .tsv, .json), frequency (typically, weekly, bi-weekly, monthly, or quarterly) and transfer protocol (sftp, ftp, or Amazon s3) are agreed upon and you get your first feeds.
In conclusion, feeds remain a primary avenue to exchange product information. As omni-channel commerce expands its influence, product feeds will continue to be a vital piece of this as their value goes beyond the traditional applications.
Also published on Medium.