Using Hierarchy in Data
Hierarchical data is all around us. Think of any business and you are bound to find a large number of hierarchies: management structures, product categories, geographical organizations, and many others. Hierarchies are not just convenient ways to organize information, but are themselves a form of information.
Yet in many statistical methods, dealing with hierarchical data is difficult. Often, such data have high cardinality and the hierarchical structure itself presents a challenge, since many methods need a "flattened," or denormalized, analytical view. Here are a few common examples of such business data.
- Sales offices and retail locations: a company may have several hundred physical locations that sell and service customers, with great variation in their size, composition, and trade areas. Moreover, at each location, we are likely to have additional hierarchies, such as sales management/sales force or department/category/sub-category.
- Content categories and sub-categories: Web sites, printed directories, and similar properties may have hundreds or thousands of categories, each attractive to certain types of advertisers, although some advertisers also list in multiple categories. Content itself is often arranged hierarchically, so that users can start with general information and drill down to specific items;
- Markets and trade areas: A company may operate in hundreds of markets...think telephone directories, retailers, and home-based services, to name but a few. Depending on how granular the analysis needs to be, markets potentially number in the tens of thousands (e.g. census tracts, of which there are over 66,000).
Let's use as an example, a publisher that serves 800 markets with directories that contain 2,000 category headings. We want to have a model that prioritizes leads for a field sales force, who call on local businesses.
Plumbers are good advertisers nationally, but the markets they operate in are also a strong determinant: in some markets, plumbers are prospering and advertising; in others, they are not, perhaps due to local economic conditions. So we know there are interactions between the advertising category, geographical markets, and spending on advertising.
If we just look at the plumbing category, we may see that on a national level, the differences between plumbers in different markets tend to cancel each other out. If this is the case, then the plumbing category will not be useful in the model, and even if it is, we then have a variable that gives the average for plumbers nationally, even though their spending seems quite polarized. So we know that there is variation that would be helpful in modeling, but it is difficult to use (especially considering that we still have another 1,999 categories to consider!).
One strategy might be to build a clustering model that lets us group similar markets and categories together; these groupings would then be used in the model. Hypothetically, plumbers might be grouped with other building trades and car dealers, and trial lawyers with chiropractors. In the same way, we might group markets together, so that Detroit and Las Vegas are in one bucket, and Dallas and New York in another. But deciding what is similar is subjective, and the categories may not remain similar. For example, we currently have a mini-boom in auto sales, due to the government's cash-for-clunkers program, which has injected $3 billion into that market. Construction, however, continues to be depressed nationally, and we can expect plumbers in many areas to be tight with their spending.
There are other many other data transformation methods that might be employed to try to reduce the number of categories and markets into a compact form suitable for modeling. Again most will either require us to make some subjective decisions, or flatten some of the data and thereby lose information. There are also some analysts that might advocate a particular technology – neural networks, for example – in the belief that the data can just be dumped into software application, which will figure out how to make sense of the data.
Fortunately, there are well-established ways in which such hierarchical data can be incorporated. One elegant method we employ often is hierarchical Bayes modeling, which, as the name suggests, explicitly accommodates the use of hierarchical information. While the methodology is technically challenging (many marketing analysts may not be wholly comfortable even with simpler Bayesian methods), we have found the resulting models to be both robust and stable. In some cases, they have also been much more compact, as direct use of hierarchy reduces the number of variables in the model, compared with more traditional approaches.
We feel that this is a rich area for growth in marketing analytics. Hierarchical data is so ubiquitous, yet so under-used, that we in essence have significant new data sources for analysis. This has been an active area of research for Fulcrum: beyond hierarchical Bayes, our R&D team has found several other useful methods for using hierarchical data.
For those interested in learning more, feel free to contact us. You can also catch a presentation on this topic by our Chief Scientist, Hongjie Wang, at the National Center for Database Marketing conference, in December 2009.
Posted at 03:43AM Aug 28, 2009
by David King in Analytics |
