Learning Product Characteristics and Consumer Preferences from Search Data

A building block of many models in empirical industrial organization is a characteristic space, where products are modeled as a bundle of characteristics over which consumers have preferences. The ability of such models to predict counterfactual outcomes depends on how well this characteristic space representation can capture substitution patterns. A limitation of existing methods is that product characteristics must be observable. In this paper, we extend a machine learning approach (Bayesian Personalized Ranking) that allows us to jointly learn latent product characteristics and consumer preferences from search data. We then show how this can be combined with existing demand estimation approaches to predict demand. Our application is to the hotel market, where we combine two datasets: consumers' web browsing histories, and hotel prices and occupancy rates. Using an event study design, we show that closeness in latent characteristic space predicts competition: hotels that are close to new entrants lose the most market share post-entry. We take a more structural approach to the 2016 merger of Marriott and Starwood, demonstrating that by using latent characteristics and consumer preferences learned from search data, we can substantially improve post-merger predictions of demand relative to standard baselines