Seminar by Assistant Prof Ruohan Zhan, Hong Kong University of Science and Technology, 27 Aug 2024, ABN Seminar Room 1-1

Title: Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach

Time: August 27, 2024 (Tuesday), 4pm – 5pm

Venue: Seminar Room 1-1, Academic Building North

Abstract:  Recommender systems are essential for content-sharing platforms by curating personalized content. To evaluate updates to recommender systems targeting content creators, platforms frequently rely on creator-side randomized experiments. The treatment effect measures the change in outcomes when a new algorithm is implemented compared to the status quo. We show that the standard difference-in-means estimator can lead to biased estimates due to recommender interference that arises when treated and control creators compete for exposure. We propose a "recommender choice model” that describes which item gets exposed from a pool containing both treated and control items. By combining a structural choice model with neural networks, this framework directly models the interference pathway while accounting for rich viewer-content heterogeneity. We construct a debiased estimator of the treatment effect and prove it is $\sqrt{n}$-consistent and asymptotically normal with potentially correlated samples. We validate our estimator's empirical performance with a field experiment on Weixin short-video platform. In addition to the standard creator-side experiment, we conduct a costly double-sided randomization design to obtain a benchmark estimate free from interference bias. We show that the proposed estimator yields results comparable to the benchmark, whereas the standard difference-in-means estimator can exhibit significant bias and even produce reversed signs.

Bio:  Ruohan Zhan is an assistant professor in the Department of Industrial Engineering and Decision Analytics at the Hong Kong University of Science and Technology. Her primary research interest lies in the understanding and optimization of online marketplaces. Ruohan studies the causal evaluation of marketplace interventions, economic analysis of the dynamics and interactions among multiple stakeholders, and optimization of platform operations, including recommendation algorithms and digital experimentation. Methodologically, She is interested in causal inference, econometrics, statistical learning and machine learning. Her research has been published in Management Science and Proceedings of National Academy of Sciences, as well as machine learning conferences including NeurIPS, ICML, ICLR, WWW, and KDD. Ruohan earned her PhD from Stanford University and her BS from Peking University.