《SSE15-18-Neumann-Alla》.pdf

Real-time Recommendations using Spark Jan Neumann Sridhar Alla Comcast Labs, Washington, DC 1 Who we are 鈥 Jan Neumann, Lead of Big Data Research Team 鈥 Sridhar Alla, Director/Big Data Architect, Enterprise BI 鈥 Comcast Labs DC 鈥 Content Discovery for Comcast 鈥 Innovation 鈥 Machine Learning/Data Science Expertise for all of Comcast 鈥 Prototype new product ideas What do we do? Goal: Real-time TV Recommendations + = 鈥淭rending For You鈥 How can we do this? 鈥 Challenges 鈥 We have millions of users, thousands of programs 鈥 Programs on live TV are constantly changing (Cold- Start) 鈥 Approach inspired by 鈥淕oogle news personalization: scalable online collaborative filtering鈥? Das et al., 2007 Real-time Recommendations Algorithm 鈥 Cluster user by taste profiles and geographic proximity 鈥 Calculate Top K trending programs for each cluster 鈥 Look up cluster for user and return trending programs Cluster - User - User User- Trending Trending Request Cluster Programs Programs Real-time Recommendations in Spark 鈥 Thanks to Spark we could implement this quickly Batch: User History User Clustering from HDFS with MlLib Real-time Program recommendations per user Real-time: Live Tune TopK Trending Activity via Programs per