基于spark的矩阵分解推荐算法recommendation-计算机应用.pdfVIP

Journal of Computer Applications ISSN 1001-9081 2015- 【-10 计算机应用,2015, 【( 【): 【- 【 CODEN JYIIDU 基于 Spark 的矩阵分解推荐算法 郑凤飞*，黄文培，贾明正 (西南交通大学 信息科学与技术学院，成都 611756) (*通信作者电子邮箱 1207264887@) 摘 要: 针对传统矩阵分解算法在处理海量数据信息时所面临的处理速度和计算资源的瓶颈问题，利用Spark 在内存计算 和迭代计算上的优势，提出了Spark 框架下的矩阵分解并行化算法。首先，依据历史数据矩阵初始化用户因子矩阵和项目因子 矩阵；其次，迭代更新因子矩阵，将迭代结果置于内存中作为下次迭代的输入；最后，迭代结束时得到矩阵推荐模型。通过 在GroupLens 网站上提供的MovieLens 数据集上的实验表明，Speedup 值达到了线性的结果，该算法可以提高协同过滤推荐算 法在大数据规模下的执行效率。 关键词: 协同过滤；推荐算法；矩阵分解；迭代最小二乘法；Spark 中图分类号:TP301.6 （算法理论） 文献标志码: A Recommendation Algorithm with Matrix Factorization Method Based on Spark ZHENG Fengfei*, HUANG Wenpei, JIA Mingzheng (School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China) Abstract: In order to solve the bottleneck problems of processing speed and resource allocation, a spark based matrix factori- zation recommendation algorithm was proposed. Firstly, user factor matrix and item factor matrix were initialized according to history data; Secondly, factor matrix were iteratively updated and the result were stored in memory as the input of next iteration; Finally, recommendation model were generated when iteration ended. The experiment on MovieLens shows that the speedup was liner and spark based algorithm can save time and significantly improve the execution efficiency of collaborative filtering recommendation algorithm. Keywords: collaborative filtering; recommendation algorithm; matrix factorization; alternating least square; Spark 和用户近邻模型的算法，解决了数据稀疏的问题，但存在模 0 引言 型过拟合的问题。文献[6]提出了一种支持不完整评分矩阵的