基于Hadoop中医药数据管理策略研究.docVIP

基于Hadoop中医药数据管理策略研究 　　摘要：目的 为解决传统方法采集、存储和处理海量中医药数据的低效问题，探索数据管理的新策略。方法 根据中医药数据的典型特征，设计基于Hadoop的分层管理架构，对串行数据挖掘算法进行MapReduce化改进；部署单节点服务器和分布式集群，采用8组不同规模的数据集，进行数据采集实验和串并行算法实验。结果 数据传输时间在非分布式环境下通常超过3000 s，增幅较大，而在分布式集群下一般不超过300 s，增幅平缓；当数据规模超过一定范围后，与伪分布式和完全分布式下的并行算法比较，非分布式下串行算法的运行耗时急剧增加。结论 与传统单节点系统相比，基于Hadoop的中医药数据管理平台采集、存储及处理海量数据的效率明显提高，尤其适用于大规模非结构化或半结构化的中医药数据。 　　关键词：中医药数据；Hadoop；分层管理；MapReduce；分布式 　　DOI：10.3969/j.issn.1005-5304.2018.05.021 　　中?D分类号：R2-03 文献标识码：A 文章编号：1005-5304（2018）05-0096-05 　　Research on TCM Data Management Strategy Based on Hadoop 　　LIANG Yang1， 2， DING Chang-song1， YU Jun-yang3 　　1. School of Information Science and Engineering， Hunan University of Chinese Medicine， Changsha 410208， China； 2. School of Information Science and Engineering， Central South University， Changsha 410083， China； 　　3. Software School， Henan University， Kaifeng 475001， China 　　Abstract： Objective To solve the inefficiencies of traditional methods of collecting， storing and processing mass TCM data； To explore new strategies for data management. Methods According to the typical characteristics of TCM data， a hierarchical management architecture based on Hadoop was designed and a processing algorithm based on MapReduce was improved. The single node server and Hadoop distributed clusters were deployed. Data acquisition experiment and serial and parallel algorithm experiments were conducted， using eight groups of data sets of different sizes. Results The data transfer time was usually more than 3000 seconds with larger increase under non-distributed environment， while it generally did not exceed 300 seconds with moderate growth rate in distributed clusters. In addition， when the data size exceeded a certain range， the running time of the serial algorithm under non-distributed environment was drastically increased， comparing with the parallel algorithm under pseudo-distributed and fully distributed environment. Conclusion Compared with the traditional single node system， th