分布式数据处理系统的研究与应用计算机系统结构专业论文.docx

II II Abstract With the universal application of network and accelerating rate of data generation, it has become an important topic on how to process data effectively. Traditional data processing system takes static finite dataset as its processing object and obtains the result by calling a static query script. Nowadays, with data becoming continuous, rapid and time-varying, the shortcomings of traditional data processing system gradually appears which makes it unable to meet the needs of modernization work. In the new background of application, higher functionality and performance of database system is required as well as the automation and intellectualization of data processing. The thesis first analyzes the research status of distributed data processing system at home and abroad. Then, the thesis designs a distributed data processing system scheme and discusses its two key technologies. Finally, the scheme is applied in a packet transmission system. Two key technologies of distributed data processing are studied in this thesis: query and load shedding. Unlike traditional static query, continuous queries always execute the query and output query results while data sources uninterruptedly access to the system. I study the characteristics of continuous queries and analyze the allocation of memory space, then design a greedy strategy based on dynamic sliding window query algorithm to improve the efficiency of continuous query processing. Load shedding is a method implemented for load fluctuations caused by irregular data flow rate. The arrival rate of the data stream is usually unpredictable. When the input rate exceeds system capacity; system overload occurs and causes deterioration of the performance of the system. This thesis designed a intelligent algorithm based on ant, which meet the conditions to find out the optimal path and makes full uses of the idle compute nodes. Key words: distributed system, data stream, continuous query, load shedding PAGE