处理海量数据80801（Deal with mass data 80801）.docVIP

处理海量数据80801（Deal with mass data 80801） First, the amount of data is too large, the data may exist in any case. If there are 10 data, so can each to check for handling, if there are hundreds of data, can also be considered, if the data to the level of tens of millions, even billions of dollars, that is not the hand can solve, must be processed by tools or programs, especially the mass data, what are there may be, for example, data somewhere in the format of a problem, especially in the treatment program, but also in front of the normal processing, suddenly to a place there, to terminate the program. Two, software and hardware need high, system resource occupation rate is high. Dealing with massive data, in addition to good methods, the most important thing is to use tools reasonably and allocate the system resources reasonably. In general, if the data over TB, minicomputer is to be considered, the ordinary machine if there is a good way to consider, but also must increase CPU and memory, like to have the courage not light thousands upon thousands of horses and soldiers, a soldier is difficult to win. Three, the need for high processing methods and skills. This is also the purpose of writing this paper. The good method is the accumulation of long-term work experience of an engineer, and also a summary of personal experience. There are no general methods of handling, but there are general principles and rules. Well, what are the experience and skills of dealing with mass data, and Ill list what I know for your reference: First, select excellent database tools Now the database tools more manufacturers, the processing of massive data need to be relatively high for the use of database tools, generally use Oracle or DB2, the Microsoft Corp recently released SQL Server 2005 performance is also good. In addition, in the field of BI database, data warehouse, multidimensional database, data mining and other related tools to choose, like good ETL tools and OLAP tools are necess