Instructionsforthepreparationofacamera-readypaperin.docVIP

A novel approach to data deduplication over the engineering-oriented cloud systems Zhe SUN a, b, Jun SHENa, *, Jianming YONGc a School of Information Systems and Technology, University of Wollongong, Wollongong, NSW, Australia b Information Management Center, Huaneng Shandong Shidao Bay Nuclear Power Co., Ltd, Shandong, China c School of Information Systems, University of Southern Queensland, Toowoomba, QLD, Australia Abstract. This paper presents a duplication-less storage system over the engineering-oriented cloud computing platforms. Our deduplication storage system, which manages data and duplication over the cloud system, consists of two major components, a front-end deduplication application and a mass storage system as back-end. Hadoop distributed file system (HDFS) is a common distribution file system on the cloud, which is used with Hadoop database (HBase). We use HDFS to build up a mass storage system and employ HBase to build up a fast indexing system. With a deduplication application, a scalable and parallel deduplicated cloud storage system can be effectively built up. We further use VMware to generate a simulated cloud environment. The simulation results demonstrate that our deduplication storage system is sufficiently accurate and efficient for distributed and cooperative data intensive engineering applications. Keywords: Cloud storage, data deduplication, Hadoop distributed file system, Hadoop database Introduction Modern society is a digital universe. Almost no information or industry applications can survive without this digital universe. The size of this digital universe in 2007 is 281 exabytes and in 2011 [10], it becomes 10 times larger than it was in 2007. The most critical issue is that nearly half the digital universe cannot be stored properly in time. This is caused by several reasons: firstly, it is hard to find such a big data container; secondly, even if a big container can be found, it is still impossible to manage such a vast dataset;