through.NETTechnology.PDF

I.J. Modern Education and Computer Science, 2014, 8, 59-65 Published Online August 2014 in MECS (/) DOI: 10.5815/ijmecs.2014.08.07 Implementation of Parallel Web Crawler through .NET Technology Md. Abu Kausar Dept. of Computer System Sciences, Jaipur National University, Jaipur, India Email: kausar4u@ V. S. Dhaka Dept. of Computer System Sciences, Jaipur National University, Jaipur, India Email: vijaypal.dhaka@ Sanjeev Kumar Singh Dept. of Mathematics, Galgotias University, Gr. Noida, India Email: sksingh8@ Abstract—The WWW is increasing at very fast rate and crawler, which is used for gathering of web pages data or information present over web is changes very indexed by the various search engines. Moreover web frequently. As the web is very dynamic, it becomes very crawlers are used in many other applications that process difficult to get related and fresh information. In this paper large numbers of web pages. we design and develop a program for web crawler which The rest of this paper is organized as follow. Sections uses multiple HTTP for crawling the web. Here we use two describe related work. Sections three describe the multiple threads for implementation of multiple HTTP web crawling terminology Sections four describe the connection. The whole downloading process can be