robots.txt写法完全攻略（机器人。txt写法完全攻略）.docVIP

robots.txt写法完全攻略（机器人。txt写法完全攻略） Robots.txt is the most. When the search engine to a site, it will first check the website of the root domain is a plain text file called robots.txt. Robots.t XT file for defining the search engine access on its website, which tell the search engine website in which files are allowing it to perform retrieval (download). This is what we often see on the network to access Robots standard (Robots Exclusion Standard). Below we referred to as RES. Robots.txt file format: Robots.txt file format is special, it is composed of records. These records by blank lines separate. Each record consists of two fields: 1) a User-Agent (user agent) serial character; 2) number of serial Disallow characters. Record format: Field valuegt :; We have the following the two domains of further explanation. User-agent (user agent): User-agent (user agent) is used to specify a search engine name robot, with Googlebot search program Google for example: User-agent:, Googlebot A robots.txt must have at least one User-agent record. If there is a User-agent record, then there is a number of RO bot will be subject to the limitations of the RES standard. Of course, if you want to specify all of the robot, just use a wildcard * fix, namely: User-agen * t: Disallow (denial of access): In the Robots.txt file, each record of the second domain is Disallow: command line. The Disallow statement is access to files and do not want the website (or directory). For example, the Disallow: email.htm access to the file of statement, ban Spiders download site email.htm file. While Disallow:/cgi-bin/ access to the cgi-bin directory of the Spiders statement, refused to enter the directory and its subdirectories. Disallow also has the function of wildcard declaration line. For example, in this example, the Disallow:/cgi-bin/ statement to search engine access to the cgi-bin directory and its subdirectories, and Disallow:/bob to search engine for /bob.html and /bob/indes.html access (i.e. w