robots.txt写法（机器人。txt写法）.docVIP

robots.txt写法（机器人。txt写法） Robots.txt writing Robots.txt plays a very important role in the optimization of SEO site, robots.txt is the first file search engine when you visit a website to view, search engine spiders came to a site, first of all to robots.txt, check the site root directory if there is scope will be determined in accordance with the access to the contents of the file, if the file is not there will be free to grab, may have to repeat the path or error page. Below I will talk about rotbots.txt writing, I hope to help you. For example, robots.txt written: 1, stop all search engines any department access to the site, also is to prevent all Pastebin your website. * User-agent: Disallow: / Any department 2, allowing all search engines to access your site, this is the usual usage. * User-agent: Disallow: Probably * User-agent: Allow: / 3, if a directory of your web site do not want to be included in the search engine, written as follows: * User-agent: Disallow: / 1/ directory name Disallow: / 2/ directory name Disallow: / 3/ directory name Can not be written carefully: Disallow: / 1/ / directory name directory name 2/ this situation, each directory to a separate line of special note. 4, only allow a search engine to access your site. Baidu is also Baiduspider Google is Googlebot User-agent: Baiduspider Allow: User-agent: Googlebot Allow: 5, only a refrain from visiting your site search engine. Google Googlebot Baidu Baiduspider User-agent: Baiduspider Disallow: / User-agent: Googlebot Disallow: / 6, stop the search engine dynamic pages all access Web sites (dynamic page is URL in any ? Page) * User-agent: Disallow: / * *? 7, only allow search engines to access a particular file suffix form . * User-agent: Allow:. The suffix form (such as.Html,.Htm,.Php and so on). Disallow: / 8, allow search engines to access a particular directory * User-agent: Allow: / 1/ directory 2 (allowing access directory 2) Allow: / 3/ directory (4 pages in 4 allowed access directory)