robots.txt写法大全和robots.txt语法的作用(超详细)（Robots.txt writing Daquan and the role of robots.txt Grammar (super detailed)）.docVIP

robots.txt写法大全和robots.txt语法的作用(超详细)（Robots.txt writing Daquan and the role of robots.txt Grammar (super detailed)） Robots.txt robots.txt and the role of grammar writing. 1 if you allow all search engines to access the site in all parts of the words we can set up a blank text document, named root directory for the robots.txt on the site can be. Robots.txt written as follows: * User-agent: Disallow: perhaps * User-agent: Allow: / 2 if we ban all search engines to access the site of all parts of the words Robots.txt written as follows: * User-agent: Disallow: / 3 if we need a search engine to grab it, such as Baidu, Baidu index ban our website words Robots.txt written as follows: User-agent: Baiduspider Disallow: / 4 if we ban Google index of our website, in fact, with 3 examples, is the User-agent: header file name into Google Googlebot spider that will do Robots.txt written as follows: User-agent: Googlebot Disallow: / 5 if all the website search engine index we banned except Google us. Robots.txt written as follows: User-agent: Googlebot Disallow: * User-agent: Disallow: / 6 if all the website search engine index we banned except Baidu our words Robots.txt written as follows: User-agent: Baiduspider Disallow: * User-agent: Disallow: / 7 if we need to prevent spider access to a directory, such as admin, CSS, images and other prohibited directory indexed words Robots.txt written as follows: * User-agent: Disallow: /css/ Disallow: /admin/ Disallow: /images/ 8 if certain. A directory allows us to visit our site spider in it Robots.txt written as follows: * User-agent: Allow: /css/my Allow: /admin/html Allow: /images/index Disallow: /css/ Disallow: /admin/ Disallow: /images/ 9 we see some sites in the robots.txt Disallow or Allow will see a lot of symbols, such as an asterisk mark what, if you use the *, is mainly to limit access to a domain name suffixes, banned all with .Htm suffix URL to access the /html/ directory (including subdirectories). Robots.txt written as fol