https://wiki.hostek.com/index.php?action=history&feed=atom&title=Robots.txt 2026-05-08T22:37:19Z Revision history for this page on the wiki MediaWiki 1.24.2 https://wiki.hostek.com/index.php?title=Robots.txt&diff=2446&oldid=prev 2015-03-25T18:41:39Z

<p>‎<span dir="auto"><span class="autocomment">Robots.txt File</span></span></p> <table class='diff diff-contentalign-left'> <col class='diff-marker' /> <col class='diff-content' /> <col class='diff-marker' /> <col class='diff-content' /> <tr style='vertical-align: top;'> <td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td> <td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 18:41, 25 March 2015</td> </tr><tr><td colspan="2" class="diff-lineno">Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>===Robots.txt File===</div></td><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>===Robots.txt File===</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>We suggest creating a '''robots.txt''' file in the web root of the domain to address two issues. First to control the rate at which the website is being crawled which can help prevent a bot/spider from creating a massive number of database connections at the same time. Second to prevent specific bots from crawling the website. We suggest the following defaults, however you might want to add or remove the user agents denied, and adjust the crawl rate <del class="diffchange diffchange-inline">but we suggest nothing lower than 3 seconds</del>.</div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>We suggest creating a '''robots.txt''' file in the web root of the domain to address two issues. First to control the rate at which the website is being crawled which can help prevent a bot/spider from creating a massive number of database connections at the same time. Second to prevent specific bots from crawling the website. We suggest the following defaults, however you might want to add or remove the user agents denied, and adjust the crawl rate.</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>&lt;pre style=&quot;white-space: pre-wrap&quot;&gt;</div></td><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>&lt;pre style=&quot;white-space: pre-wrap&quot;&gt;</div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>User-agent: *</div></td><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>User-agent: *</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>Crawl-delay: <del class="diffchange diffchange-inline">10</del></div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>Crawl-delay: <ins class="diffchange diffchange-inline">2</ins></div></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>User-agent: Baiduspider</div></td><td class='diff-marker'>&#160;</td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>User-agent: Baiduspider</div></td></tr> </table>

Jakeh https://wiki.hostek.com/index.php?title=Robots.txt&diff=986&oldid=prev 2012-12-03T09:58:37Z

<p>Created page with &quot;===Robots.txt File=== We suggest creating a &#039;&#039;&#039;robots.txt&#039;&#039;&#039; file in the web root of the domain to address two issues. First to control the rate at which the website is being ...&quot;</p> <p><b>New page</b></p><div>===Robots.txt File===<br /> We suggest creating a '''robots.txt''' file in the web root of the domain to address two issues. First to control the rate at which the website is being crawled which can help prevent a bot/spider from creating a massive number of database connections at the same time. Second to prevent specific bots from crawling the website. We suggest the following defaults, however you might want to add or remove the user agents denied, and adjust the crawl rate but we suggest nothing lower than 3 seconds.<br /> &lt;pre style=&quot;white-space: pre-wrap&quot;&gt;<br /> User-agent: *<br /> Crawl-delay: 10<br /> <br /> User-agent: Baiduspider<br /> Disallow: /<br /> <br /> User-agent: Sosospider<br /> Disallow: /<br /> &lt;/pre&gt;<br /> <br /> [[Category:Windows]]<br /> [[Category:ColdFusion]]<br /> [[Category:Railo]]<br /> [[Category:VPS]]</div>

Carlb