robot text generator, make your own robots.txt
for all you web designers out there. This little tool generates a robots.txt for dissallowing robots to enter speified directories.
What’s more useful is that if you tick the box it will add a load of code to disallow a whole host of evil spambots which could harvest your email address. I normaly don’t allow my email address on any of my sites in plain text form anyway – but this is another level of defense.
Who knows if they obey it!
Filed under: Web Technology on October 10th, 2004
Leave a Reply