mary-planet at puzzling.org
Wed May 31 09:16:06 EST 2006
On Wed, May 31, 2006, Jeff Waugh wrote:
> Thoughts? Would you use this on your own sites (particularly interested in
> answers from major public Planet administrators)?
I have used a flat-out deny policy via robots.txt on live.linuxchix.org
for its entire life. 'Chix is pretty edgy about making people's
personal stuff available for archiving, although of course those that
don't set that policy will be archived, just not through us.
robots.txt doesn't make the index/follow distinction, but I'd probably
keep doing it that way. Makes the robot requests smaller.
More information about the devel