Include noindex,follow?

[Mary Gardiner]

On Wed, May 31, 2006, Jeff Waugh wrote:
> Thoughts? Would you use this on your own sites (particularly interested in
> answers from major public Planet administrators)?

I have used a flat-out deny policy via robots.txt on live.linuxchix.org
for its entire life. 'Chix is pretty edgy about making people's
personal stuff available for archiving, although of course those that
don't set that policy will be archived, just not through us.

robots.txt doesn't make the index/follow distinction, but I'd probably
keep doing it that way. Makes the robot requests smaller.

-Mary