queso made mention of this,

and I think brad could be looking at this, to keep them that want to be ‘crawler free’ free

The three good ones so far seem to be standards for robot exclusion,killing scrapers,spider info resource.

of course, this goes counter to what I think the web is used for, and if you publish to the net, folks will find you, sooner or later.

personally, if it’s not a private entry or friend’s only, I want to be indexed like crazy, and to be found by whoever wants to find me.

Related Posts

Leave a Reply