and I think brad could be looking at this, to keep them that want to be ‘crawler free’ free
The three good ones so far seem to be standards for robot exclusion,killing scrapers,spider info resource.
of course, this goes counter to what I think the web is used for, and if you publish to the net, folks will find you, sooner or later.
personally, if it’s not a private entry or friend’s only, I want to be indexed like crazy, and to be found by whoever wants to find me.
Related Posts
- April 25, 2008 9254 – Friday – 18 month anniversary today. After work, BHK and I headed out to Silver Spring to meet up […]
- April 24, 2008 9253 -Thurs Lovely day today. Lots of gypsy moth and / or tent caterpillar nests in the crooks of […]
- October 17, 2007 9027 – weds 'Second Earth' found, 20 light years awayScientists have discovered a warm and rocky […]
- April 30, 2006 8331 – silly aim prank bots – “I say, old bean, have you seen my hat?” I keep getting hit by the aim bot that starts out with "i say, old bean, have you seen […]
- November 6, 2005 7900 – only 2100 more until I’ve hit 10,000 entries!
Like this:
Like Loading...
Related