Hot tags and Sweet Chin Music (wasRe: IEEE doesn't like web robots )

Mr. FoRK fork_list@hotmail.com
Wed, 8 Jan 2003 12:34:33 -0800


----- Original Message -----
From: "Tom" <tomwhore@slack.net>

> I agres, seing as wsmf.org is just one tiny little web server with
> relativley little on it...I mean what would happen if I put my mirror of
> Projet Guttenburg, soem more old timem radio shows, a few php/mysql
> dynamic pages and other fun project stuff.

I don't know much about how robots/spiders work, but wouldn't they respect
caching information - like 'this file never expires' and stuff?
Then there are spiders that keep hitting links even though the server says
they are permanantly gone - 404 response status codes.