Hot tags and Sweet Chin Music (wasRe: IEEE doesn't like web robots )

Mr. FoRK
Wed, 8 Jan 2003 12:34:33 -0800

----- Original Message -----
From: "Tom" <>

> I agres, seing as is just one tiny little web server with
> relativley little on it...I mean what would happen if I put my mirror of
> Projet Guttenburg, soem more old timem radio shows, a few php/mysql
> dynamic pages and other fun project stuff.

I don't know much about how robots/spiders work, but wouldn't they respect
caching information - like 'this file never expires' and stuff?
Then there are spiders that keep hitting links even though the server says
they are permanantly gone - 404 response status codes.