Hot tags and Sweet Chin Music (wasRe: IEEE doesn't like webrobots )

Luis Villa louie@ximian.com
08 Jan 2003 16:30:58 -0500


On Wed, 2003-01-08 at 15:35, Mr. FoRK wrote:
> ----- Original Message -----
> From: "Luis Villa" <louie@ximian.com>
> >
> > Yeah, see, this is the difference; we were getting not hundreds, but
> > tens of thousands of hits from the google bots. Per second.
> That seems excessive, especially from Google. Perhaps it was a bug that they
> resolved?

No, it's just bugzilla (1) not having a robots.txt by default and (2)
having lots of links (from stored reports and queries, mainly) that lead
to ever-changing URLs. Basically, bugzilla (particularly gnome's
bugzilla, with lots of semi-hard-coded reports) looks a hell of a lot
like one of those spam-trap CGIs that people used to design to trap
spambots- lots of URLs, all linking to each other, and always with
slightly different URLs and such so that a moron^Wbugbitch^Wbot can
cycle through them endlessly. 

Luis 'bugbitch' Villa