> that it is caused by some kind of personal search agent, or perhaps a
> misbehaved personal mirror tool. (a correcly behaving HTTP mirriring
> application should default to use GET IMS, not HEAD)
>
The Free Software Foundation's adopted mirroring
tool, wget, does this, and it is pretty well
behaved generally, e.g. it honours robots.txt and
delays between requests.
At a guess, it uses HEAD because that is all that
it can rely on being supported - it's probably
fairly safe to assume that a server that rejects
HEAD will ignore I-M-S. I think it is also a
reasonable assumption that anything that doesn't
support HEAD doesn't want to be mirrored, in spite
of having no robots.txt.
Received on Thu Sep 09 1999 - 06:06:56 MDT
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:48:22 MST