Re: More problems with NOVM version

So spake "Miguel A.L. Paraz" <map@iphil.net>:

>I've hit the problem in the NOVM Squid version with two separate Linux
>machines - one with Linux 2.0.25, the other with 1.2.13. Both are
>64 MB RAM and 2.5 GB swap machines. From the looks of it the TCP
>socket stops accepting connections, but does not refuse them - it just
>hangs. Strangely, a USR2 signal to stimulate detailed logging output
>reawakens it, but only for a while. The resulting log output is normal
>though it cannot be used to trace much since these are very busy caches,
>with lots of requests, and the machines have to be restarted very
>quickly. :)

This sounds somewhat like a problem I had that I have pretty well
traced to routing problems. You may or may not be experiencing the
same.

I was running squid (not NOVM; don't recall exact version) on a
firewall (chrooted; have since moved it elsewhere), and after much time
spent trying to figure out what the problem was, I noticed SYN-ACK
packets on the *outside* of the firewall for nets located on the
inside. (TCP uses a 3-way handshake to setup connections; client:SYN,
server:SYN-ACK, client:ACK.)

It turned out that certain nets on our WAN had a route to me, but I did
not have a route back to them. So the SYN-ACK packets were never
received by the client, so the client never sent the ACK packet. I
conjectured that the server was filling up with half-open connections
and could not accept any more until they timed out. This would explain
why I had the exact same problem with each of three entirely
independent HTTP proxies that I tried; it was not specific to Squid.

Could this be the per-process limit of 5 (on sunos) of pending
connections in the listen(2) queue? It did not affect the other
processes on the system. Sending a HUP signal to Squid causes it to
close the listen socket (I presume); don't know about USR2, though. So
on closing the socket either the kernel would shed those half-open
connections bound to it, or at least they would not be ascribed to the
new socket that Squid is listening on. (Would it even be possible to
program around this problem with the BSD Sockets API?)

We have a moderately busy connection, so I wrote a Perl script to tail
the Squid access log and send a HUP signal to Squid whenever the last
message was more than 2 minutes old. This helped to keep user
complaints down until I could figure out what the heck was happening.
In this case I fixed it with the introduction of some static routes on
the machine to the nets in question.

--
KH