Re: [squid-users] Re: Squid CPU 100% infinite loop

On 18/05/2013 8:14 a.m., Stuart Henderson wrote:
> On 2013-05-17, Loïc BLOT <loic.blot_at_unix-experience.fr> wrote:
>> I have found the problem. In fact it's the problem mentionned on my last
>> mail, is right. Squid FD limit was reached, but squid doesn't mentionned
>> every time the freeze appear that it's a FD limit problem, then the
>> debug was so difficult.
>> Also, i think you must add a non portable check to check if squid FD
>> limit isn't greater than system ulimit (ulimit -n under OpenBSD). Squid
>> doesn't mention that and when this appear, it's also a freeze (squid FD
>> saturation) and no warning is mentionned in the logs. This check would
>> be at the squid start/reload.
>>
>> What do you think about it ?
> Checking (e.g. via getrlimit) to see if the limit is too small is the
> wrong approach. System calls needing FDs can fail if *either* the current
> process hits its limits, *or* if the system file table is full.
>
> Various software in OpenBSD has been audited for problems in this
> area fairly recently, here are a few of the commits which resulted
> showing examples of various approaches we used,
>
> nginx: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=c2d55a68002aae78bcd8ba0198f0a2d3f92cd777
> tmux: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=a5b691fe6fa8b648e7d7465167a96f667e40295c
> unbound: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=b9781e8c60007fbc60cf8a5509154b0283874682
> openbgp: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=bee609c47fa933f06b924435e7f41c661a8a3b40
> identd: http://anoncvs.estpak.ee/cgi-bin/cgit/openbsd-src/commit/?id=9ca56282c96c6482f23cbd459d2ab9d0cc55a68e
>
> cheers,
> Stuart
>

All of which are checking for and handling ENFILE / EMFILE errors. If
you audit Squid you will find that this has been present since forever.

Squid contains getrlimit() checks at startup/reconfigure to ensure that
the config files and environment are suitably aligned and cut the limit
down to whatever the minimum of those are. ~100 FD are "reserved" below
the actual limit to allow for extra FD which may be necessarily opened
while servicing the N-100'th connection.

When Squid encounters ENFILE/EMFILE at run-time it cuts the limit down
to _below_ the newely detected maximum and *pauses* new inbound
connections until enough FD have been closed to start accepting new
clients. You can see this logged as "Reserved FD adjusted from X to Y"
at the point where FD limits are changed.

Amos
Received on Sat May 18 2013 - 04:43:52 MDT