[squid-users] Leaking ICAP connections

I am using Squid 3.1.0.14, configured to make REQMOD and RESPMOD requests
to a local ICAP server. Everything seems to work fine, except the number
of connections between Squid and the ICAP server sometimes keeps
increasing over the course of days or weeks.

I haven't been able to figure out what is triggering the problem, but it
appears that in certain circumstances, Squid just stops using one of the
ICAP connections - from what I can tell, the ICAP server has finished
dealing with a request and is waiting for the next one, but Squid never
sends a new request. Squid continues to operate ok, bringing up more ICAP
connections to accommodate more client requests whilst the "lost"
connection stays dormant.

The ICAP server is configured to allow a maximum of 100 concurrent
connections and eventually, the number of lost connections becomes so
great that this limit is reached and the ICAP server starts rejecting the
new connections that Squid is bringing up. At this point the users start
getting Squid's ICAP error page.

Since I'm unfamiliar with the internal structure of Squid, I'm
not really sure where to begin with debugging Squid itself. I think I've
done as much debugging as is possible from the ICAP server side (this
seems to indicate that the ICAP session itself has been fine - the last
ICAP request from Squid looks fine and has terminated and the last ICAP
response from the ICAP server looks fine and the server is sat waiting for
a new request that never comes).

This problem isn't something that can be reliably worked around on the
ICAP server end - the ICAP server has no way of knowing if a connection
from Squid has been "lost" (i.e. still open but will never again be used)
or if it is simply idle. Because of this, having the ICAP server time
out idle connections would introduce a race condition - if the connection
is just idle, rather than lost, the ICAP server might time it out and
close it just as Squid starts sending a new request; in this case the
request would fail and the user would get an error page.

Any suggestions on how to debug the problem would be greatfully received.

Thanks.

-- 
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com