Re: [squid-users] Intermittent Hangups with Squid and NTLM Auth

On Tue, Jul 31, 2007, Brian Kirk wrote:
> Red Hat ES release 4 update 4
> Squid 2.6 Stable 9
> Samba 3.0.23
>
> We from time to time have problems with the ntlm_auth where every
> process in the NTLM User Authenticator Stats on the cachemgr.cgi seems
> to be hung. We have tried from 30 processes up to 100 processes, but
> it doesn't help. Seems to happen during high traffic times.

I've seen this under Squid/NTLM. It was mitigated by Henrik's IP
authentication cache work. I didnt have the time to sort out exactly
why samba/winbind was falling behind on processing requests but it
was being handed ~200 authentication requests a second during
peak time.

I'd love to sort it out for good but I don't have a good windows test
environment and the above patch solved the problem for the client.

Can you correlate these crashes against request rates and see if
they match? (You said traffic, I don't know if that means requests
or bytes..)

Adrian

> We see core files in the /var/cache directory.
>
> We see these lines repeated over again when it first started today in
> the /var/log/samba/winbindd.log:
> [2007/07/31 09:58:58, 0] nsswitch/winbindd.c:process_loop(832)
> winbindd: Exceeding 200 client connections, no idle connection found
> [2007/07/31 09:58:58, 0] nsswitch/winbindd.c:process_loop(813)
> winbindd: Exceeding 200 client connections, no idle connection found
>
> We start seeing these messages in the /var/log/squid/cache.log:
> 2007/07/31 09:57:31| WARNING: up to 405 pending requests queued
> 2007/07/31 09:57:31| Consider increasing the number of
> ntlmauthenticator processes to at least 505 in your config file.
> 2007/07/31 09:57:49| urlParse: Illegal character in hostname
> 'www.harpoonharry's.com'
> 2007/07/31 09:58:01| WARNING: All ntlmauthenticator processes are busy.
> 2007/07/31 09:58:01| WARNING: up to 478 pending requests queued
> 2007/07/31 09:58:01| Consider increasing the number of
> ntlmauthenticator processes to at least 578 in your config file.
> 2007/07/31 09:58:12| storeDirWriteCleanLogs: Starting...
> 2007/07/31 09:58:12| WARNING: Closing open FD 145
> 2007/07/31 09:58:12| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
> on fd=145: (1) Operation not permitted
> 2007/07/31 09:58:12| 65536 entries written so far.
> 2007/07/31 09:58:12| Finished. Wrote 71404 entries.
> 2007/07/31 09:58:12| Took 0.0 seconds (2146262.3 entries/sec).
> FATAL: Too many queued ntlmauthenticator requests (501 on 100)
> Squid Cache (Version 2.6.STABLE9): Terminated abnormally.
>
> I have found message in samba groups but no one every responded to it:
> http://lists.samba.org/archive/samba/2005-September/110414.html
>
> Any help would be greatly appreciated.
>
> Thank you,
> Brian Kirk

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level bandwidth-capped VPSes available in WA -