Squid bug - Or machine problem?

[ooops - hit the send key by mistake - sorry :)]

Hi all.

We were running squid-2.1-release for a customer's proxy solution with FreeBSD 2.2.8-RELEASE, on an old P133, 16Megs RAM and 2.5 gigs of harddisk (1 gig allocated to the cache). They are also using it as gateway with simple IP route statements for a low traffic telnet system.

Just this morning, I got a phonecall from them - the machine was 'beeping' every three seconds. The following was from the syslog files:

>Feb 26 19:00:01 freebsd /kernel: swap_pager: out of swap space
>Feb 26 19:05:01 freebsd /kernel: pid 5057 (squid), uid 65534, was killed: out of swap space
>Feb 26 19:07:35 freebsd squid[5094]: Starting Squid Cache version 2.1.RELEASE for i386-unknown-freebsd2.2.8...
>Feb 26 19:07:35 freebsd squid[5094]: Process ID 5094
>Feb 26 19:07:35 freebsd squid[5094]: With 1024 file descriptors available
>Feb 26 19:07:35 freebsd squid[5094]: Performing DNS Tests...
>Feb 26 19:07:35 freebsd squid[5094]: Successful DNS name lookup tests...
>Feb 26 19:07:35 freebsd squid[5094]: helperOpenServers: Starting 5 'dnsserver' processes
>Feb 26 19:08:50 freebsd squid[5094]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:10:04 freebsd squid[5094]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:11:19 freebsd squid[5094]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:12:34 freebsd squid[5094]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:13:49 freebsd squid[5094]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:13:50 freebsd squid[5094]: Unlinkd pipe opened on FD 9
>Feb 26 19:13:50 freebsd squid[5094]: Swap maxSize 1048576 KB, estimated 80659 objects
>Feb 26 19:13:50 freebsd squid[5094]: Target number of buckets: 1613
>Feb 26 19:13:50 freebsd squid[5094]: Using 8192 Store buckets, replacement runs every 10 seconds
>Feb 26 19:13:50 freebsd squid[5094]: Max Mem size: 8192 KB
>Feb 26 19:13:50 freebsd squid[5094]: Max Swap size: 1048576 KB
>Feb 26 19:13:50 freebsd squid[5094]: Rebuilding storage in Cache Dir #0 (DIRTY)
>Feb 26 19:13:51 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:13:51 freebsd squid[5094]: Loaded Icons.
>Feb 26 19:13:51 freebsd squid[5094]: Accepting HTTP connections on port 3128, FD 32.
>Feb 26 19:13:51 freebsd squid[5094]: Accepting ICP messages on port 3130, FD 33.
>Feb 26 19:13:51 freebsd squid[5094]: Ready to serve requests.
>Feb 26 19:13:53 freebsd /kernel: swap_pager: out of swap space
>Feb 26 19:13:54 freebsd squid[5094]: Done reading Cache Dir #0 swaplog (17878 entries)
>Feb 26 19:13:54 freebsd squid[5094]: Finished rebuilding storage disk.
>Feb 26 19:13:54 freebsd squid[5094]: 17878 Entries read from previous logfile.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Entries scanned from swap files.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Invalid entries.
>Feb 26 19:13:54 freebsd squid[5094]: 0 With invalid flags.
>Feb 26 19:13:54 freebsd squid[5094]: 17878 Objects loaded.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Objects expired.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Objects cancelled.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Duplicate URLs purged.
>Feb 26 19:13:54 freebsd squid[5094]: 0 Swapfile clashes avoided.
>Feb 26 19:13:54 freebsd squid[5094]: Took 80 seconds ( 223.5 objects/sec).
>Feb 26 19:13:54 freebsd squid[5094]: Beginning Validation Procedure
>Feb 26 19:13:54 freebsd squid[5094]: Completed Validation Procedure
>Feb 26 19:13:54 freebsd squid[5094]: Validated 17878 Entries
>Feb 26 19:13:54 freebsd squid[5094]: store_swap_size = 145685k
>Feb 26 19:15:00 freebsd /kernel: pid 5094 (squid), uid 65534, was killed: out of swap space
>Feb 26 19:15:11 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:15:51 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:17:34 freebsd squid[5109]: Starting Squid Cache version 2.1.RELEASE for i386-unknown-freebsd2.2.8...
>Feb 26 19:17:34 freebsd squid[5109]: Process ID 5109
>Feb 26 19:17:34 freebsd squid[5109]: With 1024 file descriptors available
>Feb 26 19:17:34 freebsd squid[5109]: Performing DNS Tests...
>Feb 26 19:17:34 freebsd squid[5109]: Successful DNS name lookup tests...
>Feb 26 19:17:34 freebsd squid[5109]: helperOpenServers: Starting 5 'dnsserver' processes
>Feb 26 19:17:46 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:18:26 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:18:49 freebsd squid[5109]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:20:04 freebsd squid[5109]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:20:16 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:21:19 freebsd squid[5109]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:22:17 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:22:34 freebsd squid[5109]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:22:57 freebsd ppp[191]: tun0: Warning: Chat script failed
>Feb 26 19:23:49 freebsd squid[5109]: WARNING: Cannot run '/usr/local/squid/bin/dnsserver' process.
>Feb 26 19:23:50 freebsd squid[5109]: Unlinkd pipe opened on FD 9
>Feb 26 19:23:50 freebsd squid[5109]: Swap maxSize 1048576 KB, estimated 80659 objects
>Feb 26 19:23:50 freebsd squid[5109]: Target number of buckets: 1613
>Feb 26 19:23:50 freebsd squid[5109]: Using 8192 Store buckets, replacement runs every 10 seconds
>Feb 26 19:23:50 freebsd squid[5109]: Max Mem size: 8192 KB
>Feb 26 19:23:50 freebsd squid[5109]: Max Swap size: 1048576 KB
>Feb 26 19:23:50 freebsd squid[5109]: Rebuilding storage in Cache Dir #0 (DIRTY)
>Feb 26 19:23:51 freebsd squid[5109]: Loaded Icons.
>Feb 26 19:23:51 freebsd squid[5109]: Accepting HTTP connections on port 3128, FD 32.
>Feb 26 19:23:51 freebsd squid[5109]: Accepting ICP messages on port 3130, FD 33.
>Feb 26 19:23:51 freebsd squid[5109]: Ready to serve requests.
>Feb 26 19:23:52 freebsd /kernel: swap_pager: out of swap space
>Feb 26 19:23:52 freebsd squid[5109]: xcalloc: Unable to allocate 1 blocks of 56 bytes!
>Feb 26 19:23:53 freebsd squid[5109]: Not currently OK to rewrite swap log.
>Feb 26 19:23:53 freebsd squid[5109]: storeDirWriteCleanLogs: Operation aborted.
>Feb 26 19:23:53 freebsd /kernel: pid 5109 (squid), uid 65534: exited on signal 6
>Feb 26 19:23:56 freebsd Feb 26 19:23:56squid: xmalloc: Unable to allocate 6 bytes!
>Feb 26 19:23:56 freebsd /kernel: pid 5121 (squid), uid 0: exited on signal 6
>Feb 26 19:23:59 freebsd Feb 26 19:23:59squid: xmalloc: Unable to allocate 6 bytes!
>Feb 26 19:23:59 freebsd /kernel: pid 5122 (squid), uid 0: exited on signal 11
>Feb 26 19:24:02 freebsd Feb 26 19:24:02squid: xmalloc: Unable to allocate 6 bytes!
>Feb 26 19:24:02 freebsd /kernel: pid 5123 (squid), uid 0: exited on signal 11
>Feb 26 19:24:05 freebsd Feb 26 19:24:05squid: xmalloc: Unable to allocate 6 bytes!
>Feb 26 19:24:05 freebsd /kernel: pid 5124 (squid), uid 0: exited on signal 11
>Feb 26 19:24:08 freebsd Feb 26 19:24:08squid: xmalloc: Unable to allocate 6 bytes!
>Feb 26 19:24:08 freebsd /kernel: pid 5125 (squid), uid 0: exited on signal 11

I'm looking mainly at the kernel: out of swap space error. The DNSserver fails, are, I belive to do with the odd firewall setup that they have there - but the only way to get the machine working again was a reboot.

I have just upgraded it to 2.1-PATCH2, with clientHandleIMSReply memory leak patch also. Does anyone know then, if this is a known problem - will perhaps be fixed, or could it be the OS or the machine's hardware about to die?

Thanks for the help;

Steven Fletcher
steven@shellnet.com
Received on Sun Feb 28 1999 - 04:52:43 MST