Squid crash bug in COSS

Hi,

We're seeing one or more Squid crash bugs since we started using COSS
heavily.

When I restarted a Squid 2.6.4 server today, it kept on crashing every
few minutes, in a constant loop. Upon investigation this proved to be a
bug in COSS, during rebuilding of its cache store.

This is the backtrace given in cache.log by --enable-backtrace:

(squid)[0x428805]
(squid)[0x48dde1]
(squid)[0x48b78a]
(squid)[0x474c36]
(squid)[0x427c66]
(squid)[0x452352]
/lib/libc.so.6(__libc_start_main+0xdb)[0x2aaaab1ca49b]
(squid)[0x40417a]

addr2line gives:

root@sq13:~# addr2line -e /usr/sbin/squid
0x428805
/home/mark/wikimedia/squid-2.6.4/src/debug.c:508
0x48dde1
coss/store_dir_coss.c:1256
0x48b78a
aufs/async_io.c:315
0x474c36
/home/mark/wikimedia/squid-2.6.4/src/store_dir.c:510
0x427c66
/home/mark/wikimedia/squid-2.6.4/src/comm_epoll.c:179
0x452352
/home/mark/wikimedia/squid-2.6.4/src/main.c:860

Clearing the cache dirs (block devices) with zeros seemed to fix it.

I believe there are one or more other crash bugs in COSS, because ever
since we started using it, sometimes suddenly a lot of Squids crash at
around the same time. This doesn't happen very often, I'd say about once
a week. Overall, COSS seems very performant and promising - we're very
happy with it.

-- 
Mark Bergsma <mark@wikimedia.org>
System & Network Administrator, Wikimedia Foundation