Re: [squid-users] Re: False disk full errors

On Mon, Sep 21, 1998 at 11:15:19PM +0200, Henrik Nordstrom wrote:

> Chris Tilbury wrote:
>
> > 1998/09/20 23:42:27| diskHandleWrite: FD 29: disk write error: (28) No space left on device
>
> A wild speculation is that you ran out of FS transaction log space or
> something similar. Other possible causes is a hardware write error or
> SCSI bus error.
>
> If Sun has done their job correctly there should be a message in the
> syslog stating what the real error is.

Indeed. There's the very helpful message

Sep 21 13:35:25 bluebell unix: NOTICE: realloccg /proxy/cache: file system full

Daryl Collins (daryl@beaker.htb.com.au) has mailed me to tell me that he's
having the same problems with Solaris 2.6 and squid beta versions since
1.2b19.

From what Daryl has discovered, it looks like a problem with the allocation
of space within large filesystems and fragmentation, under Solaris 2.6 (he's
not using disksuite, and has 20x4.2Gb disks).

I've done a little digging and found a Sun bug report (#4141030) to the
effect that "newfs" was broken in Solaris 2.6 and that it passes incorrect
parameters to mkfs when it generates filesystems. In essence, newfs will
create the filesystem with too few cylinder groups:

root@bluebell [scripts]# newfs -Nv /dev/md/rdsk/d32
mkfs -F ufs -o N /dev/md/rdsk/d32 24914320 80 19 8192 1024 256 1 90 8192 t 0 -1 8 16
/dev/md/rdsk/d32: 24914320 sectors in 16391 cylinders of 19 tracks, 80 sectors
        12165.2MB in 261 cyl groups (63 c/g, 46.76MB/g, 5888 i/g)

instead of

root@bluebell [scripts]# newfs -Nv -c 16 /dev/md/rdsk/d32
mkfs -F ufs -o N /dev/md/rdsk/d32 24914320 80 19 8192 1024 16 1 90 8192 t 0 -1 8 16
/dev/md/rdsk/d32: 24914320 sectors in 16391 cylinders of 19 tracks, 80 sectors
        12165.2MB in 1025 cyl groups (16 c/g, 11.88MB/g, 1472 i/g)

(the latter is what a working 2.5.1 would have done). Daryl mentions he's
seen the problem under 2.5.1, but it occurs less frequently and at higher
disk capacities. This sounds quite fishy, therefore.

Alas, the only workaround for this bug is to recreate the filesystem. I've
done this and now all I can do is wait for another week to go by to see what
happens when I hit the same cache storage levels (we couldn't afford the
cache downtime needed to backup and restore the cache filesystem).

Failing this, about the only alternative I can see is to use something like
VxFS, although I'm not sure how suitable that is for squid, either.

Cheers,

Chris

-- 
Chris Tilbury, UNIX Systems Administrator, IT Services, University of Warwick
EMAIL: cudch+s@csv.warwick.ac.uk PHONE: +44 1203 523365(V)/+44 1203 523267(F)
                            URL: http://www.warwick.ac.uk/staff/Chris.Tilbury