[squid-users] Require help with memory usage of my squid.

To the list,

I have some concerns with the amount of memory my squid process is using on one of the caching servers that I manage.

Version is 2.3.STABLE4
running on a Mandrake Linux 7.2 on COMPAQ DL380 hardware, 1Gb RAM 1.5Gb swap. Reiserfs is used for the two cache dirs.

Here is a cut down version of the current config file:

cache_mem 10485760 bytes
cache_swap_low 80
cache_swap_high 82
maximum_object_size 8388608 bytes
minimum_object_size 0 bytes
ipcache_size 1024
ipcache_low 90
ipcache_high 95
fqdncache_size 1024
cache_dir ufs /var/spool/squid/cache1 33000 77 256
cache_dir ufs /var/spool/squid/cache2 33000 77 256
memory_pools on
memory_pools_limit 52428800 bytes
replacement_policy GDSF
reference_age 604800 seconds

Store Directory Statistics:
Store Entries : 8501395
Maximum Swap Size : 67584000 KB
Current Store Swap Size: 54066291 KB
Current Capacity : 80% used, 20% free

Store Directory #0: /var/spool/squid/cache1
First level subdirectories: 77
Second level subdirectories: 256
Maximum Size: 33792000 KB
Current Size: 27039639 KB
Percent Used: 80.02%
Filemap bits in use: 2905712 of 4194304 (69%)
Filesystem Space in use: 35243136/35551996 KB (99%)
Filesystem Inodes in use: 0/-1 (0%)
Flags:

Store Directory #1: /var/spool/squid/cache2
First level subdirectories: 77
Second level subdirectories: 256
Maximum Size: 33792000 KB
Current Size: 27026652 KB
Percent Used: 79.98%
Filemap bits in use: 2900647 of 4194304 (69%)
Filesystem Space in use: 34979420/35551996 KB (98%)
Filesystem Inodes in use: 0/-1 (0%)
Flags: SELECTED

From the File Descriptors page in cachemgr.cgi:
Usually about 127 filedescriptors are open.

From the mem utilisation page from cachemgr.cgi the following is provided:
Cumulative allocated volume: 88.95 GB
Current overhead: 174007 bytes (0.031%)
Idle pool limit: 50.00 MB

Two directories with about 35Gb in each. Currently the cache is 85% full.
Roughly about 55Gb stored in the cache.

The result returned from top is

squid SIZE = 908M RSS= 908M SWAP = 259M %CPU= 5.9 %MEM = 88.7

Here are some results returned from cachemgr.cgi:

Connection information for squid:
        Number of clients accessing cache: 3
        Number of HTTP requests received: 1162624
        Number of ICP messages received: 0
        Number of ICP messages sent: 0
        Number of queued ICP replies: 0
        Request failure ratio: 0.00%
        HTTP requests per minute: 732.8
        ICP messages per minute: 0.0
        Select loop called: 16456617 times, 5.784 ms avg
Cache information for squid:
        Request Hit Ratios: 5min: 29.2%, 60min: 31.3%
        Byte Hit Ratios: 5min: 10.0%, 60min: 9.1%
        Storage Swap size: 54065426 KB
        Storage Mem size: 10232 KB
        Storage Replacement Threshold: -1.000000
        Mean Object Size: 9.31 KB
        Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
        HTTP Requests (All): 0.52331 0.52331
        Cache Misses: 0.61549 0.64968
        Cache Hits: 0.02190 0.00919
        Near Hits: 0.44492 0.49576
        Not-Modified Replies: 0.01469 0.01387
        DNS Lookups: 0.04237 0.05078
        ICP Queries: 0.00000 0.00000
Resource usage for squid:
        UP Time: 95191.283 seconds
        CPU Time: 6052.090 seconds
        CPU Usage: 6.36%
        CPU Usage, 5 minute avg: 12.94%
        CPU Usage, 60 minute avg: 12.44%
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 1199290
Memory usage for squid via mallinfo():
        Total space in arena: 897457 KB
        Ordinary blocks: 888300 KB 45964 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 35936 KB 5 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 9156 KB
        Total in use: 924236 KB 103%
        Total free: 9156 KB 1%
Memory accounted for:
        Total accounted: 545391 KB
File descriptor usage for squid:
        Maximum number of file descriptors: 1024
        Largest file desc currently in use: 157
        Number of file desc currently in use: 120
        Files queued for open: 0
        Available number of file descriptors: 904
        Reserved number of file descriptors: 100
        Store Disk files open: 12
Internal Data Structures:
        8499537 StoreEntries
          2328 StoreEntries with MemObjects
          2286 Hot Object Cache Items
        5804570 Filemap bits set
        5804554 on-disk objects

Obviously there are some major differences between the "total in use" memory and the "accounted for" memory!
As you can see, 734 http requests per min is quite busy.

Twice I've had the machine run out of memory to the point where I couldn't even login. No memory to fork the login shell. Cold reset was required.

So can someone PLEASE point me in the right direction for a solution to this problem. Do I reduce the cache_dir sizes? Install more RAM? Upgrade to 2.4.STABLE1?

Any solution that you can think of would be great!

I know that the cache_mem parameter is not going to limit mem usage as squid uses memory for other things however I need some solution to limit the memory usage of the squid process.

Is my cache_dir just to large for the machine?

This machine is only a squid caching machine. No other services are on the machine, and it is setup as the main caching proxy. Other internal proxies are chained to the squid and then the squid is chained to our ISP proxy.

Performance has been great, until now. Once the squid process starts using more swap the machine starts to bog down in swapping and the internal proxies can't connect to the squid or the performance becomes pathetic.

Any help would be greatly appreciated.
If you require further info then please let me know.

Thanks
Damien

-------------------------------------------------------------
Damien McIntosh
Internet Systems Security Administrator
Brisbane City Council
Phone: +61 7 3403 4950
Email: cioc3@brisbane.qld.gov.au
--------------------------------------------------------------

This mail has passed through an insecure network.
All enquires should be directed to the message author.
Received on Tue May 15 2001 - 22:08:17 MDT