RE: [squid-users] squid: Memory utilization higher than expected since moving from 3.3 to 3.4 and Vary: working

Well - there are 4 Points here:
a) this is gradual behavior that takes month to accumulate
b) extra used memory is much bigger than Cache (1.5x aprox), which looks wrong to me as the other parameters you have given (diskcache,...) are not
c) we have not seen this behavior with 3.3 and have just switched to 3.4 two month ago and it is running since then showing this increase.
d) if you look at the memory report (mgr:mem) which I have shared you find: 900MB for 542k HttpHeaderEntries and 2GB for 791k Short_Strings - and that is almost 3 out of 4.2G for this report alone.

So I was guessing there was some sort of memory allocation bug or something - maybe something to do with the now working handling of "vary" headers, that triggers this memory leak.
To me it looks as if those "Vary" nodes are not clean up properly and that is why we are slowly increasing memory...

There are specially a few things in the msg:info block that made me suspicious:
Memory usage for squid via mallinfo():
        Total space in arena: -2064964 KB
        Ordinary blocks: 2047465 KB 161900 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 38432 KB 9 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 81875 KB
        Total in use: 81875 KB -4%
        Total free: 81875 KB -4%
        Total size: -2026532 KB
Memory accounted for:
        Total accounted: 1269447 KB -63%
        memPool accounted: 9658055 KB -477%
        memPool unaccounted: -11684587 KB -0%
        memPoolAlloc calls: 507567155336
        memPoolFree calls: 508713005568
Especially those negative Numbers and percentages - but maybe that is just a reporting bug, which - if so - is hindering debugging the real thing

Thanks,
        Martin

P.s: it would be helpful if the output of mgr:mem would be formatted in a way that would allow it to get easily pasted into Excel or similar - no spaces in pool names would be sufficient...


-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Montag, 07. Juli 2014 15:40
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] squid: Memory utilization higher than expected since moving from 3.3 to 3.4 and Vary: working

On 2014-07-07 20:46, Martin Sperl wrote:
> Hi!
>
> We have found out that since we moved from squid 3.3 to 3.4.3 (and the
> corresponding Vary) the memory utilization of squid has increased.
> It is 10GB right now 9.6G in memory, but we have only configured 4GB
> for in Memory caching.

Yes. 4GB for memory caching *only*, and all your reports confirm the RAM
cache allocated 4GB worth of memory blocks for caching purposes.

But Squid uses memory for other things as explained by the FAQ.

<http://wiki.squid-cache.org/SquidFaq/SquidMemory#I_set_cache_mem_to_XX.2C_but_the_process_grows_beyond_that.21>

<http://wiki.squid-cache.org/SquidFaq/SquidMemory#Why_does_Squid_use_so_much_memory.21.3F>

Amos


This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp
Received on Tue Jul 08 2014 - 07:38:18 MDT