Re: [squid-users] high memory usage (squid 3.2.0)

Mr Dash Four wrote:
> Alex Rousskov wrote:
>> On 04/10/2013 09:22 AM, Mr Dash Four wrote:
>>
>>
>>> the total memory used of all objects placed in ram should
>>> not be 6 times the cache_mem value
>>>
>>
>> There is no such rule or law of physics. cache_mem is not squid_mem (a
>> not-yet-supported option that would limit total Squid memory usage).
>>
>> The total memory used by Squid is, roughly:
>>
>> 1) initial memory used by Squid shortly after start +
>> 2) memory used to store memory-cached objects +
>> 3a) memory used to index memory-cached objects +
>> 3b) memory used to index disk-cached objects +
>> 4) memory used for current transactions +
>> 5) memory leaked since Squid start
>>
>> cache_mem determines the maximum value of (2)
>>
>> cache_mem and mean object size in memory cache determine the maximum
>> value of (3a) because they determine the maximum number of objects in
>> the memory cache.
>>
>> cache_mem has no significant direct effect on the other four components.
>> Depending on the local circumstances, every other component alone may
>> exceed cache_mem 5 times or more, especially when cache_mem is
>> relatively small, e.g., 200MB. And taken together, they can exceed
>> 6*cache_mem level even easier.
>>
>> For example, a 10KB/s memory leak will consume 1GB of RAM in less than
>> two days. Similarly, an overloaded Squid may grow (4) at rates exceeding
>> 1MB/s.
>>
> Well, in that case, whoever did the testing of squid before releasing
> such a buggy software should be shot on site or be locked in a dark
> garage 24/7 somewhere and taught how to use valgrind.
>
>> Now back to your actual problem. An idle Squid with 200 MB memory cache
>> and 200 MB disk cache should not use 1.3GB of RAM unless component (1)
>> is very high due to some huge ACL lists or other unusual complications.
>>
> Nope. I have 5 rules, that's all.
>
>> If your (1) is just a few MBs and your idle Squid uses 1.3GB, then I
>> would suspect a memory leak.
>>
> At start, squid memory consumption is about 25-30MB - give or take a
> few megabytes, keeping in mind that according to the logs I have in
> excess of 14500 objects (verified by squid at startup) in the disk cache.
>
>> If you want to reduce Squid memory usage, I recommend eliminating memory
>> leaks as suspects first because they are bugs and because they may
>> consume all available RAM on your machine.
> I am not a squid developer - I am a user.
>
>> Whether your Squid leaks
>> memory, I do not know. I do know that you are running v3.2.0 that
>> probably does have leaks and a set of other nasty bugs fixed in
>> currently supported Squid versions.
>>
> I'll try to see if the latest version has the above behaviour and if
> that turns out to be the case and I have found no solution the the
> problem I described in my initial post, then I have to just forget
> that squid exists and use something else which is more reliable, like
> polipo for example.
>
> I do thank you for your time and your comments though - much appreciated.
I am keeping the entire post to which I reply, because it has been quite
a while since I posted in this thread.

Just to recap - I was using squid 3.2.0 and had massive memory problems
with my system (atom-based machine with 2GB RAM) due to squid
cannibalising my entire free memory pool, using in excess of 1.3GB of it
(keeping in mind that I have "restricted" squid to use 200MB, not more
than 6 times that amount).

After a period of discussing various issues on this thread, I decided to
upgrade squid to 3.3.4 and see if that cures the problem. Well, it did!

My squid memory usage currently stands at little over 104MB and that is
after nearly 2 months of continues use, bearing in mind that with the
previous version (3.2.0) that memory was in excess of 1.3GB in just over
a month of such use. I did not post these results earlier as I wanted to
be sure that this isn't some sort of fluke and also that the new squid
is capable of running for much longer than the the old one, while
keeping the memory use in check.

It needs to be noted that there is no (significant) differences in the
configuration between the old and the new versions, apart from minor
corrections to account for changed option names (transition from 3.2.0
to 3.3.4). There isn't any difference in the usage (nature or intensity)
either. Apart from a new kernel, the only difference is the new squid
version.

Conclusion: squid 3.2.0 should be confined to the dust bin!

I've read an advisory which recommends upgrading to 3.3.7 (and now 3.3.8
is released), so I'll upgrade to that in the coming days, but wanted to
post this update in the meantime for the benefit of this group.
Received on Sat Aug 10 2013 - 11:22:38 MDT