Re: [squid-users] Repost: Squid 3.1.x uses considerably more memory than 2.7.x or 3.0.x

On Wed, 28 Jul 2010 18:13:03 +0200, Ralf Hildebrandt
<Ralf.Hildebrandt_at_charite.de> wrote:
> * Murilo Moreira de Oliveira <murilo.moreira_at_gmail.com>:
>> Hi guys. I'm using squid 3.1.4 and since of first 3.1 stable release
I'm
>> suffering from memory leak problems too. My production server is a
CentOS
>> 5.4 32 bits (kernel 2.6.18-164.6.1.el5). It has 2GB of RAM (512MB
>> reserved
>> to squid cache_mem) and 6GB of disk space reserved to squid's disk
>> cache. In
>> approximately 24h, squid consumes all the available memory of my server
>> and
>> swap starts to be consumed. I'm currently restarting squid once a day
in
>> order to workaround the problem. Follow attached a squidclient output
>> made
>> some hours before all the server memory be consumed.
>
> Today I performed some more tests and completely disabled delay_pools
> in 3.1.5 and checked if that may fix the excessive memory usage
>
> After a short while, memory use skyrocketed - again.
>
> One interessant observation I made is that squid-rrd/1day.cgi reports a
> sudden spike in "Cached Objects -- 1 day"
>
> With 3.0.x, the number of objects cached in memory was more or less
> constant (see green line), once 3.1.5 took over, the numbers skyrocket
> (peak at 1.8M, meaning it's actually only 18.000 items)

maximum_object_in_memory default was increased a bit in 3.1 to encompass
most of the bell curve of small 'average' objects. 3.0 and earlier only
memory-cached the lower half of the range.
Unless you explicitly configured cache_mem its default size has also taken
a bigger jump to accommodate ~10,000 of the average memory cacheable
objects.
3.1 will now also do true promotion of objects from disk into memory cache
if they become hot.

Two other changes in 3.1 related to HTTP/1.1 which have since been shown
to indirectly affect the hot objects:

 * There are two confirmed major content providers whose server software
sends out different cacheability headers based on whether the client is
HTTP/1.0 or HTTP/1.1. They seem to let 1.1 clients cache more than 1.0
clients.

 * during the 1.0->1.1 upgrade of replies Squid synthesizes a Date: header
which increases the cacheability of many responses up from zero.

>
> The image can be found here:
http://www.arschkrebs.de/bugs/squid-object.png
>

Access forbidden apparently.

> This image had been created using poll.pl:
>
> # Originally at:
> # http://wessels.squid-cache.org/squid-rrd/
> # Rewritten by Dan Kogai <dankogai_at_dan.co.jp>
> # Rewritten by Duane Wessels <wessels_at_squid-cache.org>
> # poll.pl,v 1.9 2004/03/30 18:55:27 wessels Exp
>
> According to the source, these (the green graph) are created from the
> "hot" items:
>
> $Vals{nhot} = $1 if (/(\d+)\s+Hot Object Cache Items/i);
>
>
> objects => {
> DS => {
> disk => { format => '%d', type => 'GAUGE', valkey =>
> 'nobj', },
> mem => { format => '%d', type => 'GAUGE', valkey => 'nhot', },

Amos
Received on Thu Jul 29 2010 - 04:28:43 MDT