Re: [squid-users] Object Hit/Byte Hit accounting with Multiple Instances

On 16/12/10 17:37, Michael Hendrie wrote:
> On 16/12/2010, at 12:44 PM, Amos Jeffries wrote:
>> On 15/12/10 14:38, Michael Hendrie wrote:
>>> Hello List,
>>>
>>> I have server running 3 instances of squid-3.0.STABLE19 using a
>>> configuration similar to that documented at
>>> http://wiki.squid-cache.org/MultipleInstances. Each instance has all
>>> other instance configured as siblings using the "proxy-only" directive
>>> to allow sharing of cache without duplicating objects. This setup is
>>> working very well and has increased server performance by over 50%.
>>>
>>> I'm now trying to get an accurate indication of byte savings I'm
>>> achieving with this configuration however I'm not sure that the
>>> calculations I'm using are giving the correct results. Because each
>>> instance maintains a separate cache_dir this seems to be a little
>>> difficult to calculate. When instance 1 records a request as a MISS it
>>> may in fact be a HIT (from an entire system point of view) if the object
>>> is retrieved from the cache of instance 2 or 3.
>>>
>>> Using a combination of "squidclient mgr:counters" and SNMP, I grab
>>> counter values from each instance, tally and use the following formula
>>> to calculate the byte hit ratio:
>>>
>>> (mgr:counters:client_http.hit_kbytes_out +
>>> snmp:cacheClientHTTPHitKb.sibling_addresses) /
>>> (mgr:counters:client_http.kbytes_out -
>>> snmp:cacheClientHTTPHitKb.sibling_addresses) * 100 = % cache byte hit
>>> ratio
>>>
>>> Using this formula, I always seem to get inconsistencies between what
>>> squid reports and what my benchmarking tool reports (web-polygraph). In
>>> the few cases I've checked so far, squid is always reporting a 4-5% less
>>> byte hit than what web-polygraph reports.
>>
>> That sounds about the size of header overheads to me.
>> Give 3.2 workers a try out now and see if that is usable. The stats
>> calculations are fixed there for multiple workers.
>>
>
> Unfortunately I must use this version (for the moment) for reasons
> beyond my control. Just to clarify
>
> 1). Are you saying that headers aren't counted in the any of hit_kb_out
> counters so I would still see the discrepancies in figures between
> web-polygraph and a single instance squid (never had a need to check
> before now).

I'm saying 4-5% is about the header size. I can't find anywhere in code
which is eliding them but I didn't spend much time looking.

>
> 2). Excluding the fact that headers may not be counted, does the formula
> I'm using sound like the correct way to calculate hit % with a
> multi-instance setup

It make sense to me. I'd use the SNMP counters for everything though.
Calls to cachemgr will add avoidable skew. The local traffic to all
clients (peers included) can be found at cacheHttpOutKb and the local
total from all servers (peers included) at cacheServerInKb.

>
> 3). From the 3.2 wiki page - http://wiki.squid-cache.org/Features/SmpScale
> Currently, Squid workers do not share and do not synchronize other
> resources or services, including:
> • object caches (memory and disk) -- there is an active project to allow
> such sharing;
>
> Can 3.2 workers be configured with other workers as siblings to make use
> of their cache.

Yes.
They are essentially multiple instances running out of one config file.
With some new config tools/settings to make the management far easier.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.9
   Beta testers wanted for 3.2.0.3