Re: [squid-users] Squid performance profiling

On Fri, Jun 21, 2013 at 10:41 AM, Alex Rousskov
<rousskov_at_measurement-factory.com> wrote:
> On 06/20/2013 10:47 PM, Ahmed Talha Khan wrote:
>> On Fri, Jun 21, 2013 at 6:17 AM, Alex Rousskov wrote:
>>> On 06/20/2013 02:00 AM, Ahmed Talha Khan wrote:
>>>> My test methodology looks like this
>>>>
>>>> generator(apache benchmark)<------->squid<------>server(lighttpd)
>>> ...
>>>> These results show that squid is NOT CPU bound at this point. Neither
>>>> is it Network IO bound because i can get much more throughput when I
>>>> only run the generator with the server. In this case squid should be
>>>> able to do more. Where is the bottleneck coming from?
>
>
>>> The "bottleneck" may be coming from your test methodology -- you are
>>> allowing Squid to slow down the benchmark instead of benchmark driving
>>> the Squid box to its limits. You appear to be using what we call a "best
>>> effort" test, where the request rate is determined by Squid response
>>> time. In most real-world environments concerned with performance, the
>>> request rate does not decrease just because a proxy wants to slow down a
>>> little.
>
>
>> Then the question becomes why squid is slowing down?
>
> I think there are 2.5 primary reasons for that:
>
> 1) Higher concurrency level ("c" in your tables) means more
> waiting/queuing time for each transaction: When [a part of] one
> transaction has to wait for [a part of] another before being served,
> transaction response time goes up. For example, the more network sockets
> are "ready" at the same time, the higher the response time is going to
> be for the transaction which socket happens to be the last one ready
> during that specific I/O loop iteration.
>

Are these queues maintained internally inside squid? What can be done
to reduce this?

> 2a) Squid sometimes uses hard-coded limits for various internal caches
> and tables. With higher concurrency level, Squid starts hitting those
> limits and operating less efficiently (e.g., not keeping a connection
> persistent because the persistent connection table is full -- I do not
> remember whether this actually happens, so this is just an example of
> what could happen to illustrate 2a).

Can you point me to some of the key ones and their impact? So that I
can test by changing
these limits and seeing if it enhances/degrades the performance. Also,
any tweaks in
the network stack that might help with that. I am primarily interested
in enhancing the SSL performance.

>
> 2b) Poor concurrency scale. Some Squid code becomes slower with more
> concurrent transactions flying around because that code has to iterate
> more structures while dealing with more collisions and such.
>

 Well all that can be done on this front is that I have to wait for
the changes to go in.

> There is nothing we can do about #1, but we can improve #2a and #2b
> (they are kind of related).
>
>
>> best effort tests also
>> give a good measure of what the proxy(server) can do without breaking
>> it.
>
> Yes, but, in my experience, the vast majority of best-effort results are
> misinterpreted: It is very difficult to use a best-effort test
> correctly, and it is very easy to come to the wrong conclusions by
> looking at its results. YMMV.
>

 Do you see any wrong conclusion that I might have made in
interpreting these results?

> BTW, a "persistent load" test does not have to break the proxy. You only
> need to break the proxy if you want to find where its breaking point
> (and, hence, the bottleneck) is with respect to load (or other traffic
> parameters).
>
>

Sure

>> Do you see any reason from the perf results/benchmarks
>> why squid would not be utilizing all CPU and giving out more requests
>> per second?
>
> In our tests, Squid does utilize virtually all CPU cycles (if we push it
> hard enough). It is just a matter of creating enough/appropriate load.
>

Why would it not do in my test setup? I does use all CPU cores to the
fullest in the case of HTTPS, but not in the case
of HTTP as i pointed out earlier

> However, if you are asking whether Squid can be changed to run faster
> than it does today, then the answer is yes, of course. There is still a
> lot of inefficient, slow code in Squid.
>
>
> HTH,
>
> Alex.
>

--
Regards,
-Ahmed Talha Khan