Re: [squid-users] Squid slows under load

Hi again,

Went into work the next day and carried on only to find that one of the
SSDs was spewing CRC failures into the log. I don;t recall seeing any
of these in the previous days and the logs didn;t contain any entries
prior to my email so I assume that was a new problem.

I've pulled the SSDs and replaced then with 160GB SATA II disks. I then
ran curl-loader with 150 client load and the resulting 5 minute stats
dump is posted below.

sample_start_time = 1299342747.28318 (Sat, 05 Mar 2011 16:32:27 GMT)
sample_end_time = 1299343047.45842 (Sat, 05 Mar 2011 16:37:27 GMT)
client_http.requests = 115.309931/sec
client_http.hits = 0.003333/sec
client_http.errors = 0.000000/sec
client_http.kbytes_in = 25.411849/sec
client_http.kbytes_out = 195.575242/sec
client_http.all_median_svc_time = 0.012346 seconds
client_http.miss_median_svc_time = 0.013086 seconds
client_http.nm_median_svc_time = 0.000000 seconds
client_http.nh_median_svc_time = 0.000000 seconds
client_http.hit_median_svc_time = 0.000000 seconds
server.all.requests = 0.299982/sec
server.all.errors = 0.000000/sec
server.all.kbytes_in = 1.573241/sec
server.all.kbytes_out = 0.333314/sec
server.http.requests = 0.266651/sec
server.http.errors = 0.000000/sec
server.http.kbytes_in = 1.103269/sec
server.http.kbytes_out = 0.253319/sec
server.ftp.requests = 0.000000/sec
server.ftp.errors = 0.000000/sec
server.ftp.kbytes_in = 0.000000/sec
server.ftp.kbytes_out = 0.000000/sec
server.other.requests = 0.033331/sec
server.other.errors = 0.000000/sec
server.other.kbytes_in = 0.469973/sec
server.other.kbytes_out = 0.076662/sec
icp.pkts_sent = 0.000000/sec
icp.pkts_recv = 0.000000/sec
icp.queries_sent = 0.000000/sec
icp.replies_sent = 0.000000/sec
icp.queries_recv = 0.000000/sec
icp.replies_recv = 0.000000/sec
icp.replies_queued = 0.000000/sec
icp.query_timeouts = 0.000000/sec
icp.kbytes_sent = 0.000000/sec
icp.kbytes_recv = 0.000000/sec
icp.q_kbytes_sent = 0.000000/sec
icp.r_kbytes_sent = 0.000000/sec
icp.q_kbytes_recv = 0.000000/sec
icp.r_kbytes_recv = 0.000000/sec
icp.query_median_svc_time = 0.000000 seconds
icp.reply_median_svc_time = 0.000000 seconds
dns.median_svc_time = 4.177065 seconds
unlink.requests = 0.000000/sec
page_faults = 0.000000/sec
select_loops = 202.268185/sec
select_fds = 498.770865/sec
average_select_fd_period = 0.002005/fd
median_select_fds = 0.000000
swap.outs = 0.063330/sec
swap.ins = 0.000000/sec
swap.files_cleaned = 0.000000/sec
aborted_requests = 0.049997/sec
syscalls.polls = 202.268185/sec
syscalls.disk.opens = 0.063330/sec
syscalls.disk.closes = 0.063330/sec
syscalls.disk.reads = 0.000000/sec
syscalls.disk.writes = 0.406643/sec
syscalls.disk.seeks = 0.000000/sec
syscalls.disk.unlinks = 0.000000/sec
syscalls.sock.accepts = 220.630446/sec
syscalls.sock.sockets = 99.554185/sec
syscalls.sock.connects = 0.216654/sec
syscalls.sock.binds = 99.554185/sec
syscalls.sock.closes = 209.474431/sec
syscalls.sock.reads = 147.911360/sec
syscalls.sock.writes = 242.719155/sec
syscalls.sock.recvfroms = 2.516520/sec
syscalls.sock.sendtos = 0.079995/sec
cpu_time = 8.350000 seconds
wall_time = 300.017524 seconds
cpu_usage = 2.783171%

Whether the SSD replacement will give a real world cure will be seen Monday.

Thanks again,

Julian

On 03/03/11 17:59, Pieter De Wit wrote:
> Hi Julian,
>
> The one stat that I can't see here is disk access. I know you said
> that you have SSD's, but what is the disk stats for your logging
> volume and the squid volume ? If you totally bypass the proxy, does it
> improve ? (could be that the squid server is getting shaped ?)
>
> Cheers,
>
> Pieter
>
> On 4/03/2011 06:46, Julian Pilfold-Bagwell wrote:
>> Hi All,
>>
>> I've been having some problems with Squid and Dansguardian for a
>> while now and despite lots of time on Google, haven't found a solution.
>>
>> The problem started a week or so back when I noticed that squid was
>> slowing. A quick look through the logs showed it was running out of
>> file descriptors so I upped the level to take account. The server
>> was ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz
>> G6950 CPU & 4GB of RAM). At the same time, I bought in 2 x 60GB SSD
>> drives to use as cache space with the system on a RAID 1 array with
>> 160GB SATA II disks.
>>
>> On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from
>> apt) and Dansguardian 2.10.1.1. The kernel version is
>> 2.6.32-24-server and the server authenticates via a Samba PDC (v
>> 3.5.6) using OpenLDAP/Winbind. The Samba version on the proxy
>> machine is v 3.4.7 as supplied from the Ubuntu repo.
>>
>> This however also seems to run out of steam. My first thought was
>> that it may have been running out of RAM so I ran htop. Both CPUs
>> were topping out at 20% and out of the 4GB of RAM, 1.3GB was used.
>> Next I checked the load on the NIC and found that it was running on
>> average 400kB/s, with the odd burst at 5MB/s. As the load increased,
>> web pages were taking up to 30-45 seconds to load. I bypassed
>> Dansguardian and went in on 3128 with no change in performance.
>>
>> Following the recommendations on other sites discovered via Google, I
>> tuned and tweaked settings with no real benefit and I can't see that
>> I changed anything to cause it to happen. The log files look fine, I
>> have 10000 file descriptors available and cachemgr shows plenty of
>> spares. There are 50% more NTLM authenticators than are in use at any
>> given time.
>>
>> The config file for Squid is shown below. I have had the number of
>> authenticators set to 400 as I have 350 users but the number in use
>> still peaked at around 50. If I've been a numpty and done something
>> glaringly obvious, I'd be grateful if someone could point it out. If
>> not, ask for info and I'll provide it.
>>
>> Thanks,
>>
>> Jools
>>
>>
>> ## Squid.conf
>> ## Start with authentication for clients
>>
>> auth_param ntlm program /usr/bin/ntlm_auth
>> --helper-protocol=squid-2.5-ntlmssp
>> auth_param ntlm_param children 100
>> auth_param ntlm keep_alive on
>>
>> auth_param basic program /usr/bin/ntlm_auth
>> --helper-protocol=squid-2.5-basic
>> auth_param basic children 100
>> auth_param basic realm Squid proxy-caching web server
>> auth_param basic credentialsttl 2 hours
>>
>> ## Access Control Lists for filter bypass ##
>> acl realtek dstdomain .realtek.com.tw
>> acl tes dstdomain .tes.co.uk
>> acl glogster dstdomain .glogster.com
>> acl adobe-installer dstdomain .adobe.com # allow installs from adobe
>> download manager
>> acl actihealth dstdomain .actihealth.com .actihealth.net # Allow
>> direct access for PE dept activity monitors
>> acl spybotupdates dstdomain .safer-networking.org .spybotupdates.com
>> # Allow updates for Spybot S&D
>> acl sims-update dstdomain .kcn.org.uk .capitaes.co.uk
>> .capitasolus.co.uk .sims.co.uk # Allow SIMS to update itself directly
>> acl kcc dstdomain .kenttrustweb.org.uk # Fix problem with county
>> acl frenchconference dstdomain flashmeeting.e2bn.net
>> acl emsonline dstdomain .emsonline.kent.gov.uk
>> acl clamav dstdomain .db.gb.clamav.net
>> acl ubuntu dstdomain .ubuntu.com .warwick.ac.uk
>> acl windowsupdate dstdomain windowsupdate.microsoft.com
>> acl windowsupdate dstdomain .update.microsoft.com
>> acl windowsupdate dstdomain download.windowsupdate.com
>> acl windowsupdate dstdomain redir.metaservices.microsoft.com
>> acl windowsupdate dstdomain images.metaservices.microsoft.com
>> acl windowsupdate dstdomain c.microsoft.com
>> acl windowsupdate dstdomain www.download.windowsupdate.com
>> acl windowsupdate dstdomain wustat.windows.com
>> acl windowsupdate dstdomain crl.microsoft.com
>> acl windowsupdate dstdomain sls.microsoft.com
>> acl windowsupdate dstdomain productactivation.one.microsoft.com
>> acl windowsupdate dstdomain ntservicepack.microsoft.com
>> acl windowsupdate dstdomain download.adobe.com
>> acl comodo dstdomain download.comodo.com
>> acl simsb2b dstdomain emsonline.kent.gov.uk
>> acl powerman dstdomain pmstats.org
>> acl ability dstdomain ability.com
>> acl fulston dstdomain fulstonmanor.kent.sch.uk
>> acl httpsproxy dstdomain .retiredsanta.com .atunnel.com .btunnel.com
>> .ctunnel.com .dtunnel.com .ztunnel.com .partyaccount.com
>>
>> ## Access Control for filtered users ##
>> acl all src 0.0.0.0/0.0.0.0
>> acl manager proto cache_object
>> acl localhost src 127.0.0.1/255.255.255.255
>> acl to_localhost dst 127.0.0.0/8
>> acl SSL_ports port 443
>> acl ntlm_users proxy_auth REQUIRED
>>
>> acl SSL_ports port 443 # https
>> acl SSL_ports port 563 # snews
>> acl SSL_ports port 873 # rsync
>> acl Safe_ports port 80 # http
>> acl Safe_ports port 21 # ftp
>> acl Safe_ports port 443 # https
>> acl Safe_ports port 70 # gopher
>> acl Safe_ports port 210 # wais
>> acl Safe_ports port 1025-65535 # unregistered ports
>> acl Safe_ports port 280 # http-mgmt
>> acl Safe_ports port 488 # gss-http
>> acl Safe_ports port 591 # filemaker
>> acl Safe_ports port 777 # multiling http
>> acl Safe_ports port 631 # cups
>> acl Safe_ports port 873 # rsync
>> acl Safe_ports port 901 # SWAT
>> acl purge method PURGE
>> acl CONNECT method CONNECT
>>
>> ## Allow/Deny Lists ##
>> http_access allow manager localhost
>> http_access deny manager
>> http_access allow purge localhost
>> http_access deny purge
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>>
>> http_access allow emsonline
>> http_access allow clamav
>> http_access allow realtek
>> http_access allow ubuntu
>> http_access allow tes
>> http_access allow glogster
>> http_access allow kcc
>> http_access allow fulston
>> http_access allow comodo
>> http_access allow ability
>> http_access allow powerman
>> http_access allow windowsupdate
>> http_access allow simsb2b
>> http_access allow adobe-installer
>> http_access allow actihealth
>> http_access allow spybotupdates
>> http_access allow sims-update
>> http_access allow frenchconference
>> http_access allow ntlm_users
>> http_access deny httpsproxy
>> http_access allow localhost
>> http_access deny all
>> icp_access deny all
>>
>> ## Cache Settings ##
>> log_fqdn off
>> half_closed_clients off
>> maximum_object_size 1024 KB
>> cache_access_log none
>> cache_store_log none
>> http_port 3128
>> redirect_children 750
>> hierarchy_stoplist cgi-bin ?
>> cache_mem 128 MB
>> memory_replacement_policy lru
>> cache_replacement_policy lru
>> cache_dir ufs /fastcache1 15000 16 256
>> cache_dir ufs /fastcache2 15000 16 256
>> refresh_pattern ^ftp: 1440 20% 10080
>> refresh_pattern ^gopher: 1440 0% 1440
>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>> refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880
>> refresh_pattern . 0 20% 4320
>> acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
>> upgrade_http0.9 deny shoutcast
>> acl apache rep_header Server ^Apache
>> broken_vary_encoding allow apache
>> extension_methods REPORT MERGE MKACTIVITY CHECKOUT
>> cache_effective_user proxy
>> ## Hash out effective group as it stops access to winbind privileged
>> pipe and breaks authentication - jpb
>> # cache_effective_group proxy
>> max_filedescriptors 10000
>> dns_nameservers 172.20.0.253 172.31.49.46 172.31.81.46
>> hosts_file /etc/hosts
>> coredump_dir /var/spool/squid
>>
>
>

-- 
Julian Pilfold-Bagwell,
Network Manager,
Borden Grammar School,
Avenue of Remembrance,
Sittingbourne,
Kent,
ME10 4DB.
Tel: 01795 424192