Re: [squid-users] Heavy load squid with high CPU utilization...

Dejan,

Squid is known to be CPU bound under heavy load and the
Quad core running at 1.6 GHz in not the fastest.
A 3.2 GHz dual core will give you double speed.

The config parameter "minimum_object_size 10 KB"
prevents that objects smaller than 10 KB are not written to disk.
I am curious to know why you have this value and if you
benchmarked it, can you share the results ?

The mean object size is 53 KB and the parameter
maximum_object_size_in_memory 50 KB
implies that you have a relatively large number of hot objects
that do not stay in memory.
The memory hit % is low and the disk hit % is high, so the
maximum_object_size_in_memory should be increased.
I suggest 96 KB, monitor the memory hit % and increase more
if necessary.

client_persistent_connections and server_persistent_connections
are off. The default is on and usually gives better performance.
Why are they off ?

The TCP window scaling is off. This is a performance penalty
for large objects since it uses the select/epoll loop a lot more
because objects arrive in more smaller pieces.
Why is it off ?
If you have a good reason to set it off I recommend to use
the maximum size for fixed TCP window size: 64K (squid parameter
tcp_recv_bufsize) to reduce the number of calls to select/epoll.

You use one disk solely for cache. This can be better
if you use a battery-backed disk I/O controller with
256MB cache.
And the obvious: more disks is good for overall performance

Marcus

Dejan Zivanic wrote:
> Regards,
>
> we have heavy load (over 6k requests per minute) intercepting squid
> loading about 70-80Mbps traffic.
> I have notices that CPU usage of squid process never goes down from 50%
> and usually goes up to over 90%.
>
> We plan to upgrade to 120Mbps link and this can be major problem if we
> cannot solve it.
> Every suggestions will be appreciated...
>
> Best regards,
> Zivanic Dejan
>
> Server: FujitsuSiemens Primergy Econel 200 S2 (Quad Xeon E5310 on
> 1.6Ghz) with 6GB ECC ram.
> Storage: 2xSata udma6 250GB
> OS: Fedora14 x64
>
> OS is installed on sda1, sda2 is only for cache_dir.
>
> Squid Cache: Version 3.1.11
> configure options: '--prefix=/usr' '--localstatedir=/var'
> '--sysconfdir=/etc/squid' '--with-filedescriptors=16384'
> '--enable-removal-policies=heap,lru' '--enable-delay-pools'
> '--enable-epoll' '--enable-stopreio=ufs,aufs,diskd' '--enable-async-io'
> '--with-pthreads' '--disable-dlmalloc' '--with-large-files'
> '--enable-htcp' '--enable-large-cache-files' '--enable-wccpv2'
> '--enable-esi' '--with-aio' '--with-dl' '--enable-ltdl-convenience'
> '--enable-linux-netfilter' --with-squid=/root/install/squid-3.1.11
>
> Linux 2.6.35.6-45.fc14.x86_64 #1 SMP Mon Oct 18 23:57:44 UTC 2010 x86_64
> x86_64 x86_64 GNU/Linux
> squid]# uptime
> 20:27:36 up 4 days, 22:07, 3 users, load average: 0.93, 0.84, 0.85
> squid]# free
> total used free shared buffers cached
> Mem: 6125216 4956620 1168596 0 377916 2568624
> -/+ buffers/cache: 2010080 4115136
> Swap: 4194296 9824 4184472
>
> Squid Object Cache: Version 3.1.11
> Start Time: Mon, 21 Mar 2011 12:42:30 GMT
> Current Time: Tue, 22 Mar 2011 19:19:01 GMT
> Connection information for squid:
> Number of clients accessing cache: 999
> Number of HTTP requests received: 11471323
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 6246.2
> Average ICP messages per minute since start: 0.0
> Select loop called: 320350087 times, 0.344 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 10.0%, 60min: 9.4%
> Hits as % of bytes sent: 5min: 12.2%, 60min: 11.9%
> Memory hits as % of hit requests: 5min: 11.5%, 60min: 10.3%
> Disk hits as % of hit requests: 5min: 60.4%, 60min: 59.5%
> Storage Swap size: 72469364 KB
> Storage Swap capacity: 39.3% used, 60.7% free
> Storage Mem size: 1038552 KB
> Storage Mem capacity: 100.0% used, 0.0% free
> Mean Object Size: 53.58 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.22004 0.23230
> Cache Misses: 0.27332 0.25890
> Cache Hits: 0.01648 0.01745
> Near Hits: 0.25890 0.18699
> Not-Modified Replies: 0.00179 0.00179
> DNS Lookups: 0.07618 0.08334
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 110191.361 seconds
> CPU Time: 41856.331 seconds
> CPU Usage: 60.99%
> CPU Usage, 5 minute avg: 75.64%
> CPU Usage, 60 minute avg: 81.66%
> Process Data Segment Size via sbrk(): 1474024 KB
> Maximum Resident Size: 5983984 KB
> Page faults with physical i/o: 13
> Memory usage for squid via mallinfo():
> Total space in arena: 1474156 KB
> Ordinary blocks: 1467177 KB 10620 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 24216 KB 9 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 6978 KB
> Total in use: 1491393 KB 100%
> Total free: 6978 KB 0%
> Total size: 1498372 KB
> Memory accounted for:
> Total accounted: 1332348 KB 89%
> memPool accounted: 1332348 KB 89%
> memPool unaccounted: 166023 KB 11%
> memPoolAlloc calls: 2768926328
> memPoolFree calls: 2896226918
> File descriptor usage for squid:
> Maximum number of file descriptors: 16384
> Largest file desc currently in use: 1069
> Number of file desc currently in use: 867
> Files queued for open: 0
> Available number of file descriptors: 15517
> Reserved number of file descriptors: 100
> Store Disk files open: 8
> Internal Data Structures:
> 1364435 StoreEntries
> 65986 StoreEntries with MemObjects
> 65640 Hot Object Cache Items
> 1352638 on-disk objects
>
> sysctl.conf:
> //
> fs.file-max = 360000
> vm.drop_caches = 3
> vm.swappiness = 3
> net.ipv4.ip_local_port_range = 2048 65000
> net.core.rmem_max = 16777216
> net.core.wmem_max = 32777216
> net.ipv4.tcp_low_latency = 1
> net.core.netdev_max_backlog = 4000
> net.ipv4.tcp_no_metrics_save = 0
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_timestamps = 0
> net.ipv4.tcp_window_scaling = 0
> net.ipv4.tcp_sack = 0
> net.ipv4.tcp_timestamps = 0
> net.ipv4.tcp_mem = 786432 1048576 1572864
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 87380 16777216
> net.ipv4.tcp_tw_recycle = 1
> net.ipv4.tcp_tw_reuse = 1
> vm.min_free_kbytes = 70000
> net.core.somaxconn = 65536
> net.ipv4.ip_forward = 1
> net.ipv4.conf.default.rp_filter = 1
> net.ipv4.conf.default.accept_source_route = 0
> kernel.sysrq = 0
> kernel.core_uses_pid = 1
> net.ipv4.tcp_syncookies = 1
> kernel.msgmnb = 65536
> kernel.msgmax = 65536
> kernel.shmmax = 68719476736
> kernel.shmall = 4294967296
> net.netfilter.nf_conntrack_max = 1048576
> net.nf_conntrack_max = 1048576
> net.netfilter.nf_conntrack_buckets = 1048576
> net.netfilter.nf_conntrack_tcp_max_retrans = 3
> net.netfilter.nf_conntrack_tcp_be_liberal = 0
> net.netfilter.nf_conntrack_tcp_loose = 3
> net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 100
> net.netfilter.nf_conntrack_log_invalid = 0
> net.netfilter.nf_conntrack_generic_timeout = 600
> net.netfilter.nf_conntrack_icmp_timeout = 10
> net.netfilter.nf_conntrack_udp_timeout_stream = 180
> net.netfilter.nf_conntrack_udp_timeout = 10
> net.netfilter.nf_conntrack_tcp_timeout_close = 10
> net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
> net.netfilter.nf_conntrack_tcp_timeout_last_ack = 10
> net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
> net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 10
> net.netfilter.nf_conntrack_tcp_timeout_established = 21600
> net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 5
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 5
> net.netfilter.nf_conntrack_checksum = 1
> //
>
> squid.conf:
> //
> http_port 5555 intercept
> icp_port 0
> maximum_icp_query_timeout 3000
> mcast_icp_query_timeout 2000
> dead_peer_timeout 10 seconds
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> no_cache deny QUERY
> cache_mem 1024 MB
> cache_swap_low 98
> cache_swap_high 99
> maximum_object_size 250 MB
> minimum_object_size 10 KB
> maximum_object_size_in_memory 50 KB
> ipcache_size 16384
> ipcache_low 90
> ipcache_high 95
> cache_replacement_policy heap LFUDA
> memory_replacement_policy heap GDSF
> cache_dir aufs /cache00 180000 32 256
> cache_access_log /var/log/squid/access.log common
> cache_log /var/log/squid/cache.log
> cache_store_log none
> cache_swap_log /var/spool/squid/cache_swap_log
> log_ip_on_direct off
> pid_filename /var/run/squid.pid
> debug_options ALL,1
> dns_timeout 20 seconds
> dns_nameservers 212.200.191.150 212.200.113.130
> auth_param basic children 15
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
> auth_param basic casesensitive off
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern . 0 20% 4320
> forward_timeout 20 seconds
> connect_timeout 10 seconds
> peer_connect_timeout 10 seconds
> read_timeout 30 seconds
> request_timeout 10 seconds
> persistent_request_timeout 10 seconds
> client_lifetime 360 minutes
> half_closed_clients off
> pconn_timeout 10 seconds
> shutdown_lifetime 10 seconds
> acl manager proto cache_object
> acl localhost src 127.0.0.1
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443 563
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 563 # https, snews
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny opstina2 !vreme
> http_access allow kladovonet
> http_access allow kladovonet-public
> http_access deny all
> tcp_outgoing_address 172.29.203.200
> cache_peer 172.29.202.252 parent 8080 7 name=dsl1 round-robin no-query
> no-digest weight=2
> cache_peer 172.29.202.253 parent 8080 7 name=dsl2 round-robin no-query
> no-digest weight=2
> cache_peer 172.29.202.251 parent 8080 7 name=dsl3 round-robin no-query
> no-digest weight=2
> acl kroz-adsl url_regex -i "/etc/adsl"
> cache_peer_access adsl1 allow kroz-adsl
> cache_peer_access adsl2 allow kroz-adsl
> cache_peer_access adsl3 allow kroz-adsl
> never_direct allow kroz-adsl
> http_reply_access allow all
> icp_access deny all
> cache_mgr Zivanic_Dejan_[zivanicd_at_kladovonet.com]
> cache_effective_user squid
> logfile_rotate 10
> memory_pools off
> error_directory /usr/share/errors/sr-latn
> ignore_unknown_nameservers off
> client_persistent_connections off
> server_persistent_connections off
> //
>
>
>
>
Received on Wed Mar 23 2011 - 02:27:33 MDT