Re: [squid-users] Squid and Squidguard using high disk IO

On 11/09/2013 05:04 PM, Rafael Akchurin wrote:
> Hello Kaya,
>
> May I recommend to try using qlproxy together with your Squid?
> Qlproxy is an ICAP web filtering which may in your particular case do better as Squid Guard. At least you may give it a try to compare if the disk io goes down.
>
> Best regards,
> Raf

I'll take a look at it - thanks!

I was also thinking about using Adzapper but I'll do more reading and
figure out which is the best one for my setup.

> Is this line too high: url_rewrite_children 500
> YES!!

Oops.... the guide I was working from suggested that.

> Basically how can I get the IO usage down and get the system to work
> again?
> For how many users exactly?
> Just a note that I am not in a favor of any OS by default but I would
> feel better Using Linux.

At the moment I'm just testing with one user! Using sqtop I can see that
there are 30+ connections being passed to Squid.

But overall this runs on my main router; hence I can't use Linux due to
the fact that the router is running OpenBSD and needs some special stuff
from the OS.

> In order to find the right number of children start with 40 and see if
> it fits you and then see what is the bottle neck in the whole setup.
>
> Eliezer

I tried 5 and it was a bit better but not too much.... I just cranked it
up to 40 now.

I also disabled DNS lookups from squidclamav.conf which seems to have
helped a bit though still am experiencing issues. :-(

As mentioned above I am thinking of running Adzapper and then chaining
squidGuard on that; though it might just be squidclamav that's causing
this???

The issue seems to get resolved after stopping Squid, then killing the
remaining squidguard processes so it's really confusing as to where to
look for the "bottleneck".

Regards,

Kaya

>
> -----Original Message-----
> From: Kaya Saman [mailto:kayasaman_at_gmail.com]
> Sent: Saturday, November 09, 2013 4:58 PM
> To: squid-users_at_squid-cache.org
> Subject: [squid-users] Squid and Squidguard using high disk IO
>
> Hi,
>
> I'm wondering if anyone has any ideas on this one.....
>
> Basically I have created a standard Squid proxy using Squid 3.3.8 built from OpenBSD ports - OS version is OpenBSD 5.4 Current.
>
> Additionally from ports as well I have installed squidGuard 1.4p6.
>
>
> The configuration seems ok as everything is working; the acls setup in squidGuard are redirecting to the proper "blocked" page when unwanted information is embedded in a site: eg. ads, p%rn.
>
> Here is the rule list:
>
> dest ads {
> domainlist blacklists/ads/domains
> urllist blacklists/ads/urls
> }
>
> dest adv {
> domainlist blacklists/adv/domains
> urllist blacklists/adv/urls
> }
>
> dest spyware {
> domainlist blacklists/spyware/domains
> urllist blacklists/spyware/urls
> }
>
> dest porn {
> domainlist blacklists/porn/domains
> urllist blacklists/porn/urls
> expressionlist blacklists/porn/expressions
> # Logged info is anonymized to protect users' privacy
> log anonymous dest/porn.log
> }
>
> acl {
> lan {
> # The built-in 'in-addr' destination group matches any IP address.
> pass !ads !adv !porn all
> }
> default {
> # Default deny to reject unknown clients
> pass none
> redirect http://127.0.0.1/blocked.html
>
> }
> }
>
> I removed the "spyware" option from the 'lan' acl as I'm trying to debug currently....
>
> squidGuard is called by Squid using these lines in the squid.conf:
>
> # Path to the redirector program
> url_rewrite_program /usr/local/bin/squidGuard
>
> # Number of redirector processes to spawn url_rewrite_children 500
>
> # To prevent loops, don't send requests from localhost to the redirector
> url_rewrite_access deny localhost
>
>
> The issue I'm currently seeing is that the disk IO process is hammered???
>
> The 'lan' clients are therefor unable to access the web through the proxy.
>
> Running 'top' and 'ps' I can see that squidGuard has spawned many processes which seems to be causing the high IO usage.
>
> The systems' hardware is quite powerful with 8GB RAM and a Xeon E5 CPU @3.6GHz, currently being tested with 3x lan machines.
>
>
> What can I do to improve performance with this?
>
>
> Is this line too high: url_rewrite_children 500
>
> or simply have a misconfigured something?
>
>
> I additionally have 'c-icap' running with squidclamav coupled to clamd
> in case that is of importance - not using the squidGuard line in the
> squidclamav.conf file!!!
>
> Basically how can I get the IO usage down and get the system to work again?
>
> - the logs don't indicate anything outside of 'starting squidGuard
> process' many times.
>
>
> Regards,
>
>
> Kaya
>
Received on Sat Nov 09 2013 - 18:29:27 MST