Re: [squid-users] Squid API to External Helper.

On Tue, 24 Jul 2007, Adrian Chadd wrote:

> On Mon, Jul 23, 2007, gonzales@linuxlouis.net wrote:
>> Squid 2.6.stable14
>>
>> I've got a small dillema. I've written an external Perl helper to return
>> "OK" or "ERR" dependant upon some regular expressions and/or domains
>> stored in an external Postgresql database.
>
> Pretty simple.
>
>> What I've noticed is that for each URL/FQDN that is requested, squid
>> passes 'every' URL embedded in the webpage, one by one, to the external
>> helper(s that are running, until I add the asynchronous piece that allows
>> just one instance of the helper to run, which will then fork children
>> processes as needed). My external helper, makes a call to a postgresql db
>> and checks each domain/url against a single table of 50K entries, this
>> KILLS the performance and end user experience.
>
> Yup, run multiple helpers.
Multiple or single with ansychronous threads, being better? I would think
that running with simply a rigid number of helpers would constrain
performance, in the case of the N+1 request, where the first N helper
instances are 'busy' working. Would think that single helper that can
fork and deal with IPC would be better suited. Thoughts on this?

>
>> Does anyone have any suggestions as to 'how' to make this work well? Or
>> Henrik, do you have any suggestions as to where I might start looking in
>> the Squid code, as to how I can modify 'how' URL's are passed to the
>> external helper?
>
> See why the Postgresql lookups are taking so long. me, I'm doing
> this stuff in my spare time by building DBM type files (currently playing
> with TDB) and I'm able to do rather large hash table lookups rather quickly.
>
> SQL isn't necessarily the right tool for this.
I don't think it's the lookup, explain:
bandaboo1=> explain select * from black_list_fqdn ;
                                QUERY PLAN
------------------------------------------------------------------------
  Seq Scan on black_list_fqdn (cost=0.00..5648.59 rows=311559 width=32)
(1 row)

I think it's how many/how frequent Squid has to make entries into the
helper, is there a way for Squid to go the URL requested by the user, and
send 'every' URL from the page to the external helper at once? Rather
than one at a time, return value, enter again, ... ad infinitum.

Adrian,
thanks for initial feedback!
> >
>
> Adrian
>

-- 
Louis Gonzales
louis.gonzales@linuxlouis.net
http://www.linuxlouis.net