Re: squid-smp

On Tue, 13 Oct 2009 10:54:14 -0400, Sachin Malave <sachinmalave_at_gmail.com>
wrote:
> On Tue, Oct 13, 2009 at 8:12 AM, Amos Jeffries <squid3_at_treenet.co.nz>
> wrote:
>> Sachin Malave wrote:
>>>
>>>
>>> On Mon, Oct 12, 2009 at 8:33 PM, Amos Jeffries <squid3_at_treenet.co.nz
>>> <mailto:squid3_at_treenet.co.nz>> wrote:
>>>
>>> On Tue, 13 Oct 2009 00:29:56 +0200, Henrik Nordstrom
>>> <henrik_at_henriknordstrom.net <mailto:henrik_at_henriknordstrom.net>>
>>> wrote:
>>> > fre 2009-10-09 klockan 01:50 -0400 skrev Sachin Malave:
>>> >
>>> >> I think it is possible to have a thread , which will be
watching
>>> >> AsyncCallQueue, if it finds an entry there then it will execute
>>> the
>>> >> dial() function.
>>> >
>>> > Except that none of the dialed AsyncCall handlers is currently
>>> thread
>>> > safe.. all expect to be running in the main thread all alone.
>>>
>>> Which raises the issue of whether to add a second main queue loop
>>> for
>>> thread-safe calls. Then schedule calls which have been audited and
>>> found
>>> safe to that queue instead of the current main queue. Usage would
be
>>> low to
>>> start with but would allow ongoing incremental SMP improvements by
>>> gradually migrating chunks of code to be thread-safe.
>>>
>>> An alternate would be thread-safe the queue and add a flag to say
>>> particular calls are thread-safe. That would mean walking the queue
>>> repeatedly looking for them. Which is perhaps less desirable at the
>>> start
>>> of conversion when few calls are threaded. But gains in utility
>>> relative to
>>> the thread-safety progress.
>>>
>>> This involves a small amount of extra code in schedule() to flag
>>> which
>>> queue the calls is sent to, and a chunk of extra memory for
>>> duplicate queue
>>> management objects.
>>>
>>>
>>>
---------------------------------------------------------------------------------------------------------------
>>> Ok if that is possible then would like to make those changes, Either
>>> of
>>> them will be tried...
>>>
>>> One more thing...
>>>
>>> Are you thinking about spawning multiple threads or single thread
>>> separated from main is sufficient for handling all scheduled calls.
>>> Here multiple threads means, we could have threads all trying to dial
>>> entries in AsyncCallQueue simultaneously.....
>>
>> That would be up to you.
>>
>> I had not thought more than one dialer thread per CPU necessary at this
>> stage. Though with both verified thread-safe calls and a thread-safe
>> queue,
>> multiple dialer threads should not be an issue. Doing more than
necessary
>> would merely be a waste of resources.
>
> Yeah !!!!! I am also thinking the same...
>
>>
>> Squid would essentially segment into multiple 'main' threads / dialers
>> running one to a CPU and sharing minimal amounts of state. Slightly
more
>> efficient and far easier to configure than current setups of multiple
>> interlinked Squid instances.
>
> Ok, This could be done... will give good performance results.. But
> want to know more about this, please come again with more precise
> definition.
> Are you talking about multi-instance squid ?
> http://wiki.squid-cache.org/MultipleInstances.

Yes. That is the current way of handling SMP. Rather nasty from the admin
viewpoint.
I'm just looking far ahead to the end-result of adding multiple dialer
threads. It's a happier place :).

>>
>> Without knowing too much, I'm assuming the Job ID can be used to
identify
>> calls a particular thread/job runs.
>>
>> Amos
>>
>
>
>
>
>>>
>>>
-------------------------------------------------------------------------------------------------------------------------------
>>>
>>> >
>>> >> can we separate dispatchCalls() in EventLoop.cc for that
>>> purpose?
>>> We
>>> >> can have a thread executing distatchCalls() continuously
>>>
>>> This is an end-goal. Jumping straight there for everything is
>>> usually a
>>> mistake. But good to re-state it anyway.
>>>
>>> >> and if error
>>> >> condition occurs it is written in "error" shared variable.....
>>> which
>>> >> is then read by main thread executing mainLoop....... in the
>>> same way
>>> >> returned dispatchedSome can also be passed to main thread...
>>>
>>> I think I follow. You mean something like the way errno works in
the
>>> OS?
>>> Doing that would be a major crutch in Squid. I'd rather have an
>>> error
>>> object per-job (field in the job descriptor object) which the job
>>> handlers
>>> can use according to the job needs.
>>> Some will result in data sent back to the client, some in a
>>> completely
>>> altered handling pathway.
>>>
>>> Amos
>>>
>>>
>>>
>>>
>>> --
>>> Mr. S. H. Malave
>>> Computer Science & Engineering Department,
>>> Walchand College of Engineering,Sangli.
>>> sachinmalave_at_wce.org.in <mailto:sachinmalave_at_wce.org.in>
>>

Amos
Received on Wed Oct 14 2009 - 04:05:17 MDT