Re: [squid-users] CARP Failover behavior - multiple parents chosen for URL

Moving this to squid-dev due to increasingly propellerhead-like
content... :)

Looking over the code and some debugging output, it's pretty clear
what's happening here.

The carpSelectParent() function does the appropriate hashing of each
URL+parent hash and the requisite ranking of the results. To determine
whether or not the highest-hash-value parent is the parent that
should, in fact, be returned, it uses peerHTTPOkay() as its test.

The problem here is that peerHTTPOkay only returns 0 if the peer in
question has been marked DEAD; carpSelectParent has no way of knowing
if the peer is down unless squid has "officially" marked it DEAD.

So, if the highest-ranked peer is a peer that is refusing connections
but isn't marked DEAD yet, then peer_select tries to use it, and when
it fails, falls back to ANY_PARENT - this actually shows up in the
access.log, which I didn't realize when I initially sent this in. Once
we've tried to hit the parent 10 times, we officially mark it DEAD,
and then carpSelectParent() does the Right Thing.

So, we have a couple option here as far as how to resolve this:

1. Adjust PEER_TCP_MAGIC_COUNT from 10 to 1, so that a parent is
marked DEAD after only one failure. This may be overly sensitive
however. Alternatively, carpSelectParent() can check peer->tcp_up and
disqualify the peer if it's not equal to PEER_TCP_MAGIC_COUNT; this
will have a similar effect without going through the overhead of
actually marking the peer DEAD and then "reviving" it.

2. Somehow have carpSelectParent() return the entire sorted list of
peers, so that if the to choice is found to be down, then
peer_select() already knows where to go next...

3. Add some special-case code (I'm guessing this would be either in
forward.c or peer_select.c) so that if a connection to a peer selected
by carpSelectParent() fails, then increment a counter (which would be
unique to that request) and call carpSelectParent() again. This
counter can be used in carpPeerSelect to ignore the X highest-ranked
entries. Once this peer gets officially declared DEAD, this becomes
moot.

Personally, I'm partial to #3, but other approaches are welcome :)

Thanks,

-C

On May 6, 2009, at 10:13 PM, Amos Jeffries wrote:

>>
>> On May 6, 2009, at 8:14 PM, Amos Jeffries wrote:
>>
>>>> Hi,
>>>>
>>>> I've noticed a behavior in CARP failover (on 2.7) that I was
>>>> wondering
>>>> if someone could explain.
>>>>
>>>> In my test environment, I have a non-caching squid configured with
>>>> multiple CARP parent caches - two servers, three per box (listening
>>>> on
>>>> ports 1080/1081/1082, respectively, for a total of six servers.
>>>>
>>>> When I fail a squid instance and immediately afterwards run GETs to
>>>> URLs that were previously directed to that instance, I notice that
>>>> the
>>>> request goes to a different squid, as expected, and I see the
>>>> following in the log for each request:
>>>>
>>>> May 6 11:43:28 cdce-den002-001 squid[1557]: TCP connection to
>>>> http-
>>>> cache-1c.den002 (http-cache-1c.den002:1082) failed
>>>>
>>>> And I notice that the request is being forwarded to a different,
>>>> but
>>>> consistent, parent.
>>>>
>>>> After ten of the above requests, I see this:
>>>>
>>>> May 6 11:43:41 cdce-den002-001.den002 squid[1557]: Detected DEAD
>>>> Parent: http-cache-1c.den002
>>>>
>>>> So, I'm presuming that after ten failed requests, the peer is
>>>> considered DEAD. So far, so good.
>>>>
>>>> The problem is this: During my test GETs, I noticed that
>>>> immediately
>>>> after the "Detected DEAD Parent" message was generated, the parent
>>>> server that the request was being forwarded to changed - as if
>>>> there's
>>>> an "interim" decision made until the peer is officially declared
>>>> DEAD,
>>>> and then another hash decision made afterwards. So while consistent
>>>> afterwards, it's apparent that during the failover, the parent
>>>> server
>>>> for the test URL changed twice, not once.
>>>>
>>>> Can someone explain this behavior?
>>>
>>> Do you have 'default' set on any of the parents?
>>> It is entirely possible that multiple paths are selected as usable
>>> and
>>> only the first taken.
>>>
>>
>> No, my cache_peer config options are
>>
>> cache_peer http-cache-1a.den002 parent 1080 0 carp http11 idle=10
>> <repeat for each hostname>
>>
>>
>>
>>> During the period between death and detection the dead peer will
>>> still be
>>> attempted but failover happens to send the request to another
>>> location.
>>> When death is detected the hashes are actual re-calculated.
>>>
>>
>> OK, correct me if I misread, but my understanding of the spec is that
>> each parent cache gets its own hash value, each of which is then
>> combined with the URL's hash to come up with a set of values. The
>> parent cache corresponding with the highest result is the cache
>> chosen. If that peer is unavailable, the next-best peer is selected,
>> then the next, etc etc.
>>
>> If that is correct, what hashes are re-calculated when a dead peer is
>> detected? Any why would those hashes result in different results than
>> the pre-dead peer run of the algorithm
>>
>> And more importantly, will that recalculation result in URLs being
>> re-
>> mapped that weren't originally pointed to the failed parent? I
>> thought
>> avoiding such an arbitrary re-mapping was the whole point of the CARP
>> algorithm.
>>
>> -C
>>
>>> If anyone wants a task it may be useful to see whether leaving dead
>>> peers
>>> in the existing hash and omitting the dead peers at the selection
>>> time
>>> instead of connection time is more responsive like this while
>>> reducing the
>>> double-change.
>>>
>>
>> Again, I'm not clear on the difference between the two - educate me
>> please :)
>
> If you have actually read the CARP specs you are one up on me. I've
> only
> read the peer selection code.
> Given a few minutes to think about it yes, your point above about URL
> 'stickiness' must be right.
>
> I was thinking that the peer hash of the pairing gets re-calculated
> since
> a peer #n has dropped of the possible list. Maybe thats not true. In
> which
> case one of the peer choices you have noticed should not be
> happening at
> all. My thought (without any backing) is that the second choice (after
> death) is probably the right one which should have been made at the
> first
> failover point.
>
>
> Amos
>
Received on Tue May 12 2009 - 03:15:32 MDT