Re: [squid-users] Tuning for very expensive bandwidth links

Hi

>> My thought was to investigate having the internet side proxy add etag
>> headers to all content based on some quality hash function. Then have
>> the (expensive) remote side proxy rewrite the request headers to always
>> use If-None-Match? The idea is that the bandwidth is cheap on internet
>> connected side, so it can refresh it's cache of the whole page, generate
>> a new hash, but still return a "not modified" response if the end result
>> is the same string of bytes. How much of that can I implement in Squid
>> 3.x today..?
>
> 3.1.10+ will validate If-None-Match and ETag, but will not add them to
> requests itself.

Thanks - can you expand on what it means to "validate" in this case?

I think you mean that if the content is cached with a given eTag then
requests for that content will be returned from cache if the request has
an appropriate If-None-Match - is this the case?

>> Note, I realise this could lead to some side effects where the action of
>> visiting the web page itself causes some other side effect, however, I
>> think this is a manageable problem for this requirement?
>>
>> Thanks for any pointers to ideas or other products that might help?
>
> ICAP or eCAP would be the way to go here for quick results. Making a
> plugin to do the ETag generation and alterations before sending off.

Understood.

So the remote (client) side proxy would need an eCAP plugin that would
modify the initial request to include an ETag. This would require some
ability to interrogate what we have in cache and generate/request the
ETag associated with what we have already - do you have a pointer to any
API/code that I would need to look at to do this?

Then on the internet side proxy we would do whatever we need to retrieve
the content, say fetch the asset. Then our eCap on that side would
generate a consistent ETag using our favourite hash function?

The part I'm unsure how to implement would be examining what's in
squid's cache in order to generate an ETag based on what we have got (ie
for remote side)?

> You could also look at cutting bodies off 304 replies at the Internet
> side to avoid the bandwidth expensive TCP_REFRESH_UNMODIFIED responses.

Hmm, yes that would be very sensible. Apart from via eCAP are there
other ways I might do that?

> NP: if you want to go ahead and alter Squid code adding If-None-Match on
> outbound requests is an open bug. As is proper ETag variant caching
> support.

I don't know if I have the time/ability to hack on squid code? Is there
someone who might be interested on working on this for an affordable fee?

Thanks for the very helpful feedback. Note if there are any existing
ecap/icap modules I should look at then please educate me? (I'm
currently using "Ziproxy" and looking at moving the interesting bits to
a Squid ecap module. I have also used "Rabbit" proxy which is somewhat
similar)

Thanks for your comments

Ed W
Received on Thu Mar 31 2011 - 23:09:17 MDT