Re: Store_url_rewrite for squid 3+

On 8/09/2012 7:38 a.m., Eliezer Croitoru wrote:
>
> On 09/07/2012 05:10 PM, Alex Rousskov wrote:
>
> I am not sure what "option" you are referring to in the above. The
> Store::get(key) API I have described is not optional -- it is the
> primary way of detecting a hit.
>
> +1 to use the api.
> are we talking about the "Store::Root().get" ?
> I was just getting into it now and it seems to confused me a bit.
> and I found my answer for couple things about purge and other stuff on
> the way.
> as I see if we want to make it work we need to take another logic then
> then in 2.7 (obviously).
> I will check it and make sure I can make it stick and not break
> anything in *store* on the way.
>
> The URL rewriting must happen before or during Store key calculation.
>
> The problem is that if I am rewriting the url, unlike
> url_rewrite\redirect the connection should be initiated based on the
> original url and not the rewritten one.
> What I do not know yet is what and where and based on what the request
> is done.
> if it's based on the original request or the url.
> so we need to store the original url and the store_url at least for
> the period of time the request is being served.(answer about storing
> or not the store_url in meta)
>

Correct. Also, any revalidation requests done later must be done on the
original request URL. Not the stored URL nor the potentially different
current client request URL.
Thus we need to store the object with key being the store-url and
preserve the original for server-side use.

> url_rewrite api gets the http requests as the background to any action
> so it cannot be used for store_url since any change to the request
> will lead to the change we are trying to not do exactly.
>
> mem_object contains:
> {...
> char *url;
> HttpRequest *request;
> ...}
> and the url is being used to make the hash but I still dont know what
> is being taken from the mem_object to do the request.
> when I will know what exact object and exact point in code is being
> used to fetch the request I think I can be more clever on a way to
> implement my idea.
>
> I had the end of the redirector but I lost it somewhere in couple too
> much K*lines.
>
>> STORE_META_STOREURL is unused in Squid3 AFAICT.
> Ok, but it still there and can be used with just 2-3 lines of code.
> I have tested it and the data is written if needed but is not in use
> when reading from store.
>
>
>> Why do we need to store it?
> I do not know why we need to store it and i think that the only things
> that should be saved is the full request full reply and some data\meta
> the being will be used by the replacement policy.
>
> since I dont know how everything works and about all the api in the
> code after reviewing the 2.7 code related to store_url it's seems like
> the approach wasn't that bad using the meta data.
> since couple hours ago I noticed that it was done in order to make
> things possible and not really to think in a more wide angle about the
> effect of it.
>
> the main problem I think the STORE_META_STOREURL was maybe used for is
> if and while rebuilding the cache_dir data to be able not pass the
> url to the store_url helper.
> from this point I think it's costs space but when squid rebuilds the
> cache_dir I still dont now if it's good to pass url into the helper.

It will be costly to process maybe several millions of objects through
the helper on startup/restart.

> And for the question that pops out: if a cache_dir swap.data get's
> corrupted, what the fate of the cache in the squid3.head ? (I have
> never had such a bad luck to be able feel what it's like).

Amos
Received on Sat Sep 08 2012 - 03:14:11 MDT