Re: Store_url_rewrite for squid 3+

OK,
it seems we are getting to somewhere.
i know how to patch using command but what are the proper one to get a
patch file to be run later? will look into it.

Eliezer

On 9/6/2012 3:58 AM, Amos Jeffries wrote:
> On 06.09.2012 11:58, Eliezer Croitoru wrote:
>> On 9/5/2012 9:56 AM, Eliezer Croitoru wrote:
>>> any leads,?
>> Well there is a nice progress.
>> I reviewed the 2.7 store_url_rewrite and I divided the task into more
>> detailed smaller tasks.
>
> FTR: squid-2.7 ports are exempted as suitable in most cases for
> back-porting to stable despite our "no new features" policy.
> I am happy for this to be done as a series of patches instead of a
> singular change. It can be assembled in trunk and back-ported as a
> singular later.
>
>>
>> 1. Research the url_rewrite interface code and Introduce a modified
>> version of url_rewrite as url_store_rewrite_program.
>> - this task is kind of done(passed compiling and running on 3.2.1) by
>> now and I want to get some ideas on naming conventions for the code to
>> fit the project amazing code looks.
>>
>> list of changed files and code:
>> create a mimic file of redirect.cc in ./store_rewrite.cc and change
>
> No need.
>
> What I meant earlier about src/redirect.cc being usable is that most of
> the code is an exact duplicate. You should only need to:
> * write a new start() function ie storeurlRewriteStart()
> * add new storeurl_rewriters global
>
> * adapt redirectRegisterWithCacheManager() to register a new
> "storeurl_rewrite" report (when that part is written)
> * adapt redirectInit() to setup both url_rewrite and storeurl-rewrite
> helpers.
> * adapt redirectShutdown() to cleanup both url_rewrite and
> storeurl-rewrite helpers.
fair
>
> The new storeurlRewriteStart() to be used by store code for this
> re-writing and sets up the redirect interface using the new store
> helpers and whatever callback the changed URL is to be sent to. The
> entire rest of the code for helper management is identical.
>
>
>> all the methods and variables to fit store_rewrite.
>> strip all the url_rewrite data manipulating actions.
>> change the debugging info.
>> (after the store related planning tasks get back here to redo)
>>
>> ./structs.h
>> adding the proper variables for:
>> helper naming, bypass(on\off), acl_access namespace, child configs
>> the ??_rewrites_host of url_rewrite dosnt belong for store_rewrite
>> process at all.
>>
>> ./cache_cf.cc
>> state the default configuration for the helper
>>
>
> What do you mean by this?
> doConfigure() post-configuration checking?
>
> I don't think there is anything which needs new cache_cf.cc code. The
> parsing side if things is identical for url_rewrite_*. The different
> defaults and locations are all coded in cf.data.pre ...
>
ok so later we will see how ot optimize the conf file.
but there are coupe arguments there that are crucial for compilation and
parsing the config file
>> ./cf.data.pre
>> stating all config directives for the the helper (copy and modify
>> from url_rewrite_program)
>>
>
> Okay. However, if merging the stages to trunk separately this will need
> to be the final step, since it makes the directives publicly visible. We
> want to make these changes and doc/release-notes/release-3.*.sgml at the
> same time when it is suitable for public use.
>
> Also, remove the storeurl_* entries at the top marking them as
> "obsolete"/unavilable type.
>
>> ./ClientRequestContext.h
>> adding int for state
>> adding bool for done
>>
>> ./client_side_request.h
>> stating the start method as squidexternal something
>
> Just "extern" will do. We are killing the SQUIDCEXTERN mess.
>
>>
>> ./client_side_request.cc
>> adding calls and callouts
>>
>> ./protos.h
>> stating the start init and shutdown methods.
>
> We are in the process of killing protos.h. Please create a
> store_urlrewrite.h header for these definitions instead.
>
will be done

> However, see the comments above about redirect.cc. There should not need
> to be any new files created for this. Which removes the protos.h,
> main.cc and Makefile.am changes.
>
>>
>> ./main.cc:
>> calling init and shutdown methods at start/reconfigure etc..
>>
>> ./Makefile.am && ./Makefile.in
>> adding the source ?.cc file into the commands
>>
>
> Other than the notes above. Okay.
>
> If you have a patch for that please submit for audit :-)
>
> We can pause there for the infrastructure to look fine before moving on
> to the store details. I've been waiting on assistance from Henrik or
> Alex on that for a while. They are the ones who know the answers to your
> questions below AFAIK.
>
>>
>> 2. Research the workflow of storing objects in memory and store and
>> introduce psudo for a new workflow of storing objects to avoid bad
>> effects on cache objects usage in any form that can be.
>> - I do know that squid uses some hash look-up and I have seen in the
>> things about it.
>> - as far I understood from the code:
>> client_request builds the request of the http object.
>> creates a mem-object and on the way creates a checksum.
>> a transfer from of the mem-object to a "store" happens.
>> if a store rebuild happens it takes all of the data from the file in
>> the store.
>>
>> ? question how cachemgr gets the list of urls in memory?
>>
>> so probable points of failure:
>> using the wrong url to fetch the object.
>> wrong arguments for checksum.
>> storing with wrong arguments\url leading to faulty rebuild.
>>
>> I do remember that when I looked at a stored old store_url_rewrite
>> cahce file long time ago there were two urls in the file what leads me
>> (it's a bit fogy) to think that the stored file was the memobject
>> cache rather then a set of arguments such as refresh time related
>> info,method,url,request,response.
>>
>> I will look at it later but if someone have solid knowledge on how
>> the store routing was or implemented before i'm rushing into the code
>> every piece of info will help me when looking into it.
>>
>> Eliezer
>

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il