Re: Store_url_rewrite for squid 3+

On 06.09.2012 11:58, Eliezer Croitoru wrote:
> On 9/5/2012 9:56 AM, Eliezer Croitoru wrote:
>> any leads,?
> Well there is a nice progress.
> I reviewed the 2.7 store_url_rewrite and I divided the task into more
> detailed smaller tasks.

FTR: squid-2.7 ports are exempted as suitable in most cases for
back-porting to stable despite our "no new features" policy.
I am happy for this to be done as a series of patches instead of a
singular change. It can be assembled in trunk and back-ported as a
singular later.

>
> 1. Research the url_rewrite interface code and Introduce a modified
> version of url_rewrite as url_store_rewrite_program.
> - this task is kind of done(passed compiling and running on 3.2.1) by
> now and I want to get some ideas on naming conventions for the code
> to
> fit the project amazing code looks.
>
> list of changed files and code:
> create a mimic file of redirect.cc in ./store_rewrite.cc and change

No need.

What I meant earlier about src/redirect.cc being usable is that most of
the code is an exact duplicate. You should only need to:
  * write a new start() function ie storeurlRewriteStart()
  * add new storeurl_rewriters global

  * adapt redirectRegisterWithCacheManager() to register a new
"storeurl_rewrite" report (when that part is written)
  * adapt redirectInit() to setup both url_rewrite and storeurl-rewrite
helpers.
  * adapt redirectShutdown() to cleanup both url_rewrite and
storeurl-rewrite helpers.

The new storeurlRewriteStart() to be used by store code for this
re-writing and sets up the redirect interface using the new store
helpers and whatever callback the changed URL is to be sent to. The
entire rest of the code for helper management is identical.

> all the methods and variables to fit store_rewrite.
> strip all the url_rewrite data manipulating actions.
> change the debugging info.
> (after the store related planning tasks get back here to redo)
>
> ./structs.h
> adding the proper variables for:
> helper naming, bypass(on\off), acl_access namespace, child configs
> the ??_rewrites_host of url_rewrite dosnt belong for store_rewrite
> process at all.
>
> ./cache_cf.cc
> state the default configuration for the helper
>

What do you mean by this?
  doConfigure() post-configuration checking?

I don't think there is anything which needs new cache_cf.cc code. The
parsing side if things is identical for url_rewrite_*. The different
defaults and locations are all coded in cf.data.pre ...

> ./cf.data.pre
> stating all config directives for the the helper (copy and modify
> from url_rewrite_program)
>

Okay. However, if merging the stages to trunk separately this will need
to be the final step, since it makes the directives publicly visible. We
want to make these changes and doc/release-notes/release-3.*.sgml at the
same time when it is suitable for public use.

Also, remove the storeurl_* entries at the top marking them as
"obsolete"/unavilable type.

> ./ClientRequestContext.h
> adding int for state
> adding bool for done
>
> ./client_side_request.h
> stating the start method as squidexternal something

Just "extern" will do. We are killing the SQUIDCEXTERN mess.

>
> ./client_side_request.cc
> adding calls and callouts
>
> ./protos.h
> stating the start init and shutdown methods.

We are in the process of killing protos.h. Please create a
store_urlrewrite.h header for these definitions instead.

However, see the comments above about redirect.cc. There should not
need to be any new files created for this. Which removes the protos.h,
main.cc and Makefile.am changes.

>
> ./main.cc:
> calling init and shutdown methods at start/reconfigure etc..
>
> ./Makefile.am && ./Makefile.in
> adding the source ?.cc file into the commands
>

Other than the notes above. Okay.

If you have a patch for that please submit for audit :-)

We can pause there for the infrastructure to look fine before moving on
to the store details. I've been waiting on assistance from Henrik or
Alex on that for a while. They are the ones who know the answers to your
questions below AFAIK.

>
> 2. Research the workflow of storing objects in memory and store and
> introduce psudo for a new workflow of storing objects to avoid bad
> effects on cache objects usage in any form that can be.
> - I do know that squid uses some hash look-up and I have seen in the
> things about it.
> - as far I understood from the code:
> client_request builds the request of the http object.
> creates a mem-object and on the way creates a checksum.
> a transfer from of the mem-object to a "store" happens.
> if a store rebuild happens it takes all of the data from the file in
> the store.
>
> ? question how cachemgr gets the list of urls in memory?
>
> so probable points of failure:
> using the wrong url to fetch the object.
> wrong arguments for checksum.
> storing with wrong arguments\url leading to faulty rebuild.
>
> I do remember that when I looked at a stored old store_url_rewrite
> cahce file long time ago there were two urls in the file what leads
> me
> (it's a bit fogy) to think that the stored file was the memobject
> cache rather then a set of arguments such as refresh time related
> info,method,url,request,response.
>
> I will look at it later but if someone have solid knowledge on how
> the store routing was or implemented before i'm rushing into the code
> every piece of info will help me when looking into it.
>
> Eliezer
Received on Thu Sep 06 2012 - 00:58:57 MDT