Re: Features

ZLIB is the obvious thing to use. At it's simplest, a cache item could be compressed
before storage, and decompressed for transmission.

Downsides: If in block mode: Considerable extra memory requirements (double the
object, approximately), temp space to store decompressed objects that are 'in
transit' (either coming in, ready to be compressed, or going out), CPU load.

Stream compression: This is it's own bag of worms. I've _done_ this....and in fact,
I have a library that does block/stream encryption via zlib. It was somewhat hellish
to implement, but it works. It was intended for use in HTTP, and ties to an
encryption library, but the encryption could be cut out of it. It encodes compressed
data to 6-bit printable characters (not standard base-64), but to a quickish
encoding that I cooked up at the time (we actually didn't want to be base-64
compatible when it was written).

I might look into how this might be patched into Squid....it might be useful for
inter-cache transfers.

D

Donovan Baarda wrote:

> On Mon, 24 Nov 1997, Dancer wrote:
>
> > > Compressed intercache compressions. I've made a lame hack of this for my
> > > home squid: I use ssh to setup a non-encrypted compressed pipe and use
> > > cache_host_acl to direct .html,.txt,.htm files through it.. This speeds of
> > > page loads GREATLY! This was one of the features I was working on when I
> > > was working on the cache for the Mnemonic browser. Both the gzip style and
> > > LZO type compression methods would work well here.. (Esp now with
> > > persistant connections, as the compressor will no longer get the 'hit' at
> > > the start)
> >
> > (Dancer's jaw drops). Why the hell didn't _I_ think of doing that??
> > Damnit...I've got four squid's in dire need! Can I swipe a look at your setup?
> >
>
> I've been thinking about compression and caches a bit lately.
>
> Compression can have two useful effects;
>
> 1) reduce the storage space of the object in the cache.
> 2) reduce the bandwidth of inter-cache transfers.
>
> A compressed file system gives you 1) but doesn't give you the benefit of
> reduced memory usage that truely compressed objects would. Using ssh as
> above gives you 2) but with a CPU hit. Using both gives you both but
> wastes CPU in a major way (fs has to decompress object off disk, ssh has
> to re-compress it to send it, remote ssh decompresses it, remote fs has to
> re-compress it to disk).
>
> If squid could be hacked to store and forward compressed objects, you
> would get both benefits with reduced CPU overhead. Just an idea...
>
> ABO

--
Note to evil sorcerers and mad scientists: don't ever, ever summon powerful
demons or rip holes in the fabric of space and time. It's never a good idea.
ICQ UIN: 3225440