Re: Some optimisations

Andres Kroonmaa wrote:

> For each server socket squid mallocs 64KB buffer that consumes RAM,

Actually it doesn't. It is to dumb to do this ;-)

For all HTTP connections it uses a fixed buffer of 64 KB, and then
copies the retreived data to the MemObject. The buffer size is actually
free, at the cost of the data copy.

> It seems that disk reads are related to client socket buffering, and these
> at 4096 is seeming too small. 8192 would be my suggested minimum, and
> even 16KB would be more appropriate, as this size covers most of object
> sizes (90% of web objects fit into 16KB). By upping client socket buffers
> to 16KB we potentially get larger writes, meaning less poll()ing and less
> system calls overall.

Hmm.. probably true. The buffer size for client connections is 4096
bytes. Defined as CIENT_SOCK_SZ in defines.h. As data is read from disk
into the client socket bufer data is also read in 4KB chunks.

> From these observations I decided to limit serverside buffer size to
> 16KB, increase disk buffers from 8KB to 16KB, and also increase client
> buffers from 4KB to 16KB and watch how it matters.

That will increase the memory consumption quite a lot when I/O gets
bound by the client speed. However for many requests I guess the I/O is
actually server bound.. And it will give a better disk read pattern.

> Further strangeness in squid is that when it swapouts an object, it first
> writes out metadata (storeentry+url) with whatever size is needed, and from
> then on writes in diskblocks. This means that _all_ swapouts occur in badly
> aligned manner:

This is a unfortunate effect of the "new" cache file format where a
header was added, However on most OS'es the penalty is mostly a added
syscall, as the filesystem also buffers the write before paging it out
to disk.

If you want optimal paging behaviour then you should do block aligned
writes using page aligned pages of memory. In that case some OS'es avoid
data copying by duplicating the page in the disk cache and mark is as
COW.

/Henrik
Received on Sat Sep 16 2000 - 10:04:55 MDT