Re: Cacheoff notes (was 2.5 profiling)

Joe Cooper wrote:

> Event i/o from Henrik will help--though it's not the full picture. I
> think we're doing some inefficient things in handling of
> objects...parsing too many times, moving through too many memory
> locations, allocating too many times, etc.

Certainly. Known fact and has been discussed here some times.. (last
time was not too long ago I think..)

The request path in Squid is terrible, especially on misses or aufs I/O.
Plenty of data copying and redundant operations.

Typical request:

1. Connection accepted
2. Request read
3. Request line parsed
4. Request headers area identified
5. Request headers parsed in full detail, with many allocations and
copying per header.
6. Access control etc
7. Outgoing connection opened
8. Request duplicated to make the outgoing request. Again many
allocations and copying operations
9. Request modified
10. Request packed into ascii form
11. Request sent to the origin/peer server
12. Reply heaeders received
13. Reply headers parsed
14. Reply headers processed to determine cacheability etc..
15. Reply heaers put back into ascii form and appended to the memory
object
16. Client side receives the ascii form headers
17. Client side parses the headers in full detail again.
18. Client side modifies the parsed headers
19. The headers are put back into ascii form, and sent to the client
20. Data is received from the server, and copied to the memory object
21. Client side copies the data from the memory object, and sends it to
the client
[and so on]

And data stored into the memory object are also spooled to disk if the
object is cacheable. In the case of current aufs design this involves
yet another copying in userspace, plus a rather high CPU overhead in
general..

Adrian and Robert has been working a bit on improving this. There are at
least two obvious points to attack:

1. The vasteful redundant parsing/packing of headers. Rather pass along
and clone the parsed headers.
2. The extensive copying of data. Should be addressed by storing the
data once in reference counted buffers.

> Even so, the insistence on writing everything to disk is killing our
> throughput, because writes are so expensive (think unlinking and
> writing--too many ops to be efficient). More intelligent writes will be
> a very big win (bigger than I had previously thought...again this
> workload is more accurate and pokes Squid a little differently). COSS
> should fix this right up, I think.

If COSS is done correctly it will, but I don't think it is just there
yet. Adrian?

I know the cyclic cache fs design will (cache recycling is free of
charge there, and writes are very cheap), but CCFS most likely won't see
the light in Squid-2 I think due to the radically different store
requirements.

Regards
Henrik
Received on Fri Nov 23 2001 - 18:19:37 MST