[squid-users] Processing entire objects before callback to client-side

We are trying to find a place in the Squid2.5STABLE14 code that would allow
us to act on, and process, a complete file received by Squid prior to the
client receiving the information. We thought that turning storebuffer() on,
in the httpReadReply function in http.c, would allow the storeentry to
accumulate the received data without sending a callback to the client side.
In theory we could then process the storeentry buffer and when finished with
it, send it on to the client side as usual.

However, with storebuffer() set as above, it seems as though a buffer is
being overrun, or a socket is being trashed, as we receive only part of the
data before hanging. This seems to occur only on sites with larger objects.
The following is from our logfile and shows a transition from (comm_select:
1+0 FDS ready) to (comm_select: 0+0 FDs ready) after the 0+0 the log goes
into a huge loop of the eventRun: RUN ID process (effectively hanging).

commSetSelect: FD 16 type 1
comm_select: 1+0 FDs ready at 1149795153
comm_select: FD 16 ready for reading
httpReadReply: FD 16: len 8192.
storeAppend: appending 8192 bytes for 'B0BDE150557B85A613E2E53C272B1955'
memAppend: len 8192
httpPconnTransferDone: FD 16
httpPconnTransferDone: content_length=31463
commSetTimeout: FD 16 timeout 60
commSetSelect: FD 16 type 1
comm_select: 0+0 FDs ready at 1149795153
comm_select: 0+0 FDs ready at 1149795154
comm_select: time out: 1149795154
eventRun: RUN ID 74
eventRun: Running 'ipcache_purgelru', id 62
eventAdd: Adding 'ipcache_purgelru', in 10.000000 seconds
ipcache_purgelru: removed 0 entries
comm_select: 0+0 FDs ready at 1149795154
comm_select: time out: 1149795154
eventRun: RUN ID 75
eventRun: Running 'idnsCheckQueue', id 72
comm_select: 0+0 FDs ready at 1149795155
comm_select: time out: 1149795155
eventRun: RUN ID 76
eventRun: Running 'MaintainSwapSpace', id 73
storeMaintainSwapSpace: f=0.000000, max_scan=100, max_remove=10
cbdataFree: 00DED5C0
cbdataFree: Freeing 00DED5C0
storeUfsDirMaintain: d:/squid/var/cache removed 0/10 f=0.000 max_scan=100
eventAdd: Adding 'MaintainSwapSpace', in 1.000000 seconds
comm_select: 0+0 FDs ready at 1149795155
comm_select: 0+0 FDs ready at 1149795156
comm_select: time out: 1149795156

Is this the best place to accomplish operating on incoming objects prior to
sending them to the client side, or is there a better place to inject code
for that purpose?

Rich Maring
Received on Thu Jun 08 2006 - 15:05:03 MDT