Re: [squid-users] cache_dir aufs grows larger than config'ed

Amos Jeffries wrote:
> On 28/09/10 12:03, Rich Rauenzahn wrote:
>> Hi,
>>
>> Our squid servers are consistently goes over their configured disk
>> limits. I've rm'ed the cache directories and started over several
>> times... yet they slowly grow to over their set limit and fill up the
>> filesystem.
>>
>> These are du -sk's of the squid directories:
>>
>> squid2: 363520856 /squid/
>> squid3: 343399160 /squid/
>> squid4: 356593016 /squid/
>
> Okay so, about 350 GB per disk is used....
>
>>
>> These are clearly over the 300,000K limit -- and the swap stat files
>> are less than 1MB.
>
> Um, you mean the 300 GB limit configured. 307,200,000 KB to be precise.
>
>>
>> cachemgr.cgi says we are using way less:
>>
>> Store Directory Statistics:
>> Store Entries : 5698
>> Maximum Swap Size : 307200000 KB
>> Current Store Swap Size: 275852960 KB
>> Current Capacity : 90% used, 10% free
>>
>> Store Directory Statistics:
>> Store Entries : 5479
>> Maximum Swap Size : 307200000 KB
>> Current Store Swap Size: 260064224 KB
>> Current Capacity : 85% used, 15% free
>>
>> Store Directory Statistics:
>> Store Entries : 13510
>> Maximum Swap Size : 307200000 KB
>> Current Store Swap Size: 276444512 KB
>> Current Capacity : 90% used, 10% free
>>
>
> Which indicates that something other than Squid data cache is going into
> those disks. OR, the unlinkd eraser process is not working.
>
>>
>> We're storing large files, some can be several GB in size.
>>
>> Here's the config:
>>
>> http_port 80 defaultsite=foo-squid
>> http_port 8081 defaultsite=foo-squid
>> icp_port 3130
>> cache_peer foo-download parent 80 0 no-query originserver
>> name=foodownload weight=1
>> cache_peer foo-download1 parent 80 0 no-query originserver
>> name=foodownload1 round-robin weight=10 connect-fail-limit=1
>> cache_peer foo-download2 parent 80 0 no-query originserver
>> name=foodownload2 round-robin weight=10 connect-fail-limit=1
>> cache_peer foo-download3 parent 80 0 no-query originserver
>> name=foodownload3 round-robin weight=10 connect-fail-limit=1
>> cache_peer foo-maven-repo parent 8081 0 no-query originserver
>> name=foomavenrepo login=PASS
>> cache_peer foo-squid1 sibling 80 3130 proxy-only name=foosquid1
>> cache_peer foo-squid2 sibling 80 3130 proxy-only name=foosquid2
>> cache_peer foo-squid3 sibling 80 3130 proxy-only name=foosquid3
>> cache_peer foo-squid4 sibling 80 3130 proxy-only name=foosquid4
>> http_access allow all
>> acl mavenpath urlpath_regex ^/artifactory
>> acl mavenpath urlpath_regex ^/nexus
>> acl mavenport myport 8081
>> cache_peer_access foodownload deny mavenpath
>> cache_peer_access foodownload deny mavenport
>> cache_peer_access foodownload1 deny mavenpath
>> cache_peer_access foodownload1 deny mavenport
>> cache_peer_access foodownload2 deny mavenpath
>> cache_peer_access foodownload2 deny mavenport
>> cache_peer_access foodownload3 deny mavenpath
>> cache_peer_access foodownload3 deny mavenport
>> cache_peer_access foomavenrepo allow mavenpath
>> cache_peer_access foomavenrepo allow mavenport
>> cache_peer_access foomavenrepo deny all
>> cache_replacement_policy heap LFUDA
>> cache_dir aufs /squid/cache 300000 64 256
>> cache_replacement_policy heap GDSF
>> cache_mem 2 GB
>> cache_effective_user squid
>> cache_effective_group squid
>> cache_mgr alert_at_foo
>> acl QUERY urlpath_regex /components/\?
>> cache deny QUERY
>> maximum_object_size 32 GB
>> maximum_object_size_in_memory 8 MB
>> quick_abort_min 0 KB
>> quick_abort_max 0 KB
>> cache_swap_low 90
>> cache_swap_high 95

If you cache very large files, you may need to change
cache_swap_low 88
cache_swap_high 89
to force the cleanup process to be more aggressive with
removing the oldest cached files

Marcus

>> redirect_rewrites_host_header off
>> buffered_logs on
>> log_icp_queries off
>> access_log /var/log/squid/access.log
>> cache_log none
>
> You may want to locate and remove the "none" files created by the above
> logging. If they are in /squid/cache that is likely your problem.
>
> cache_log contains the critical and important administrative messages
> when things go horribly wrong. There may be something logged there that
> will make your problem clearer.
>
> Amos
Received on Wed Sep 29 2010 - 15:25:01 MDT