Re: [squid-users] Re: is there any thing wrong from cache manager logs ?!!

OK another small test:
##START
ls /proc/sys/net/netfilter/nf_conntrack_* |tee
ls /proc/sys/net/netfilter/nf_conntrack_* |xargs -n1 -l1 cat
ls /proc/net/nf_conntrack
##END

If the "/proc/net/nf_conntrack" file do exits then run these:
##START
cat /proc/net/nf_conntrack |wc -l
cat /proc/net/nf_conntrack |grep UDP |wc -l
cat /proc/net/nf_conntrack |grep ESTABLISHED |wc -l
cat /proc/net/nf_conntrack |grep TIME_WAIT |wc -l
cat /proc/net/nf_conntrack |grep CLOSE_WAIT |wc -l
cat /proc/net/nf_conntrack |grep LISTEN |wc -l
cat /proc/net/nf_conntrack |grep ":53" |wc -l
cat /proc/net/nf_conntrack |grep ":TPROXYPORT" |wc -l
##END

Which should show us couple statistics about the issue.

also if you want to take a snapshot of "top" you can try to start with:
top -n 1 -b

which will get lots of out put that you will need to consult the "man
top" first so you would get more info on how to clear un-needed info
from the snapshot.

Eliezer

On 11/08/2013 01:21 AM, Dr.x wrote:
> Eliezer Croitoru-2 wrote
>> OK so after Amos did the calculations (Thanks) I assume that using lsof
>> will give us more clue about it.
>> The first thing to do is to start a ssh session into the server with the
>> setting:
>> ServerAliveInterval 30
>> Added into the /etc/ssh/ssh_config (on the client side)
>>
>> When you do have this session running you wont have any troubles running
>> top or any other basic tests on the server while there is a degradation.
>>
>> Now this command is what you will need in order to make the suspicious
>> more accurate and maybe lead to something that can help you:
>> "lsof -u squid -a -i 4 -n -P"
>> (squid is the default username on centos for the proxy user)
>>
>> Dont try to run this command just like this out of the blue since the
>> output can be more then 60k lines long..
>>
>> You should try to throw it into a file at the tmp dir so this:
>> "lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1"
>> Should be safe.
>> The next thing is to find out how many FD are in sum and how many are
>> ESTABLISHED etc.. so run these:
>> ##START
>> lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1
>> cat /tmp/tmp_lsof.1 |wc -l
>> cat /tmp/tmp_lsof.1 |grep UDP |wc -l
>> cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
>> cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
>> cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
>> cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
>> cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
>> cat /tmp/tmp_lsof.1 |grep ":TPROXYPORT" |wc -l
>> ##END
>> (TPROXYPORT IS the port from squid.conf)
>>
>> Once you have all the above results before the degradation in it and
>> after we might have a clue about the source of the problem and whether
>> it comes from too much FD which are not being used but causing the
>> system to loop throw lots of them.
>
> hi , 1st of all , does i face a loop ??
>
> anyway , ive made another test
> before the degredation , 1st 5 minutes of squid
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |wc -l
> 3398
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
> 6
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
> 3134
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
> 0
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
> 1
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
> 8
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
> 73
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
> 5
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]#
> [root_at_squid ~]# lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1
> cat /tmp/tmp_lsof.1 |wc -l
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |wc -l
> 3366
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
> 6
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
> 3118
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
> 0
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
> 2
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
> 8
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
> 82
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
> 5
> ==================================================
>
> after degredation and system close to dead !
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |wc -l
> 3505
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
> 6
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
> 3262
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
> 0
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
> 5
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
> 8
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
> 57
> [root_at_squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
> 4
>
>
>
>
> -----
> Dr.x
> --
> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/is-there-any-thing-wrong-from-cache-manager-logs-tp4663156p4663180.html
> Sent from the Squid - Users mailing list archive at Nabble.com.
>
Received on Thu Nov 07 2013 - 23:52:35 MST