Re: [squid-users] TProxy Setup

Just some quick answers to your questions inline below. (I've not had
time to consider this in detail sorry.)

On 2014-07-04 03:03, Nyamul Hassan wrote:
> Thank you Amos & Eliezer for your responses!
>
> Amos, we have enabled debug_options 11,2, but that did not show any
> HTTP request being received by Squid, not even after doing the changes
> that Eliezer suggested. But they did show up, when we reverted back
> to "http_port 3127 intercept" related configuration. More details
> below.

That is the problem then. Something is blocking the traffic arriving at
Squid listening port. selinux, rp_filter or ip_forward sysctl settings I
usually find are the problem for this, although there have been a few
cases where nobody could figure out why this was happening.

>
> Eliezer, we tried with the "ip route add local default dev lo table
> 100", but still same problem. I think the wiki page
> http://wiki.squid-cache.org/Features/Tproxy4 needs to be updated as it
> clearly says "dev eth0" and not "dev lo".

see the /!\ notes under in the wiki page under the section about setting
up the route table.

The interface(s) to attach the table to is the one receiving the
packets. From your description I suspect you will have two interfaces -
one for each of Rtr1 and Rtr2.

For debugging try setting it for each interfaces receiving traffic and
see if TPROXY starts working.

>
> Our setup would need a bit explanation. Please bear with me while I
> describe as below:
>
> For Traffic From Host:
> #Start# Host (eth0 A.B.170.10/26) -->
> --> (eth2 A.B.170.1/26) Rtr1 (eth2 A.B.170.1/26) -->
> --> (eth0 A.B.170.24/26) SquidBox (eth1 A.B.169.21/28) -->
> --> (eth2 A.B.169.17/28) Rtr2 (eth1 BGP peered uplink) -->
> --> Internet #End#
>
> For Traffic From Internet:
> #Start# Internet -->
> --> (eth1 BGP peered uplink) Rtr2 (eth2 A.B.169.17/28) -->
> --> (eth1 A.B.169.21/28) SquidBox (eth0 A.B.170.24/28) -->
> --> (eth0 A.B.170.10/26) Host #End#
> * In my understanding, this should not pass through Rtr1 as as
> SquidBox eth0 is in the same subnet as Host.

All HTTP traffic should pass through SquidBox, and in fact through Squid
itself. The TCP layer ports and packet serial numbers do not match what
the client is aware of, so any traffic accidentally reaching it without
going through Squid will be dropped.

Put this off to later though. Right now the packets are not even getting
into Squid.

>
> Both Rtr1 & Rtr2 are Linux based router called Mikrotik, installed on
> x86 hardware.
> Rtr1 has the following rules:
> /ip firewall mangle add action=mark-routing chain=prerouting
> disabled=no dst-port=80 new-routing-mark=_to_squid_ passthrough=yes
> protocol=tcp src-address=A.B.170.10
> /ip route add disabled=no distance=1 dst-address=0.0.0.0/0
> gateway=A.B.170.24 routing-mark=_to_squid_ scope=30 target-scope=10
>
> Rtr2 has the following rules:
> /ip firewall mangle add action=mark-routing chain=prerouting
> disabled=no dst-address=A.B.170.10 new-routing-mark=_to_squid_
> passthrough=yes protocol=tcp src-port=80
> /ip route add disabled=no distance=1 dst-address=0.0.0.0/0
> gateway=A.B.169.21 routing-mark=_to_squid_ scope=30 target-scope=10
>
> The policy routing rules are the same on Rtr1 when we use the REDIRECT
> rule in "iptables -t nat" for "http_port 3127 intercept", and in that
> instance SquidBox works like a charm. All the HTTP request are now
> showing up in "cache.log" because of "debug_options 11,2" as Amos
> suggested.

Great. Thank you for these details. I am creating a Microtik wiki page
based on them.

>
> However, whenever we remove the nat rules and introduce the mangle
> rules + ip rule + ip route in table 100 for "http_port 3129 tproxy",
> Rtr1 shows that the packets are marked and forwarded to SquidBox.
> Even SquidBox properly logs the packets in "/var/log/messages" due to
> the mangle table rule, but Squid process on SquidBox does not seem to
> be receiving the packets. No HTTP request entry showing up in
> "cache.log".
>
> IPTables -L for mangle show the following:
> [root_at_proxy01 ~]# iptables -vxnL --line-numbers -t mangle
> Chain PREROUTING (policy ACCEPT 235 packets, 29632 bytes)
> num pkts bytes target prot opt in out source
> destination
> 1 0 0 ACCEPT all -- * * 0.0.0.0/0
> A.B.169.21
> 2 6174 821596 ACCEPT all -- * * 0.0.0.0/0
> A.B.170.24
> 3 1005 51367 ACCEPT all -- * * 0.0.0.0/0
> A.B.174.0/24
> 4 0 0 ACCEPT all -- * * 0.0.0.0/0
> M.N.0.66
> 5 49 3420 DIVERT tcp -- * * 0.0.0.0/0
> 0.0.0.0/0 socket
> 6 52 3840 LOG tcp -- * * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:80 LOG flags 0 level 4 prefix
> `TProxy: '
> 7 52 3840 TPROXY tcp -- * * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:80 TPROXY redirect 0.0.0.0:3129
> mark 0x1/0x1
>
> The IP rule & route lists, and rt_tables & rp_filter show:
>
> [root_at_proxy01 ~]# ip route list table squidtproxy
> local default dev lo scope host
>
> [root_at_proxy01 ~]# ip rule list
> 0: from all lookup local
> 32765: from all fwmark 0x1 lookup squidtproxy
> 32766: from all lookup main
> 32767: from all lookup default
>
> [root_at_proxy01 ~]# cat /etc/iproute2/rt_tables
> #
> # reserved values
> #
> 255 local
> 254 main
> 253 default
> 0 unspec
> #
> # local
> #
> #1 inr.ruhep
> 100 squidtproxy
>
> [root_at_proxy01 ~]# find /proc/sys/net/ipv4/ -iname rp_filter
> /proc/sys/net/ipv4/conf/all/rp_filter
> /proc/sys/net/ipv4/conf/default/rp_filter
> /proc/sys/net/ipv4/conf/lo/rp_filter
> /proc/sys/net/ipv4/conf/eth0/rp_filter
> /proc/sys/net/ipv4/conf/eth1/rp_filter
> /proc/sys/net/ipv4/conf/gre0/rp_filter
> /proc/sys/net/ipv4/conf/gretap0/rp_filter
>
> [root_at_proxy01 ~]# find /proc/sys/net/ipv4/ -iname rp_filter -exec cat
> {} +
> 0
> 0
> 0
> 0
> 0
> 0
> 0
>
> Amos, we also looked into the "routing loop" that you mentioned.
> Since there are two routers involved, Rtr1 & Rtr2, with Squid
> connected to both of them, we used the rules above, so, Rtr1 only
> policy-routes Host -> Squid and Rtr2 policy-routes Internet -> Squid.
> This should be correct, no? In the very least, Squid should be
> receiving the packets, and the HTTP request headers should show up in
> "cache.log", shouldn't they?
>

That should be correct. And yes cache.log should at least be showing the
original client request. Even if the traffic were looping.

> We apologize for the rather long email. Hopefully, you could
> understand what our setup looks like, and help us in running Squid +
> Tproxy.
>
> With regards
> HASSAN
>
> On Thu, Jul 3, 2014 at 7:58 AM, Eliezer Croitoru <eliezer_at_ngtech.co.il>
> wrote:
>> Hey There,
>>
>> You have seem to use the wrong rules in ip route and maybe something
>> else.
>> I need more for the picture to understand what and how you implemented
>> it.
>> What I need is the IP and wires topology.
>> Wccp is not good for you(maybe) but the examples are perfect from any
>> aspect.
>> Take a peek at:
>> http://wiki.squid-cache.org/ConfigExamples/UbuntuTproxy4Wccp2
>>
>> You can take the relevant rules from the article to correct yours.
>> basically what you need is:
>> #!/usr/bin/bash
>>
>> echo "Loading modules.."
>> modprobe -a nf_tproxy_core xt_TPROXY xt_socket xt_mark ip_gre gre
>>
>> LOCALIP="10.80.2.2"
>>
>> echo "changing routing and reverse path stuff.."
>> for i in /proc/sys/net/ipv4/conf/*/rp_filter
>> do
>> echo 0 > $i
>> done
>> echo 1 > /proc/sys/net/ipv4/ip_forward
>>
>> echo "creating routing table for tproxy..."
>>
>> ip rule add fwmark 1 lookup 100
>> ip route add local 0.0.0.0/0 dev lo table 100
>>
>> echo "creating iptables tproxy rules..."
>> iptables -A INPUT -i lo -j ACCEPT
>> iptables -A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
>> iptables -A FORWARD -i lo -j ACCEPT
>>
>> iptables -t mangle -F
>> iptables -t mangle -A PREROUTING -d $LOCALIP -j ACCEPT
>> iptables -t mangle -N DIVERT
>> iptables -t mangle -A DIVERT -j MARK --set-mark 1
>> iptables -t mangle -A DIVERT -j ACCEPT
>> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
>> --tproxy-mark
>> 0x1/0x1 --on-port 3129
>> ##END OF FILE
>>
>> The route towards the lo is important to enable the tproxy action.
>> In your settings I have seen that you have used something else which
>> will
>> probably cause some strange issues.
>>
>> All The Bests,
>> Eliezer
>>
>> On 07/03/2014 03:01 AM, Nyamul Hassan wrote:
>>>
>>> Hi,
>>>
>>> We are trying to run Squid 3.4.6 with TProxy. Earlier we used to run
>>> Squid 2.7.Stable9 in "transparent" mode with a DNAT rule on the
>>> router
>>> box to redirect traffic. This being our first jibe at Squid3, we
>>> have
>>> successfully configured "intercept" mode with the router doing a
>>> policy-based routing (instead of DNAT). All works quite well!
>>>
>>> However, when we try to do a TProxy configuration, Squid does not
>>> seem
>>> to be seeing the traffic at all. Since Squid3 is working in
>>> "intercept" we assume that is not the problem. IPTables is
>>> configured
>>> as follows:
>>>
>>> *mangle
>>> :PREROUTING ACCEPT [0:0]
>>> :INPUT ACCEPT [0:0]
>>> :FORWARD ACCEPT [0:0]
>>> :OUTPUT ACCEPT [0:0]
>>> :POSTROUTING ACCEPT [0:0]
>>> :DIVERT - [0:0]
>>> -A PREROUTING -p tcp -m socket -j DIVERT
>>> -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "TProxy: "
>>> -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1
>>> --on-port
>>> 3129
>>> -A DIVERT -j MARK --set-mark 1
>>> -A DIVERT -j ACCEPT
>>> COMMIT
>>>
>>> The Log option shows similar lines as follows (our IP omitted below):
>>> Jul 3 05:15:24 proxy01 kernel: TProxy: IN=eth0 OUT=
>>> MAC=00:22:4d:a7:9a:8c:00:15:17:c8:a0:39:08:00 SRC=<test>
>>> DST=195.93.85.193 LEN=52 TOS=0x00 PREC=0x00 TTL=1 ID=25176 DF
>>> PROTO=TCP SPT=3264 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
>>>
>>> We also tried both with and without the "IP" commands:
>>>
>>> ip rule add fwmark 1 lookup 100
>>> ip route add local default dev eth0 table 100
>>>
>>> We have searched through Google, mailing lists, Squid Docs, but seems
>>> like we are still missing through something. One thing though, a lot
>>> of the TProxy examples accompany WCCP or Bridge. Are either of them
>>> mandatory in TProxy setup? If not, could someone help us where we
>>> are
>>> doing things wrong?
>>>
>>> Thanks in advance for youguidance.
>>>
>>> Regards
>>> HASSAN
>>>
>>
Received on Fri Jul 04 2014 - 15:00:02 MDT