Re: [squid-users] TProxy Setup

Thank you Amos & Eliezer for your responses!

Amos, we have enabled debug_options 11,2, but that did not show any
HTTP request being received by Squid, not even after doing the changes
that Eliezer suggested. But they did show up, when we reverted back
to "http_port 3127 intercept" related configuration. More details
below.

Eliezer, we tried with the "ip route add local default dev lo table
100", but still same problem. I think the wiki page
http://wiki.squid-cache.org/Features/Tproxy4 needs to be updated as it
clearly says "dev eth0" and not "dev lo".

Our setup would need a bit explanation. Please bear with me while I
describe as below:

For Traffic From Host:
#Start# Host (eth0 A.B.170.10/26) -->
--> (eth2 A.B.170.1/26) Rtr1 (eth2 A.B.170.1/26) -->
--> (eth0 A.B.170.24/26) SquidBox (eth1 A.B.169.21/28) -->
--> (eth2 A.B.169.17/28) Rtr2 (eth1 BGP peered uplink) -->
--> Internet #End#

For Traffic From Internet:
#Start# Internet -->
--> (eth1 BGP peered uplink) Rtr2 (eth2 A.B.169.17/28) -->
--> (eth1 A.B.169.21/28) SquidBox (eth0 A.B.170.24/28) -->
--> (eth0 A.B.170.10/26) Host #End#
* In my understanding, this should not pass through Rtr1 as as
SquidBox eth0 is in the same subnet as Host.

Both Rtr1 & Rtr2 are Linux based router called Mikrotik, installed on
x86 hardware.
Rtr1 has the following rules:
/ip firewall mangle add action=mark-routing chain=prerouting
disabled=no dst-port=80 new-routing-mark=_to_squid_ passthrough=yes
protocol=tcp src-address=A.B.170.10
/ip route add disabled=no distance=1 dst-address=0.0.0.0/0
gateway=A.B.170.24 routing-mark=_to_squid_ scope=30 target-scope=10

Rtr2 has the following rules:
/ip firewall mangle add action=mark-routing chain=prerouting
disabled=no dst-address=A.B.170.10 new-routing-mark=_to_squid_
passthrough=yes protocol=tcp src-port=80
/ip route add disabled=no distance=1 dst-address=0.0.0.0/0
gateway=A.B.169.21 routing-mark=_to_squid_ scope=30 target-scope=10

The policy routing rules are the same on Rtr1 when we use the REDIRECT
rule in "iptables -t nat" for "http_port 3127 intercept", and in that
instance SquidBox works like a charm. All the HTTP request are now
showing up in "cache.log" because of "debug_options 11,2" as Amos
suggested.

However, whenever we remove the nat rules and introduce the mangle
rules + ip rule + ip route in table 100 for "http_port 3129 tproxy",
Rtr1 shows that the packets are marked and forwarded to SquidBox.
Even SquidBox properly logs the packets in "/var/log/messages" due to
the mangle table rule, but Squid process on SquidBox does not seem to
be receiving the packets. No HTTP request entry showing up in
"cache.log".

IPTables -L for mangle show the following:
[root_at_proxy01 ~]# iptables -vxnL --line-numbers -t mangle
Chain PREROUTING (policy ACCEPT 235 packets, 29632 bytes)
num pkts bytes target prot opt in out source
          destination
1 0 0 ACCEPT all -- * * 0.0.0.0/0
        A.B.169.21
2 6174 821596 ACCEPT all -- * * 0.0.0.0/0
        A.B.170.24
3 1005 51367 ACCEPT all -- * * 0.0.0.0/0
        A.B.174.0/24
4 0 0 ACCEPT all -- * * 0.0.0.0/0
        M.N.0.66
5 49 3420 DIVERT tcp -- * * 0.0.0.0/0
        0.0.0.0/0 socket
6 52 3840 LOG tcp -- * * 0.0.0.0/0
        0.0.0.0/0 tcp dpt:80 LOG flags 0 level 4 prefix
`TProxy: '
7 52 3840 TPROXY tcp -- * * 0.0.0.0/0
        0.0.0.0/0 tcp dpt:80 TPROXY redirect 0.0.0.0:3129
mark 0x1/0x1

The IP rule & route lists, and rt_tables & rp_filter show:

[root_at_proxy01 ~]# ip route list table squidtproxy
local default dev lo scope host

[root_at_proxy01 ~]# ip rule list
0: from all lookup local
32765: from all fwmark 0x1 lookup squidtproxy
32766: from all lookup main
32767: from all lookup default

[root_at_proxy01 ~]# cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
100 squidtproxy

[root_at_proxy01 ~]# find /proc/sys/net/ipv4/ -iname rp_filter
/proc/sys/net/ipv4/conf/all/rp_filter
/proc/sys/net/ipv4/conf/default/rp_filter
/proc/sys/net/ipv4/conf/lo/rp_filter
/proc/sys/net/ipv4/conf/eth0/rp_filter
/proc/sys/net/ipv4/conf/eth1/rp_filter
/proc/sys/net/ipv4/conf/gre0/rp_filter
/proc/sys/net/ipv4/conf/gretap0/rp_filter

[root_at_proxy01 ~]# find /proc/sys/net/ipv4/ -iname rp_filter -exec cat {} +
0
0
0
0
0
0
0

Amos, we also looked into the "routing loop" that you mentioned.
Since there are two routers involved, Rtr1 & Rtr2, with Squid
connected to both of them, we used the rules above, so, Rtr1 only
policy-routes Host -> Squid and Rtr2 policy-routes Internet -> Squid.
This should be correct, no? In the very least, Squid should be
receiving the packets, and the HTTP request headers should show up in
"cache.log", shouldn't they?

We apologize for the rather long email. Hopefully, you could
understand what our setup looks like, and help us in running Squid + Tproxy.

With regards
HASSAN

On Thu, Jul 3, 2014 at 7:58 AM, Eliezer Croitoru <eliezer_at_ngtech.co.il> wrote:
> Hey There,
>
> You have seem to use the wrong rules in ip route and maybe something else.
> I need more for the picture to understand what and how you implemented it.
> What I need is the IP and wires topology.
> Wccp is not good for you(maybe) but the examples are perfect from any
> aspect.
> Take a peek at:
> http://wiki.squid-cache.org/ConfigExamples/UbuntuTproxy4Wccp2
>
> You can take the relevant rules from the article to correct yours.
> basically what you need is:
> #!/usr/bin/bash
>
> echo "Loading modules.."
> modprobe -a nf_tproxy_core xt_TPROXY xt_socket xt_mark ip_gre gre
>
> LOCALIP="10.80.2.2"
>
> echo "changing routing and reverse path stuff.."
> for i in /proc/sys/net/ipv4/conf/*/rp_filter
> do
> echo 0 > $i
> done
> echo 1 > /proc/sys/net/ipv4/ip_forward
>
> echo "creating routing table for tproxy..."
>
> ip rule add fwmark 1 lookup 100
> ip route add local 0.0.0.0/0 dev lo table 100
>
> echo "creating iptables tproxy rules..."
> iptables -A INPUT -i lo -j ACCEPT
> iptables -A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
> iptables -A FORWARD -i lo -j ACCEPT
>
> iptables -t mangle -F
> iptables -t mangle -A PREROUTING -d $LOCALIP -j ACCEPT
> iptables -t mangle -N DIVERT
> iptables -t mangle -A DIVERT -j MARK --set-mark 1
> iptables -t mangle -A DIVERT -j ACCEPT
> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark
> 0x1/0x1 --on-port 3129
> ##END OF FILE
>
> The route towards the lo is important to enable the tproxy action.
> In your settings I have seen that you have used something else which will
> probably cause some strange issues.
>
> All The Bests,
> Eliezer
>
> On 07/03/2014 03:01 AM, Nyamul Hassan wrote:
>>
>> Hi,
>>
>> We are trying to run Squid 3.4.6 with TProxy. Earlier we used to run
>> Squid 2.7.Stable9 in "transparent" mode with a DNAT rule on the router
>> box to redirect traffic. This being our first jibe at Squid3, we have
>> successfully configured "intercept" mode with the router doing a
>> policy-based routing (instead of DNAT). All works quite well!
>>
>> However, when we try to do a TProxy configuration, Squid does not seem
>> to be seeing the traffic at all. Since Squid3 is working in
>> "intercept" we assume that is not the problem. IPTables is configured
>> as follows:
>>
>> *mangle
>> :PREROUTING ACCEPT [0:0]
>> :INPUT ACCEPT [0:0]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [0:0]
>> :POSTROUTING ACCEPT [0:0]
>> :DIVERT - [0:0]
>> -A PREROUTING -p tcp -m socket -j DIVERT
>> -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "TProxy: "
>> -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port
>> 3129
>> -A DIVERT -j MARK --set-mark 1
>> -A DIVERT -j ACCEPT
>> COMMIT
>>
>> The Log option shows similar lines as follows (our IP omitted below):
>> Jul 3 05:15:24 proxy01 kernel: TProxy: IN=eth0 OUT=
>> MAC=00:22:4d:a7:9a:8c:00:15:17:c8:a0:39:08:00 SRC=<test>
>> DST=195.93.85.193 LEN=52 TOS=0x00 PREC=0x00 TTL=1 ID=25176 DF
>> PROTO=TCP SPT=3264 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
>>
>> We also tried both with and without the "IP" commands:
>>
>> ip rule add fwmark 1 lookup 100
>> ip route add local default dev eth0 table 100
>>
>> We have searched through Google, mailing lists, Squid Docs, but seems
>> like we are still missing through something. One thing though, a lot
>> of the TProxy examples accompany WCCP or Bridge. Are either of them
>> mandatory in TProxy setup? If not, could someone help us where we are
>> doing things wrong?
>>
>> Thanks in advance for youguidance.
>>
>> Regards
>> HASSAN
>>
>
Received on Thu Jul 03 2014 - 15:04:40 MDT