RE: [squid-users] Proxy/no proxy GET results differ

Amos,

Thank you for your help. You're right. Forwarded_for is off in the proxy configuration. When I enable forwarded_for, the remote server does not return that error message. Problem solved.

Regards,

Dan

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Wednesday, April 29, 2009 2:02 AM
To: Dan Sopher
Cc: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Proxy/no proxy GET results differ

Dan Sopher wrote:
> Thank you for the reply. As requested, below are 2 captured TCP streams. The first 3 sections is 1 TCP stream with the proxy in the data path, and captured from each interface of client<->proxy<->remote_server (although can't capture from the remote server interface). The 4th section is the TCP stream without the proxy in the data path.
>
> Does it appear that the remote server is receiving a request in a format that it cannot process, and is returning the "# Server Error" page to the proxy?
>

Not really. Apache is normally capable of doing these requests easily.

The only non-standard thing about the request going back is the
butchered "X-Forwarded-For: unknown" which is unfortunately fairly
common, so I doubt its a problem.

It's very weird to get a "200 OK" server error. But they would know more
about that than any of us.

Amos

>
>
>
> ===============================================================
> PACKET CAPTURE FROM CLIENT WITH SQUID IN DATA PATH (STREAM 1):
>
> GET http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.1
> TE: deflate,gzip;q=0.3
> Connection: TE, close
> Host: mirrors.rpmfusion.org
> User-Agent: lwp-request/2.07
>
> HTTP/1.0 200 OK
> Date: Tue, 28 Apr 2009 17:44:45 GMT
> Server: Apache/2.2.11 (Fedora)
> Content-Length: 15
> Cache-Control: no-cache
> Content-Type: text/plain
> X-Cache: MISS from la-squid.twistbox.com
> X-Cache-Lookup: MISS from la-squid.twistbox.com:3128
> Via: 1.0 la-squid.twistbox.com (squid)
> Proxy-Connection: close
>
> # Server Error
>
> ===============================================================
> PACKET CAPTURE FROM CLIENT-SIDE TCP PORT OF THE PROXY(STREAM 1):
>
> GET http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.1
> TE: deflate,gzip;q=0.3
> Connection: TE, close
> Host: mirrors.rpmfusion.org
> User-Agent: lwp-request/2.07
>
> HTTP/1.0 200 OK
> Date: Tue, 28 Apr 2009 17:44:45 GMT
> Server: Apache/2.2.11 (Fedora)
> Content-Length: 15
> Cache-Control: no-cache
> Content-Type: text/plain
> X-Cache: MISS from la-squid.twistbox.com
> X-Cache-Lookup: MISS from la-squid.twistbox.com:3128
> Via: 1.0 la-squid.twistbox.com (squid)
> Proxy-Connection: close
>
> # Server Error
>
> ===============================================================
> PACKET CAPTURE FROM EXTERNAL-SIDE TCP PORT OF THE PROXY(STREAM 1):
>
> GET /mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.0
> Host: mirrors.rpmfusion.org
> User-Agent: lwp-request/2.07
> Via: 1.1 la-squid.twistbox.com (squid)
> X-Forwarded-For: unknown
> Cache-Control: max-age=259200
> Connection: keep-alive
>
> HTTP/1.1 200 OK
> Date: Tue, 28 Apr 2009 17:44:45 GMT
> Server: Apache/2.2.11 (Fedora)
> Content-Length: 15
> cache-control: no-cache
> Keep-Alive: timeout=2, max=100
> Connection: Keep-Alive
> Content-Type: text/plain
>
> # Server Error
>
> ===============================================================
> PACKET CAPTURE FROM CLIENT _WITHOUT_ THE PROXY IN DATA PATH(STREAM 2):
> GET /mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.1
> TE: deflate,gzip;q=0.3
> Connection: TE, close
> Host: mirrors.rpmfusion.org
> User-Agent: lwp-request/2.07
>
> HTTP/1.1 200 OK
> Date: Tue, 28 Apr 2009 17:44:49 GMT
> Server: Apache/2.2.11 (Fedora)
> Content-Length: 404
> cache-control: no-cache
> Connection: close
> Content-Type: text/plain
>
> # repo = free-fedora-updates-released-9 arch = i386 country = US
> http://mirror.liberty.edu/pub/rpmfusion/free/fedora/updates/9/i386
> http://astromirror.uchicago.edu/rpmfusion/free/fedora/updates/9/i386
> http://mirrors.cat.pdx.edu/rpmfusion/free/fedora/updates/9/i386
> http://mirror.web-ster.com/rpmfusion/free/fedora/updates/9/i386
> http://mirrors.tummy.com/mirrors/rpmfusion.org/free/fedora/updates/9/i386
>
>
>
>
>
>
> -----Original Message-----
> From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
> Sent: Monday, April 27, 2009 7:54 PM
> To: Dan Sopher
> Cc: squid-users_at_squid-cache.org
> Subject: Re: [squid-users] Proxy/no proxy GET results differ
>
>> Hello.
>>
>> When running the following GET commands from a shell(same results from
>> browsers), the results are different, with the error occurring when
>> squid(Squid Cache: Version 3.0.STABLE13) is in the data path. Any help is
>> appreciated. Thank you in advance.
>
> You appear to be confused by proper HTTP behavior. The web server responds
> in whatever way it needs to for the requests that comes in.
>
> Squid is not capable of HTTP/1.1 and so sends HTTP/1.0 requests as it is
> required to do.
>
> The only possible thing like a bug I can see here is that squid _might_ be
> transforming a TE: encoding request into a Content-Encoding: and the
> client unable to handle such encodings despite needing to. We need the
> headers input into squid from the client vs the headers sent to Server to
> see.
>
> Amos
>
>>
>> Examples and packet capture data is below:
>>
>>
>> The following are the results WITHOUT squid in the data path:
>>
>> root_at_host ~# GET
>> "http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-9&arch=i386"
>>
>> # repo = free-fedora-updates-released-9 arch = i386 country = US
>> http://astromirror.uchicago.edu/rpmfusion/free/fedora/updates/9/i386
>> http://mirror.liberty.edu/pub/rpmfusion/free/fedora/updates/9/i386
>> http://mirror.web-ster.com/rpmfusion/free/fedora/updates/9/i386
>> http://mirrors.cat.pdx.edu/rpmfusion/free/fedora/updates/9/i386
>> http://mirrors.tummy.com/mirrors/rpmfusion.org/free/fedora/updates/9/i386
>>
>>
>>
>> and, the following result is from using squid in the data path:
>>
>> root@host ~# GET -p http://192.168.0.22:3128
>> "http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-9&arch=i386"
>> # Server Error
>>
>>
>>
>> PACKET CAPTURE DATA:
>> ===================
>> 1. This is the result without using Squid proxy. Traffic is captured from
>> the host using this configuration:
>>
>> Data path: Host -> Juniper Netscreen firewall -> Router -> Internet
>>
>>
>> GET /mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.1
>> TE: deflate,gzip;q=0.3
>> Connection: TE, close
>> Host: mirrors.rpmfusion.org
>> User-Agent: lwp-request/2.07
>>
>> HTTP/1.1 200 OK
>> Date: Fri, 24 Apr 2009 20:53:22 GMT
>> Server: Apache/2.2.11 (Fedora)
>> Content-Length: 474
>> cache-control: no-cache
>> Connection: close
>> Content-Type: text/plain
>>
>> # repo = free-fedora-updates-released-9 arch = i386 country = US
>> http://astromirror.uchicago.edu/rpmfusion/free/fedora/updates/9/i386
>> http://mirror.liberty.edu/pub/rpmfusion/free/fedora/updates/9/i386
>> http://mirror.web-ster.com/rpmfusion/free/fedora/updates/9/i386
>> http://mirrors.cat.pdx.edu/rpmfusion/free/fedora/updates/9/i386
>> http://lordmorgul.net/pub/fedora/rpmfusion/free/fedora/updates/9/i386
>> http://mirrors.tummy.com/mirrors/rpmfusion.org/free/fedora/updates/9/i386
>>
>>
>>
>>
>> 2. This is the result using Squid proxy. Traffic captured from the
>> external interface of the proxy:
>>
>> Data path: Host -> Squid proxy -> Juniper Netscreen firewall -> Router ->
>> Internet
>>
>>
>> GET /mirrorlist?repo=free-fedora-updates-released-9&arch=i386 HTTP/1.0
>> Host: mirrors.rpmfusion.org
>> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.9)
>> Gecko/2009040821 Firefox/3.0.9 (.NET CLR 3.5.30729)
>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>> Accept-Language: en-us,en;q=0.5
>> Accept-Encoding: gzip,deflate
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>> Keep-Alive: 300
>> Via: 1.0 la-squid.twistbox.com (squid)
>> X-Forwarded-For: unknown
>> Cache-Control: max-age=259200
>> Connection: keep-alive
>>
>> HTTP/1.1 200 OK
>> Date: Fri, 24 Apr 2009 20:54:51 GMT
>> Server: Apache/2.2.9 (Fedora)
>> Content-Length: 35
>> content-encoding: gzip
>> cache-control: no-cache
>> Connection: close
>> Content-Type: text/plain; charset=UTF-8
>>
>> .....'.I..SV.N-*K-Rp-*./....$}N....
>>
>>
>>
>> CONFIGURATION:
>> =============
>> Squid Cache: Version 3.0.STABLE13
>>
>> configure options: '--build=i386-redhat-linux-gnu'
>> '--host=i386-redhat-linux-gn
>> u' '--target=i386-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr'
>> '--exec-
>> prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
>> '--in
>> cludedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec'
>> '--shared
>> statedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info'
>> '--exec
>> _prefix=/usr' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid'
>> '--localstatedi
>> r=/var' '--datadir=/usr/share' '--sysconfdir=/etc/squid'
>> '--disable-dependency-t
>> racking' '--enable-arp-acl' '--enable-auth=basic,digest,ntlm'
>> '--enable-basic-au
>> th-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
>> '--enable-
>> cache-digests' '--enable-cachemgr-hostname=localhost'
>> '--enable-delay-pools' '--
>> enable-digest-auth-helpers=password' '--enable-epoll'
>> '--enable-external-acl-hel
>> pers=ip_user,ldap_group,unix_group,wbinfo_group' '--enable-icap-client'
>> '--enabl
>> e-ident-lookups' '--with-large-files' '--enable-linux-netfilter'
>> '--enable-ntlm-
>> auth-helpers=SMB,fakeauth' '--enable-referer-log'
>> '--enable-removal-policies=hea
>> p,lru' '--enable-snmp' '--enable-ssl'
>> '--enable-storeio=aufs,diskd,null,ufs' '--
>> enable-useragent-log' '--enable-wccpv2' '--with-aio'
>> '--with-default-user=squid'
>> '--with-filedescriptors=16384' '--with-dl' '--with-openssl=/usr/kerberos'
>> '--wi
>> th-pthreads' 'build_alias=i386-redhat-linux-gnu'
>> 'host_alias=i386-redhat-linux-g
>> nu' 'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
>> -fsigned-cha
>> r -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>> -fstack-protector --pa
>> ram=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
>> -fasynchronous-unwind-tabl
>> es' 'LDFLAGS=-pie' 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall
>> -Wp,-D_FORTIFY_SOURCE=2 -f
>> exceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
>> -mtune=g
>> eneric -fasynchronous-unwind-tables' 'FFLAGS=-O2 -g -pipe -Wall
>> -Wp,-D_FORTIFY_S
>> OURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32
>> -march=i38
>> 6 -mtune=generic -fasynchronous-unwind-tables'
>>
>>
>>
>>
>>
>>
>
>

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
   Current Beta Squid 3.1.0.7