huge difference using wanproxy throught internet

[Juli Mallett]

On Fri, Aug 16, 2013 at 2:25 PM, Daniel Coletti <dcoletti at xtech.com.ar>wrote:

> 2013/8/16 Juli Mallett <juli at clockworksquid.com>
>
>>  Hi Daniel,
>>
>> With a lower-latency link like a WAN rather than the Internet, I would
>> expect to see more packets sent between WANProxy instances; WANProxy makes
>> no effort to buffer up data before sending it.  Using a slower link, you'll
>> naturally have more buffering occurring and fewer packets sent.  There's
>> some things that we could do and should do to avoid sending out gratuitous
>> small packets, and I have some changes to do that, but I haven't tested
>> them enough to check them in yet.
>>
>> I'm not sure if I understand the difference you're seeing in terms of
>> data reduction / deduplication / compression.  Which is giving the greater
>> savings in terms of data transferred, the LAN or the Internet?
>>
>
> The LAN, definitely. the savings are enormous. In a 168Mb file transfer
> the first time there's 172Mb of traffic between WANProxy instances, the
> second time it doesn't even reach 2Mb. of data transferring.
> Transfer time follows a similar pattern, 14 seconds the first download,
> 4.2 seconds the second one.
>
>
>>  And what kind of download time are you seeing to the client on each
>> configuration?
>>
>
> The transfer time evaluation is tricky using Internet because the link
> speed is not constant, well the link is but the links between the routers I
> jump to reach to the ending point might not. That's why I took a closer
> look in terms of data/pkts transfer.
> Nevertheless the transfer time difference between first and second file
> transfer is not substantial.
>
> I dug a bit deeper and I found that what seems to be causing the problem
> is not the Internet o MTU, but http-proxy I use behind the WANProxy
> "server".
> If I change the setup from this:
> WPclient->(gateway)->//internet//->WPserver+squid->//internet//->webserver
> to this:
> WPclient->(gateway)->//internet//->WPserver->webserver
>
> and download the very same file, it worked like a charm. the second time
> the file downloaded a LOT faster.
> This time instead of being at some server on the internet I put it in a
> local webserver that lives in the same server as the ending point of
> WANProxy.
> But I can't figure out why this is causing problems, and huge ones.
>
> downloading the same file twice using the first setup takes 10 to 20% less
> than the first time, when using the second setup the file is downloaded 60
> to 70% faster at the second time.
>
> If I use the ending http-proxy locally (that is by connecting via ssh to
> the server and using the local http proxy to download this very same file
> from the internet) the transfering is very fast. The internet connection of
> this server is a lot faster that the one I have at the gateway of my LAN.
>
> Could it be that the communication between WANProxy end point with a http
> proxy creates too many packets between them, and this problem gets somehow
> translated to the WAProxy starting point "the client"?
>
> I hope I explained myself, if not I'll try to send some "drawings" to
> explain the problem better. Please excuse my english.
>

That's very helpful, thank you.  If you have packet captures that might be
helpful.  I would /guess/ that it's possible that your remote proxy server
is reencoding the data being downloaded, that is that it's using the
chunked encoding and chunking the data differently, or using a deflate
encoding (so it's compressing the data) but chunking the data differently,
so the data ends up being completely different.  Does that make sense or
seem at all plausible?  Having an HTTP-aware protocol module would help
this, but I'm not sure it's even your issue.

Juli.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wanproxy.org/pipermail/wanproxy-wanproxy.org/attachments/20130816/4110dd0b/attachment-0003.htm>