<p>Hi Uttam,</p>
<p>Deflate (zlib) is fine for low entropy traffic but yields, as you suggest, no improvement at the historical scale or between connections.</p>
<p>The xcodec damage is not too hard to correct now that I have enhanced the infrastructure in the last week, but I may need to wait until the 17th or thereabouts to make the change. Hopefully the inconvenience is not great.</p>
<p>In trunk you can run with a codec of None and a compressor of zlib (see wanproxy.conf in trunk) if you want to test zlib and the proxy infrastructure for now.</p>
<p>(Forgive the terseness I am replying by phone on a touchscreen.)</p>
<p>Juli.<br>
</p>
<div class="gmail_quote">On Dec 10, 2010 7:36 PM, "Uttam Singh" <<a href="mailto:us@iptvlabs.com" target="_blank">us@iptvlabs.com</a>> wrote:<br type="attribution">> Hi Juli,<br>> <br>> Here are some additional test results:<br>
> <br>> 1. I have repeated some tests in a flat network (no WAN) and the<br>> original reported problems still remain. This was done to eliminate<br>> any issues with NAT/Firewall etc. on WAN router.<br>> <br>
> 2. The results in this flat network (switched ethernet) are consistent<br>> with the results seen in the WAN setup. Wanproxy-Trunk doesn't seem to<br>> work with anything. wanproxy-0.6.2 fares a little better with some<br>
> tests. However, iperf (windows version 1.7.0) between 2 stations with<br>> wanproxy-0.6.2 as well as wanproxy-trunk fails everytime.<br>> <br>> 3. If I set ".._codec None" - tests complete successfully with both<br>
> wanproxy-trunk and wanproxy-0.6.2. So the problems seem to be Xcodec<br>> related.<br>> <br>> I have also written a quick and dirty tcp-tunnel which essentially<br>> does wanproxy without compression. I am thinking of plugging in zlib<br>
> to get some baseline results. I noticed that you have started<br>> integrating zlib in wanproxy-trunk. Can you please share your thoughts<br>> on how much better Xcodec performance is as compared to zlib?<br>> <br>
> My thoughts are that zlib performance improvements will be marginal in<br>> TCP WAN traffic scenario as compared to a history based compression<br>> algorithm (which is what I think Xcodec is).<br>> <br>> thanks,<br>
> <br>> -Uttam<br>> <br>> On Sun, Dec 5, 2010 at 11:57 AM, Mallett, Juli <<a href="mailto:juli@clockworksquid.com" target="_blank">juli@clockworksquid.com</a>> wrote:<br>>> Hi Uttam,<br>>><br>
>> I must have unintentionally broken something recently, thanks for your<br>
>> feedback, I'll look in to that and come back to your problem. Thanks<br>>> for your patience.<br>>><br>>> Thanks,<br>>> Juli.<br>>><br>>> On Sun, Dec 5, 2010 at 09:39, Uttam Singh <<a href="mailto:us@iptvlabs.com" target="_blank">us@iptvlabs.com</a>> wrote:<br>
>>> Hi Juli<br>>>><br>>>> Thanks for your help so far.<br>>>><br>>>> wanproxy from svn/trunk seems to be broken. Please see other details below.<br>>>><br>>>> A real WAN connection is being used in my tests. I have a VDSL2<br>
>>> 100Mb/100Mb connection between client and server site. I am modifying<br>>>> DSL settings to force WAN connection speed at 5Mb/5Mb for these tests.<br>>>><br>>>> I have checked out and built wanproxy from svn/trunk and file<br>
>>> transfers (upload or, download) don't work now. I did a gmake clean<br>>>> and a top level build of everything under wanproxy.<br>>>><br>>>> I see file transfer start but it hangs pretty much after first dozen<br>
>>> packets are sent between the 2 proxies. If I revert back to the 0.6.2<br>>>> wanproxy build, without changing any other variable, things work like<br>>>> before.<br>>>><br>>>> Tack Results:<br>
>>><br>>>> 1. sample test file (~10MB binary file)<br>>>><br>>>> %tack -TQc ~/test1.bin<br>>>> 1291591407.945118 [/codec_timer] INFO: 169 timer samples.<br>>>> 1291591407.945336 [/codec_timer] INFO: 4804377 total runtime.<br>
>>> 1291591407.945406 [/codec_timer] INFO: 28428 mean microseconds/call.<br>>>><br>>>> 2. sample text file (~10MB text file)<br>>>><br>>>> %tack -TQc ~/test1.txt<br>>>> 1291592129.859450 [/codec_timer] INFO: 159 timer samples.<br>
>>> 1291592129.859670 [/codec_timer] INFO: 1353549 total runtime.<br>>>> 1291592129.859812 [/codec_timer] INFO: 8512 mean microseconds/call.<br>>>><br>>>> Please let me know if you need any debug output from wanproxy.<br>
>>><br>>>> thanks,<br>>>><br>>>> -Uttam<br>>>><br>>>> On Sat, Dec 4, 2010 at 10:00 PM, Mallett, Juli <<a href="mailto:juli@clockworksquid.com" target="_blank">juli@clockworksquid.com</a>> wrote:<br>
>>>> Hi Uttam,<br>>>>><br>>>>> To do your WAN simulation, are you using dummynet on the systems in<br>>>>> question, or are you simulating those links separately (or using real<br>
>>>> links)?<br>>>>><br>>>>> I suspect that the problem is either one of the performance<br>>>>> pessimizations that I recently fixed in Subversion or something in<br>>>>> network configuration.<br>
>>>><br>>>>> Just to be sure, can you do one or both of these:<br>>>>><br>>>>> (1) Build 'tack' in programs/tack and run 'tack -TQc file.bin' where<br>>>>> file.bin is the file you're transferring. This will give some idea of<br>
>>>> how fast the XCodec is on that file.<br>>>>> (2) Rebuild WANProxy on both systems from Subversion.<br>>>>><br>>>>> To build from Subversion do:<br>>>>> sudo pkg_add -r subversion (if you don't already have it installed)<br>
>>>> svn co <a href="http://wanproxy.org/svn/trunk" target="_blank">http://wanproxy.org/svn/trunk</a> wanproxy-src<br>>>>> cd wanproxy-src/programs/wanproxy<br>>>>> gmake NDEBUG=1<br>>>>><br>
>>>> I did a few tests on a netbook that I have with a similar CPU and it<br>>>>> looks like it should be able to handle as much as 100Mbps for some<br>>>>> kinds of data. I'm very surprised by the poor performance you note,<br>
>>>> so I am wondering if it's a problem with latency simulation or similar<br>>>>> (for instance, putting the latency between the client and the client<br>>>>> wanproxy rather than just between the two wanproxy boxes.)<br>
>>>><br>>>>> Hopefully we can work this out quickly!<br>>>>><br>>>>> Thanks,<br>>>>> Juli.<br>>>>><br>>>>> On Sat, Dec 4, 2010 at 07:11, Uttam Singh <<a href="mailto:us@iptvlabs.com" target="_blank">us@iptvlabs.com</a>> wrote:<br>
>>>>> Hi Juli<br>>>>>><br>>>>>> Thanks for your prompt response<br>>>>>><br>>>>>> I see your point, my performance results may be system bound.<br>
>>>>><br>>>>>> My Wanproxy systems have the following specs:<br>>>>>><br>>>>>> - 1.6GHz dual-core Atom<br>>>>>> - 2GB 533MHz DDR2 memory<br>>>>>> - FreeBSD 8.0 Stable dedicated for wanproxy<br>
>>>>> - each system is running build wanproxy-0.6.2 downloaded from<br>>>>>> <a href="http://wanproxy.org/releases/wanproxy-0.6.2.tar.gz" target="_blank">http://wanproxy.org/releases/wanproxy-0.6.2.tar.gz</a><br>
>>>>><br>>>>>> - In my tests I am trying to get best results possible, I am doing<br>>>>>> multiple transfers in succession (i.e. upload multiple times and then<br>>>>>> download multiple times)<br>
>>>>><br>>>>>> I changed my WAN settings to 5Mbps symmetrical (up/down).<br>>>>>><br>>>>>> Now I get following results while transferring same file multiple times:<br>
>>>>><br>>>>>> No Wan Proxy:<br>>>>>> - Up: 4Mbps<br>>>>>> - Down: 4Mbps<br>>>>>><br>>>>>> Wan Proxy:<br>>>>>> - Up: 3.23Mbps<br>
>>>>> - Down: 2.71Mbps<br>>>>>><br>>>>>> So compression may be adding to the latency here. Maybe given the<br>>>>>> system spec, this is only meaningful for slow links (1Mbps or, less).<br>
>>>>> Though I don't quite understand why the CPU usage stats are so low.<br>>>>>> Also, there is still a material performance difference between upload<br>>>>>> and download - which is an anomaly.<br>
>>>>><br>>>>>> Please see other comments inline >><br>>>>>><br>>>>>> On Fri, Dec 3, 2010 at 9:53 PM, Mallett, Juli <<a href="mailto:juli@clockworksquid.com" target="_blank">juli@clockworksquid.com</a>> wrote:<br>
>>>>>> Hi Uttam,<br>>>>>>><br>>>>>>> Can you give me some information on the systems that you are running<br>>>>>>> WANProxy on? You should see deduplication in each direction, although<br>
>>>>>> they are, right now, independent — that is, if you upload a file and<br>>>>>>> then download it, you won't get the benefit. This is a short-term<br>>>>>>> shortcoming that did not used to exist, but is necessary during a<br>
>>>>>> transitional period in which I am working on improving support for<br>>>>>>> many-clients, many-servers models of use, as opposed to the old<br>>>>>>> one-client one-server model. I think that your test involves multiple<br>
>>>>>> downloads of the same file, so you should see performance improvements<br>>>>>>> on download.<br>>>>>>><br>>>>>>> I have a few more questions (and will flesh out the one I started this<br>
>>>>>> message with):<br>>>>>>><br>>>>>>> 1) Are you using a Subversion-built version or an old release?<br>>>>>><br>>>>>><br>>>>>>>> 0.6.2 tar release<br>
>>>>><br>>>>>><br>>>>>>> 2) Are the systems WANProxy is running on very fast? I'm wondering if<br>>>>>>> 2Mbps is as fast as the build of WANProxy you are using can run — it<br>
>>>>>> seems unlikely since that is quite slow, but I want to be sure. I've<br>>>>>>> made a lot of performance improvements lately, so if you're running a<br>>>>>>> Subversion build this should definitely not be the problem unless you<br>
>>>>>> are running on systems with very slow RAM or CPU.<br>>>>>><br>>>>>><br>>>>>>>> 1.6GHz dual-core Atom with 2GB 533Mhz RAM. Wanproxy systems are dedicated and not running much of anything else.<br>
>>>>>>> While doing the file transfers I see Wanproxy top out at 5 - 8% CPU (while CPU is 90+% idle)<br>>>>>><br>>>>>><br>>>>>>> 3) Are you in fact doing multiple downloads and multiple uploads for<br>
>>>>>> your tests of the same file, or are you doing an upload and then a<br>>>>>>> download and then another upload or...what?<br>>>>>>><br>>>>>><br>>>>>>>> Multiple uploads of the same file and then multiple downloads of the same file.<br>
>>>>><br>>>>>>> I hope that we can resolve your performance issues!<br>>>>>>><br>>>>>>> Thanks,<br>>>>>>> Juli.<br>>>>>>><br>
>>>>><br>>>>><br>>>><br>>><br></div>