<div><font size=2 color=navy face=Arial>
Juli,<br><br>Thanks for the detailed response. I am testing wanproxy against Netapp Snapmirror (replication) traffic. The NetApp SAN has deduplication at the storage level, so I'm guessing that is why I'm seeing more benefit from LZF. <br><br>I agree that if we were accessing standard file shares, that the active data set is probably better suited to dedup. <br><br>I am fairly well versed in WAN acceleration, having deployed Riverbed and Expand solutions in enterprise environments. I'm happy to see some community development in the area.<br><br><br></font></div>
<br><div><hr size=2 width="100%" align=center tabindex=-1>
<font face=Tahoma size=2>
<b>From</b>: Juli Mallett <juli@clockworksquid.com>
<br><b>To</b>: Tan, William
<br><b>Cc</b>: wanproxy@lists.wanproxy.org <wanproxy@lists.wanproxy.org>
<br><b>Sent</b>: Tue Dec 08 20:07:18 2009<br><b>Subject</b>: Re: Wanproxy on Fedora/Redhat
<br></font><br></div>
Hi William,<div><br></div><div>At some point I intend to implement standard compression on top of the existing algorithm (or to at least use traditional compression for data that has not been seen before.) Note that WANProxy does deduplication, not compression. I'd be surprised if LZF does as well for a real world data set in a WAN compression sense.</div>
<div><br></div><div>For a toy example, consider two people in a remote office reviewing a document that is stored on a central server. Let's assume it's a 22M PDF — I'm using the German-language documentation for the Propellerhead Reason software as an example because it's the biggest PDF I have on my hard drive. With no WAN compression, each time it's downloaded that's 22M of traffic that's going to flow over your link.</div>
<div><br></div><div>Now assume that you have a persistent 'gzip -9' of your traffic going on. The first time somebody downloads it, it's going to take 14M of traffic. The second time somebody downloads it, it'll take about that same amount.</div>
<div><br></div><div>With the XCodec (as is used in tack and wanproxy today), things are a little bit different because the dictionary that is used for compression is persistent. The first time somebody downloads it, it's going to take ~15.5M of traffic. The second time, though, mostly only references to the previously-sent chunks need to be sent, less than 2M of traffic.</div>
<div><br></div><div>The simplest way to test this is to take your data set (a tar file of the whole thing if you want some idea of how it'll do for all of your data, or you can go a file-at-a-time) and do something like:</div>
<div><br></div><div>tack -c dataset > dataset.t</div><div>tack -c dataset dataset > dataset.t-2</div><div><br></div><div>Then look at the difference in size. That difference is the amount of data that second (or third or fourth or ...) download is going to take. Note that each file passed on the command-line to tack is like a new connection — it uses the same persistent dictionary but its own connection-specific state. If you want to see what downloading the same file twice over the same connection looks like, do something more like 'cat dataset dataset | tack -c > dataset.t-3' — it should be similar, though in some cases (very low-entropy files) it might be better.</div>
<div><br></div><div>Note that the XCodec is pretty smart and will exhaustively look for chunks of data it has seen before when encoding, so you can change things in your file or add random data to the middle and still see substantial deduplication.</div>
<div><br></div><div>Thanks,</div><div>Juli.</div><div><br><div class="gmail_quote">On Tue, Dec 8, 2009 at 16:48, Tan, William <span dir="ltr"><<a href="mailto:wtan@eci.com">wtan@eci.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">It’s no big deal, but thank you to everyone for the
responses. We have a VMWare farm, so it only took me about 15
minutes to build a Ubuntu system and compile Wanproxy.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I was interested in testing the compression algorithm against
our sample data set, so tack was all I really ended up needing. As
it turns out in our case, LZF compression seems to be more effective than the Wanproxy
compression algorithm. I’m sure that the best algorithm would
vary depending on the use case.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Perhaps as part of the dev roadmap, it might make sense to
implement LZF as an option?</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> Juli Mallett
[mailto:<a href="mailto:juli@clockworksquid.com" target="_blank">juli@clockworksquid.com</a>] <br>
<b>Sent:</b> Tuesday, December 08, 2009 6:55 PM<br>
<b>To:</b> Tan, William<br>
<b>Cc:</b> <a href="mailto:wanproxy@lists.wanproxy.org" target="_blank">wanproxy@lists.wanproxy.org</a><br>
<b>Subject:</b> Re: Wanproxy on Fedora/Redhat</span></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<p class="MsoNormal">Sorry for the trouble — I've started work on a GNU
Make build system but have run into a problem that is probably solvable
differently in a trivial way than BSD Make, but because I'm used to the latter
I haven't looked up how to do it properly in the former — defining a
target only if no target by that name has been defined. I think I might
have also had some trouble figuring out what to do with something like '.if
make(foo)' in BSD Make, but I don't remember if WANProxy uses any of those.</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Anyway, here's a trivial GNU-style Makefile that will
probably work:</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">%%%</p>
</div>
<div>
<div>
<p class="MsoNormal">SRCS= proxy_client.cc proxy_listener.cc
proxy_socks_connection.cc proxy_socks_listener.cc wanproxy.cc
wanproxy_config.cc wanproxy_config_class_codec.cc wanproxy_config_class_interface.cc
wanproxy_config_class_peer.cc wanproxy_config_class_proxy.cc
wanproxy_config_class_proxy_socks.cc wanproxy_config_type_codec.cc
../../common/buffer.cc ../../common/log.cc ../../common/timer.cc
../../config/config.cc ../../config/config_class.cc
../../config/config_class_log_mask.cc ../../config/config_object.cc
../../config/config_type_int.cc ../../config/config_type_log_level.cc
../../config/config_type_pointer.cc ../../config/config_type_string.cc
../../config/config_class_address.cc ../../config/config_type_address_family.cc
../../event/event_poll.cc ../../event/event_system.cc ../../event/timeout.cc
../../event/event_poll_poll.cc ../../io/file_descriptor.cc
../../io/io_system.cc ../../io/pipe_link.cc ../../io/pipe_null.cc ../../io/pipe_sink.cc
../../io/pipe_pair_echo.cc ../../io/pipe_simple.cc ../../io/socket.cc
../../io/splice.cc ../../io/splice_pair.cc ../../io/unix_client.cc
../../io/unix_server.cc ../../net/tcp_client.cc ../../net/tcp_server.cc
../../net/udp_client.cc ../../net/udp_server.cc ../../xcodec/xcodec_decoder.cc
../../xcodec/xcodec_encoder.cc ../../xcodec/xcodec_decoder_pipe.cc
../../xcodec/xcodec_encoder_pipe.cc</p>
</div>
<div>
<p class="MsoNormal">CXXFLAGS=-I../.. -include common/common.h -DXCODEC_PIPES
-DUSE_POLL_POLL</p>
</div>
<div>
<p class="MsoNormal">wanproxy:</p>
</div>
<div>
<p class="MsoNormal"><span> </span>c++
-o wanproxy ${CXXFLAGS} ${SRCS} -lrt</p>
</div>
<div>
<p class="MsoNormal">%%%</p>
</div>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Just put that in 'GNUmakefile' in programs/wanproxy and run
'make' there. Obviously you could also just invoke c++ by hand, given
that level of sophistication.</p>
</div>
<div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">On Tue, Dec 8, 2009 at 08:44, Tan, William <<a href="mailto:wtan@eci.com" target="_blank">wtan@eci.com</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal">I
haven’t had any luck building wanproxy (0.6.0) on Linux (Fedora 8-11 to
be specific). I am experiencing the same problem described by previous
posters with pmake not liking the .include.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Has
anyone successfully built on a recent 2.6 linux kernel, and can they provide
any guideance?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
</div>
</div>
<table border="0" cellpadding="0">
<tbody><tr>
<td style="background:white;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black"><br>
NOTICE: The information contained in this transmission is privileged,
confidential, and intended only for the use of the individual or entity named
above. If you are not the intended recipient, you are hereby notified that
any disclosure, copying, distribution, or the taking of any action in
reliance on the contents of this transmission is strictly prohibited. If you
have received this transmission in error, please notify Eze Castle
Integration, Inc. by e-mail and destroy the original message and all copies.
Thank you.<br>
<br>
<br>
</span></p>
</td>
</tr>
</tbody></table>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
wanproxy mailing list<br>
<a href="mailto:wanproxy@lists.wanproxy.org" target="_blank">wanproxy@lists.wanproxy.org</a><br>
<a href="http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org" target="_blank">http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org</a></p>
</div>
<p class="MsoNormal"> </p>
</div>
</div></div></div>
</div><div><div></div><div class="h5">
<table><tbody><tr><td bgcolor="#ffffff"><font color="#000000"><br>
NOTICE: The information contained in this transmission is privileged, confidential, and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this transmission is strictly prohibited. If you have received this transmission in error, please notify Eze Castle Integration, Inc. by e-mail and destroy the original message and all copies. Thank you.<br>
<br>
<br>
<br>
</font></td></tr></tbody></table></div></div></blockquote></div><br></div>
<table><tr><td bgcolor=#ffffff><font color=#000000><br>
NOTICE: The information contained in this transmission is privileged, confidential, and intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or the taking of any action in reliance on the contents of this transmission is strictly prohibited. If you have received this transmission in error, please notify Eze Castle Integration, Inc. by e-mail and destroy the original message and all copies. Thank you.<br>
<br>
<br>
<br>
</font></td></tr></table>