Hi, Juli,<div><br></div><div>We came up with another question:</div><div><br></div><div>We tested sending 2G files to the "client", the second time the transmission from "client" to "server" was faster , but from "server" to "intranet" , it was still the same speed as first time. So wanproxy is only optimizing traffic between "client" and "server" , but not "server" and "intranet" , correct? If so what is the purpose of the "intranet " in this setup? </div>
<div><br></div><div>Thank you. <br><br><div class="gmail_quote">On Fri, Mar 8, 2013 at 11:05 PM, Juli Mallett <span dir="ltr"><<a href="mailto:juli@clockworksquid.com" target="_blank">juli@clockworksquid.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Fri, Mar 8, 2013 at 10:54 PM, Boxiang Pan <<a href="mailto:aquarypbx@gmail.com">aquarypbx@gmail.com</a>> wrote:<br>
> Hi, Juli,<br>
><br>
> We have successfully used wanproxy to transfer files from client to the<br>
> intranet through the server. And we managed to time the transmission time.<br>
> The second time, it indeed transferred faster. We've also managed to run<br>
> wanproxy from our program. Thanks a lot for all the help!<br>
><br>
> We have a few more questions about how wanproxy is implemented at top level.<br>
><br>
> 1) Since it takes less time to transfer the second time, we assume that the<br>
> file is cached somewhere, so where is the file cached? Is it cached on the<br>
> "server", which is between the "client" and the "intranet"?<br>
<br>
</div>As you may have noticed, WANProxy works independently of the protocol<br>
being used. It has no notion of files, only data.<br>
<br>
Both client and server remember the data that has been sent between<br>
them in the past. They remember it in blocks of 2KB, and each block<br>
has a unique 64-bit name derived from the file hash. When data is<br>
being transferred between them, it is split up into blocks so that<br>
where possible they only transfer the 64-bit name for parts of the<br>
data being transferred, rather than the whole 2KB block. Where data<br>
has been inserted, removed or changed, the blocks around it will still<br>
be replaced with their names, but the new or changed piece will be<br>
transferred for the first time.<br>
<div class="im"><br>
> 2) In the scenario where a client is periodically backing up a large file,<br>
> say 2GB, to the "intranet", but each backup, the file only differs a small<br>
> portion (100MB) from the previous version, does wanproxy treat each backup<br>
> file as a completely different file and cache the entire file again, or does<br>
> it smartly only updated the new 100MB in the previously cached file? If so,<br>
> how does wanproxy tells the difference?<br>
<br>
</div>Why don't you try it? :) The above probably makes the answer<br>
obvious, but just to be clear:<br>
<br>
The data that are unchanged, that have been sent previously, will be<br>
replaced with their names, and the data which are new or which are<br>
changed will be transferred over the link and remembered for later<br>
use. So if a different 100MB changes every time, you should still<br>
only need to send about 100MB of data plus the names that correspond<br>
to the rest of it.<br>
<div class="im"><br>
> 3) If there have already been a lot of files cached on the "server", the<br>
> next time the "client" is sending a file to the "server", how does the<br>
> server know if that file has previously been cached?<br>
<br>
</div>See above. WANProxy does deduplication of TCP streams and is not file<br>
oriented in any way. As far as it knows, it could be deduplicating an<br>
IRC session, E-Mail, web traffic, file replication, etc.<br>
<div class="im"><br>
> 4) Is there any reference document/ readme about the implementation of<br>
> wanproxy that we may refer to?<br>
<br>
</div>I have sent other information to the mailing lists in the past. You<br>
can also read about how 'rsync' works, although rsync is<br>
file-oriented, but the algorithm is basically the same.<br>
<br>
There is a little bit of information here, but it is out of date about<br>
protocol details:<br>
<br>
<a href="http://wanproxy.org/performance.shtml" target="_blank">http://wanproxy.org/performance.shtml</a><br>
<br>
There have also been some posts to the mailing list, but there's not<br>
any one message that I would point you to. I'm happy to answer any<br>
more questions.<br>
<br>
Thanks,<br>
Juli.<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br></div><font color="#663333"><font face="verdana, sans-serif">Boxiang Pan</font><br><br></font><div><div><font color="#663333">Department of Electrical and Computer Engineering</font></div>
<div><font color="#663333">University of California, San Diego </font></div><div><font color="#663333">Tel: 858-999-7655<br></font><br></div><div><br></div></div>
</div>