WANproxy stability and its memory management

Juli Mallett juli at clockworksquid.com
Fri May 9 14:02:59 PDT 2014 

There must be some limit to how much I can top-reply to myself, and I sure
hope this is it.

The code from yesterday was indeed a little compromised by being a dirty
hack.  If you update to r1440, the LRU should actually work properly.

If you want to experiment with this, make clean and then apply the attached
patch and then rebuild WANProxy.  It will limit in-memory cache to 1GB for
each local codec instance and 512MB for each peer.  Note that there are no
safeguards here and that you could have connections get reset (although
it's unlikely) due to them referencing pages that get pushed out by the
LRU, if the peer takes long enough to ask for a redefinition.  You could
also potentially have issues due to names being reused for hashes, since
we're still just using the hash for the name directly, and not tracking
evictions or reuse or anything like that.  There's ways to handle all of
that, but I don't know when I'll get around to that and wanted to offer
something up that people can experiment with if they understand all of
those caveats.

Thanks,
Juli.


On Fri, May 9, 2014 at 12:55 AM, Juli Mallett <juli at clockworksquid.com>wrote:

> I wanted to post to the list and mention that although it's not exposed to
> user usage yet and doesn't have the cache of recently-referenced paged in
> the encoder pipe that I mentioned, I've added the basic low-level support
> for clamping memory usage of the in-memory cache.  It uses an LRU for
> eviction.  The commit is r1439.  If anyone wants to review it, I'd be
> grateful, since it was a bit of a late-night "I'm sick of having not done
> this" hack.
>
>
> On Thu, May 8, 2014 at 12:29 AM, Juli Mallett <juli at clockworksquid.com>wrote:
>
>> Thanks, Alain-Pierre;
>>
>> And you're absolutely right that this is one of the big missing pieces.
>>  It's actually not very hard to get working at a basic level, to make
>> WANProxy rock-solid as you say; the problem is that (more than) all of my
>> time has been spoken for.  That said, I'd really like to find time in the
>> next few months.  I may have time even in late May or early June, but I may
>> also just need to take some downtime instead.  That's always the hard
>> choice to have to make.
>>
>> My priorities would be:
>>
>> 1) Clamp memory usage (to start with, a limit for the local store, and a
>> limit for our cache of pages from each peer) and discard oldest pages.
>>  Obviously need to do some work here to ensure that if the remote side asks
>> us to give a definition for a reference we've sent, we're able to do so; so
>> you keep a cache of recently-referenced pages in the xcodec pipes so that
>> even if they disappear from the cache proper they'll be around up to some
>> reasonable limit.
>> 2) Fast, reliable persistent storage that can be used to page out some
>> pages rather than discarding them entirely.
>> 3) A smarter replacement algorithm.
>>
>> Endless improvements are possible beyond that, but there's diminishing
>> returns or at least a worsening cost-benefit ratio.  However at that point
>> WANProxy's a better platform for, say, someone doing a Master's Thesis to
>> work on what kinds of replacement algorithms, for instance, make sense for
>> WAN Optimization.  And small incremental improvements can be made, like
>> limiting the number of peers that can connect, and providing a better
>> management interface so you can purge old peers and don't bother holding
>> onto pages for peers that no longer exist, and aren't vulnerable to a
>> Denial-of-Service if someone wants to connect a million times with unique
>> UUIDs.  (That is, eventually you want a memory limit for all peers combined
>> as well, and to pressure out old peers entirely.  This isn't very hard to
>> get right, shockingly-enough.)
>>
>> I think the first two items there are what you really want, and are
>> probably 3-4 weeks of work for me; or maybe Patrick Kelsey if he has the
>> time, which I don't know.  I don't know when I'll have all that much time;
>> on the other hand, I think I could get both of them working on a
>> particularly-ecstatic weekend if the right mood hit and everything fell
>> into place.
>>
>> Which is all to say, "I can do it; I know how to do it; it won't take
>> very long once I start; I don't know when I'll have the time to start."  If
>> this is something that the community was interested in funding, I could
>> definitely put it on my radar for this summer, but unfortunately a large
>> number of other things are vying for my attention as well.  I want to do
>> right by the people who have wrestled with and made use of WANProxy,
>> however, because it's really essential to me that this be something useful,
>> reliable and easy-to-use for anyone in the world, even if they can't afford
>> fancy hardware or a support contract.
>>
>> The encouragement (and praise, for that matter) helps, and helps me to
>> want to make it a priority.  And I do want to make it a priority.  So thank
>> you.
>>
>> Juli.
>>
>>
>>
>> On Thu, May 8, 2014 at 12:13 AM, Alain-Pierre Perrin <apperrin at wanadoo.fr
>> > wrote:
>>
>>> Hello Juli and other WANproxy contributors.
>>>
>>>
>>> "Too long, won't read:" Does WANproxy "knows" when to stop allocating
>>> memory and to start purging the oldest/less frequently used pages ?
>>>
>>> This question may sound dumb but here's why I ask it.
>>>
>>> I tried to use WANproxy (versions 0.8, SVN and diegows variant) on
>>> dedicated Linux (up to date and minimalistic Arch Linux) virtual
>>> machines to compress/deduplicate replication streams on a WAN link
>>> between two Dell Equallogic SANs. With the correct configuration files
>>> (the web site should include a lot more of examples and variants because
>>> I've built my config files by trial-and-error), it works beautifully...
>>> but after some time, WANproxy gets purged by the Linux OOM (out of
>>> memory) killer when the limits of available RAM (8G) are reached on the
>>> two dedicated VMs created to host it. My replication processes get
>>> interrupted until I restart the WANproxy processes on the two proxy
>>> machines.
>>>
>>> How can we help to make WANproxy rock-solid ? I'd rather have passable
>>> performance (at least to begin) with killer stability over good
>>> performance with the need to restart WANproxy processes every 4 hours...
>>>
>>> To conclude I sincerely congratulate Juli and the other WANproxy
>>> developers/contributors for their work. Among the few (only three, with
>>> TrafficSqueezer and OpenNOP) opensourced WAN optimization software
>>> projects that I found on the net, WANproxy seemed the "least badly
>>> documented" and the most elegant in terms of simplicity (one process,
>>> one config file, no low-level OS customization needed) and modularity.
>>> With some polish in the code (to make wanproxy process more stable and
>>> resilient) and in the doc (to make wanproxy easier to understand and
>>> configure) this little tool, adhering closely to Unix philosophy, could
>>> become as indispensable as rsync, nmap or ssh in every sysadmin
>>> toolbox...
>>>
>>> Thanks in advance.
>>>
>>>
>>> Alain-Pierre Perrin
>>> _______________________________________________
>>> wanproxy mailing list
>>> wanproxy at lists.wanproxy.org
>>> http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wanproxy.org/pipermail/wanproxy-wanproxy.org/attachments/20140509/e23667b4/attachment-0002.htm>
-------------- next part --------------
Index: programs/wanproxy/wanproxy_config_class_codec.cc
===================================================================
--- programs/wanproxy/wanproxy_config_class_codec.cc	(revision 1437)
+++ programs/wanproxy/wanproxy_config_class_codec.cc	(working copy)
@@ -51,7 +51,7 @@
 
 		XCodecCache *cache = XCodecCache::lookup(uuid);
 		if (cache == NULL) {
-			cache = new XCodecMemoryCache(uuid);
+			cache = new XCodecMemoryCache(uuid, 1024 * 1024 * 1024);
 			XCodecCache::enter(uuid, cache);
 		}
 		XCodec *xcodec = new XCodec(cache);
Index: xcodec/xcodec_pipe_pair.cc
===================================================================
--- xcodec/xcodec_pipe_pair.cc	(revision 1437)
+++ xcodec/xcodec_pipe_pair.cc	(working copy)
@@ -209,7 +209,7 @@
 
 				decoder_cache_ = XCodecCache::lookup(uuid);
 				if (decoder_cache_ == NULL) {
-					decoder_cache_ = new XCodecMemoryCache(uuid);
+					decoder_cache_ = new XCodecMemoryCache(uuid, 512 * 1024 * 1024);
 					XCodecCache::enter(uuid, decoder_cache_);
 				}

More information about the wanproxy mailing list