WANproxy stability and its memory management

[Juli Mallett]

Thanks, Alain-Pierre;

And you're absolutely right that this is one of the big missing pieces.
 It's actually not very hard to get working at a basic level, to make
WANProxy rock-solid as you say; the problem is that (more than) all of my
time has been spoken for.  That said, I'd really like to find time in the
next few months.  I may have time even in late May or early June, but I may
also just need to take some downtime instead.  That's always the hard
choice to have to make.

My priorities would be:

1) Clamp memory usage (to start with, a limit for the local store, and a
limit for our cache of pages from each peer) and discard oldest pages.
 Obviously need to do some work here to ensure that if the remote side asks
us to give a definition for a reference we've sent, we're able to do so; so
you keep a cache of recently-referenced pages in the xcodec pipes so that
even if they disappear from the cache proper they'll be around up to some
reasonable limit.
2) Fast, reliable persistent storage that can be used to page out some
pages rather than discarding them entirely.
3) A smarter replacement algorithm.

Endless improvements are possible beyond that, but there's diminishing
returns or at least a worsening cost-benefit ratio.  However at that point
WANProxy's a better platform for, say, someone doing a Master's Thesis to
work on what kinds of replacement algorithms, for instance, make sense for
WAN Optimization.  And small incremental improvements can be made, like
limiting the number of peers that can connect, and providing a better
management interface so you can purge old peers and don't bother holding
onto pages for peers that no longer exist, and aren't vulnerable to a
Denial-of-Service if someone wants to connect a million times with unique
UUIDs.  (That is, eventually you want a memory limit for all peers combined
as well, and to pressure out old peers entirely.  This isn't very hard to
get right, shockingly-enough.)

I think the first two items there are what you really want, and are
probably 3-4 weeks of work for me; or maybe Patrick Kelsey if he has the
time, which I don't know.  I don't know when I'll have all that much time;
on the other hand, I think I could get both of them working on a
particularly-ecstatic weekend if the right mood hit and everything fell
into place.

Which is all to say, "I can do it; I know how to do it; it won't take very
long once I start; I don't know when I'll have the time to start."  If this
is something that the community was interested in funding, I could
definitely put it on my radar for this summer, but unfortunately a large
number of other things are vying for my attention as well.  I want to do
right by the people who have wrestled with and made use of WANProxy,
however, because it's really essential to me that this be something useful,
reliable and easy-to-use for anyone in the world, even if they can't afford
fancy hardware or a support contract.

The encouragement (and praise, for that matter) helps, and helps me to want
to make it a priority.  And I do want to make it a priority.  So thank you.

Juli.



On Thu, May 8, 2014 at 12:13 AM, Alain-Pierre Perrin <apperrin at wanadoo.fr>wrote:

> Hello Juli and other WANproxy contributors.
>
>
> "Too long, won't read:" Does WANproxy "knows" when to stop allocating
> memory and to start purging the oldest/less frequently used pages ?
>
> This question may sound dumb but here's why I ask it.
>
> I tried to use WANproxy (versions 0.8, SVN and diegows variant) on
> dedicated Linux (up to date and minimalistic Arch Linux) virtual
> machines to compress/deduplicate replication streams on a WAN link
> between two Dell Equallogic SANs. With the correct configuration files
> (the web site should include a lot more of examples and variants because
> I've built my config files by trial-and-error), it works beautifully...
> but after some time, WANproxy gets purged by the Linux OOM (out of
> memory) killer when the limits of available RAM (8G) are reached on the
> two dedicated VMs created to host it. My replication processes get
> interrupted until I restart the WANproxy processes on the two proxy
> machines.
>
> How can we help to make WANproxy rock-solid ? I'd rather have passable
> performance (at least to begin) with killer stability over good
> performance with the need to restart WANproxy processes every 4 hours...
>
> To conclude I sincerely congratulate Juli and the other WANproxy
> developers/contributors for their work. Among the few (only three, with
> TrafficSqueezer and OpenNOP) opensourced WAN optimization software
> projects that I found on the net, WANproxy seemed the "least badly
> documented" and the most elegant in terms of simplicity (one process,
> one config file, no low-level OS customization needed) and modularity.
> With some polish in the code (to make wanproxy process more stable and
> resilient) and in the doc (to make wanproxy easier to understand and
> configure) this little tool, adhering closely to Unix philosophy, could
> become as indispensable as rsync, nmap or ssh in every sysadmin
> toolbox...
>
> Thanks in advance.
>
>
> Alain-Pierre Perrin
> _______________________________________________
> wanproxy mailing list
> wanproxy at lists.wanproxy.org
> http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wanproxy.org/pipermail/wanproxy-wanproxy.org/attachments/20140508/4da3e576/attachment-0001.htm>