persistent cache

[Juli Mallett]

On Mon, Nov 21, 2011 at 09:17, Diego Woitasen <diego at woitasen.com.ar> wrote:
> https://github.com/diegows/wanproxy
>
> There is, it's an ugly hack, far from a definitive real implementation
> but works. It doesn't support ageing yet. It's was developed to test
> the performance and to demostrate to one of my clients that it's not a
> 1 year of development. I coded this in a few hours.
>
> I think because Wanproxy is modular, we can have more than one
> on-disk-cache type. :)

You're exactly right that we can support multiple, and I was thinking
about including your bdb code quite seriously, or something like it,
as a stopgap.  I'd first need to handle the LRU (or whatever) problem,
getting rid of unused segments with no outstanding references, and,
even harder, exerting some kind of pressure if all of them are in use
and a limit has been reached.  I know how I want to handle making sure
that we don't send references to data by hash and then expunge the
segment from our cache, I think, but that's not trivial, too.

Have you seen the changes I recently made to tack and to xcodec_cache?
 I added what I hope are some helpful hooks for you and added examples
to tack of (1) using an on-disk cache [only suitable for tack, not
WANProxy, unless you only care about persistence and not paging] (2)
using no cache except the backref cache, which might be useful in
WANProxy for people who want to do some light deduplication on data
before sending it to zlib, but who don't want memory usage or any
added latency because of the need to send data back and forth to
answer unresolved references, etc.

> I'll try to implement this correctly with Bdb when I have more time.
>
> Regards,
>  Diego
>
> On Fri, Nov 18, 2011 at 11:42 PM, Juli Mallett <juli at clockworksquid.com> wrote:
>> On Fri, Nov 18, 2011 at 18:15, Eric Hester <eric.hester at gmail.com> wrote:
>>>
>>> > How far away is the persistent cache feature?
>>>
>>
>> diegows has an experimental version using bdb, which is non-ideal, but good
>> enough for some deployments.  I unfortunately still don't really have the
>> time to see to completion in the near future my own effort.
>> If you google for wanproxy bdb you should be able to find Diego's github
>> tree, I think.
>> Thanks,
>> Juli.
>>
>> _______________________________________________
>> wanproxy mailing list
>> wanproxy at lists.wanproxy.org
>> http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org
>>
>>
>
>
>
> --
> Diego Woitasen
>