Segment size

[Diego Woitasen]

Hi Juli,
 I was testing again with 512 and 2048 segment sizes and yes, with
compression there is no difference. The traffic is reduced to 11% with
512 segment size and 12% with 2048, always using xcodec+zlib
compression level 6. I tested this with real web traffic captures.

Regards,
 Diego

On Tue, Oct 4, 2011 at 11:14 PM, Juli Mallett <juli at clockworksquid.com> wrote:
> Hey Diego,
>
> It's a trade-off, and ideally I'd put in the effort to make this
> runtime tunable without sacrificing too much performance (this
> actually isn't very hard; I'll spare you the details, but it's
> something I could conceivably do in a weekend just by templating all
> XCodec-related classes.)
>
> You might find reading this thread informative (start at the bottom):
>
> http://lists.wanproxy.org/pipermail/wanproxy-wanproxy.org/2011-January/000087.html
>
> The short of it is that we use fewer segments, can keep smaller tables
> in memory, have much better performance (throughput), etc., if we use
> a larger segment size.  2KB was chosen empirically and using a bit of
> math to look at what the file-size impact would be.
>
> Using a smaller chunk, of course, increases the possibility of finding
> chunks that have been seen before.  If a file is identical, though,
> using 2KB segments should result in a smaller file overall.  It's only
> in the case of coincidence or a modified file that the smaller segment
> size makes a big difference.  And, really, that should only impact a
> small number of segments around the change, which is not very much in
> terms of bytes.
>
> If you do change the XCodec segment size, you need to change the
> buffer segment size in common/buffer.h as well and rebuild everything.
>  Otherwise, you're wasting a lot of RAM.
>
> And on top of all that, whatever the segment size, we also now support
> zlib compression.  Using 2KB segments and zlib compression should give
> similar throughput to using just 128 byte segments and no zlib, and
> yield much better reduction in data size.
>
> I'm happy to discuss it and consider changes — let me know what you think!
>
> Thanks,
> Juli.
>
> On Tue, Oct 4, 2011 at 18:40, Diego Woitasen <diego at woitasen.com.ar> wrote:
>> Hi,
>>  I was testing the Wanproxy compression and have questions about the
>> segment size. The test was done with traffic captured in one of the
>> branches of one of my clients. I used chaosreader to reassemble the
>> flows.
>>
>> With the default segment size (2048) the saving is 67%. I tried with
>> segment sizes of 1024 and 512. The saving is 71% and 78% respectively.
>> I haven't test this on the wire yet, I'll do it tomorrow. In the
>> meanwhile, my question is: what's the reason of a segment size of
>> 2048? Is it based on the experience?
>>
>> Regards,
>>  Diego
>>
>> --
>> Diego Woitasen
>> _______________________________________________
>> wanproxy mailing list
>> wanproxy at lists.wanproxy.org
>> http://lists.wanproxy.org/listinfo.cgi/wanproxy-wanproxy.org
>>
>



-- 
Diego Woitasen