Performance impact of ParallelDeflateThreshold

Oct 25, 2012 at 2:47 PM
Edited Oct 25, 2012 at 2:48 PM

I have been using HV Backup for about six months now. I started on a Server 2008 R2 SP1 Hyper-V farm and recently migrated to a Server 2012 cluster. As the number of VMs increased, I noticed that backups for all the VMs could run well over a day which impacted performance during peak hours. I noticed the the transfer speed over the network would look like a waveform without any consistent throughput. I originally thought it might have been a buffer issue since I was transferring the file over the network but doing tests writing directly to disk got me the same performance.

I started to look at the source code of the application and noticed a line setting the ParallelDeflateThreshold to -1 for the zip file. Researching this I found out that this basically turns off multi-threaded compression. I then went back and ran another backup which proved that only one core of my 8 core server was being consumed. I went back and found the discussion that led to this setting being added to the code (http://hypervbackup.codeplex.com/discussions/350657) which notes a corruption of the zip being created. I noticed this discussion took place in March although the DLL that is currently provided with HV Backup wasn't released in July which means this issue was occurring on an older version of the DotNetZIp library. This got me wondering if maybe the issue is corrected in the newer version allowing me to comment out this setting and get back a huge performance increase for my backups.

I went ahead and tried it and recompiled the code and it took one of my test VM backup times from around 38 minutes to 8 minutes. I then unzipped the backup and restored it to another host server where I was able to successfully power on and use the VM. I was mainly writing this as a food-for-thought for others. I would like to keep this setting commented out in our environment although I would like to write an additional script that could verify the zip is readable after creation. Let me know if you have any questions!

Oct 31, 2012 at 9:20 AM

Hello,

i would appreciate if this setting would be tested again - backing up is really slow in the moment and speed this up 4.5 times sounds quite good.

Can either Alex modify the version for parallelDeflate again (maybe marked as alpha/beta/dangerous and not as public release first) or bestep share his compiled version for testing?

For the verify: this is surely a good idea. I am not sure if the CheckZip in DotNetZip is enough (just directory integrity?), maybe a full test (like in 7zip) or even extract (to nul:) is necessary. Maybe something like this is already in DotNetZip.

br, tm

Coordinator
Oct 31, 2012 at 9:41 AM

Hi, I just checked and the latest DotNetZip release is dated 2011. The latest tests we did were sadly not positive, so we dropped support for parallel compression, but it's definitely something that will be retested as soon as there's a new DotNetZip release.

Thanks,

Alessandro

Oct 31, 2012 at 12:31 PM

Wow I feel like an idiot! I was looking at the release date month and apparently didn't see the year listed as 2011. I guess I just assumed the DotNetZIp project was a little more active than that. Regardless, I will say that I haven't had any issues with parallel compression since I enabled it although it has only been about a week. I wonder if it would be worth it to move to another library to handle the zip files? Although I hate moving away from DotNetZip since they are clearly the most popular out there. Anyhow, sorry about that. :-)

Brent