MSDP performance issue after upgrading to NetBackup 8.1 / Appliance 3.1

A lot of NetBackup customers would have upgraded to version 8.1 (3.1 if appliance) or later by now. One of the hottest topic in my team at the moment is how some customers encounter MSDP performance issue soon after upgrading to this version.

The stronger fingerprint algorithm in NetBackup 8.1’s MSDP is definitely good in the long run, but for those who recently upgraded from older NetBackup, the algorithm conversion from MD5 to SHA-2 can be a pain point – especially for the first few days post upgrade.

The issue may manifest in different ways: NetBackup Deduplication Engine (spoold) crashing, replication and/or duplication hanging and general slowness for MSDP-related operations.

If you think you are affected, download and install EEB 3935219 attached to technote: https://www.veritas.com/support/en_US/article.100041057

Note that if you already have EEB 3935219 installed, check the revision. The above technote contains the latest revision which is 11. Anything lower than 11 should be uninstalled and replaced by revision 11.

[9 May 2018] Update:

Some users may still notice performance issue even after data conversion has finished. If you are one of them, first, try lowering the amount of memory used by NetBackup deduplication engine (spoold) down from the default value of 75% to 50%.

Second, tune the OS kernel a little bit.

The steps are:

1. Open /msdp/data/dp1/pdvol/etc/puredisk/contentrouter.cfg file and search for MaxCacheSize. If the value is not 50%, set it to 50%. Save the file and restart NetBackup services. In theory you just need to restart spoold, but if you can afford to restart all NetBackup services, you will have a cleaner start.

2. Edit /etc/sysctl.conf file and ensure the 3 parameters below are set to the values given here (if these parameters don’t exist, just create them).

vm.overcommit_ratio = 100
kernel.numa_balancing = 0

Save the file and run sysctl -p to make the change effective.