SharePoint farm backup fails if using TLS 1.2 connection and NetBackup client is version 8.1 or 8.1.1.

Transport Secure Layer (TLS) facilitates secure computer communication over network. TLS 1.2 is the latest available version that is gaining momentum as it is much more secure than its predecessors (TLS 1.1/1.0 and SSL 1.0/2.0/3.0).

Microsoft SharePoint Server supports TLS 1.2 from version 2010 onwards. If you want to enable it, obviously, you also need to configure SQL connection to strictly use TLS 1.2.

From version 8.1 onwards, NetBackup introduces a new feature called Secure Communication. Basically it is now mandatory for all NetBackup hosts to communicate securely via TLS 1.2.

Bear in mind, however, that the above feature does not yet extend to NetBackup’s interaction with SharePoint farm. If you upgrade the NetBackup software on your SharePoint farm to 8.1 or 8.1.1, and the servers happen to use TLS 1.2, content database backup and restore will fail. SQL agent backup does work fine, however.

You will see the following error in SQL Server’s bpbkar log:

03:42:10.599 [1220.796] <2> ov_log::V_GlobalLog: INF - SQL Error Source:
Microsoft OLE DB Provider for SQL Server
03:42:10.599 [1220.796] <2> ov_log::V_GlobalLog: INF - SQL Error Description:
[DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security error.

 

PS: This log is located in \Veritas\NetBackup\logs\bpbkar\. Create the folder if it does not exist and re-run the backup to get the log.

This issue will be fixed in NetBackup 8.1.2 which should be released soon. Meanwhile, you may want to roll back the NetBackup client software on the SharePoint Servers back to 7.7.3.

MSDP performance issue after upgrading to NetBackup 8.1 / Appliance 3.1

A lot of NetBackup customers would have upgraded to version 8.1 (3.1 if appliance) or later by now. One of the hottest topic in my team at the moment is how some customers encounter MSDP performance issue soon after upgrading to this version.

The stronger fingerprint algorithm in NetBackup 8.1’s MSDP is definitely good in the long run, but for those who recently upgraded from older NetBackup, the algorithm conversion from MD5 to SHA-2 can be a pain point – especially for the first few days post upgrade.

The issue may manifest in different ways: NetBackup Deduplication Engine (spoold) crashing, replication and/or duplication hanging and general slowness for MSDP-related operations.

If you think you are affected, download and install EEB 3935219 attached to technote: https://www.veritas.com/support/en_US/article.100041057

Note that if you already have EEB 3935219 installed, check the revision. The above technote contains the latest revision which is 11. Anything lower than 11 should be uninstalled and replaced by revision 11.

[9 May 2018] Update:

Some users may still notice performance issue even after data conversion has finished. If you are one of them, first, try lowering the amount of memory used by NetBackup deduplication engine (spoold) down from the default value of 75% to 50%.

Second, tune the OS kernel a little bit.

The steps are:

1. Open /msdp/data/dp1/pdvol/etc/puredisk/contentrouter.cfg file and search for MaxCacheSize. If the value is not 50%, set it to 50%. Save the file and restart NetBackup services. In theory you just need to restart spoold, but if you can afford to restart all NetBackup services, you will have a cleaner start.

2. Edit /etc/sysctl.conf file and ensure the 3 parameters below are set to the values given here (if these parameters don’t exist, just create them).

vm.overcommit_ratio = 100
kernel.numa_balancing = 0

Save the file and run sysctl -p to make the change effective.

nbcertcmd -createtoken fails with EXIT STATUS 8000: User does not have permission(s) to perform the requested operation

I wanted to deploy a NetBackup 8.1 client the other day, and one of the steps involved entering a token from master server. My master server was a NetBackup Appliance running software version 3.1.

As per the procedure, I went to the master server and logged in to the NetBackup Web Management Console first:

bpnbat -login -logintype WEB

Afterwards, I ran the following command to generate the token:

nbcertcmd -createtoken -tokenname token_name

But it failed with: EXIT STATUS 8000: User does not have permission(s) to perform the requested operation, despite using “admin” account which had root privileges.

My colleague Hoai (who is a brilliant troubleshooter, by the way) suggested to create a NetBackupCLI account:

• Go back to CLISH, and navigate to Main_Menu > Manage > NetBackupCLI
• Run: Create myUser
• You can change myUser to your user name.
• Enter the password for your user twice.
• Once created, log out and log back into the Appliance using the above account.
• Re-run bpnbat and nbcertcmd commands, they should work.

The vnetd proxy encountered an error

You may encounter this error after adding a new NetBackup 8.1 client in the policy and trying to access it via Host Properties > Clients.

One common reason for this error is the new client does not yet have a host ID-based certificate. Why is this so? Well, it could be an Administrator’s oversight during install process, especially if the install was automated using script – such as /usr/openv/netbackup/bin/install_client_files.

What you need to do is simple: deploy the client’s host ID-based certificate manually. The steps:

Go to master server and run:

/usr/openv/netbackup/bin/bpnbat -login -logintype WEB
/usr/openv/netbackup/bin/nbcertcmd -createToken -name token_name

NOTE: You can change the token_name. Once generated, copy the token string.

Now go to the client and run:

/usr/openv/netbackup/bin/nbcertcmd -getCertificate -host client_name -server netbackup -token

Replace client_name with the client’s hostname, and paste the token string when prompted.