Accelerator backup slow? Don’t ignore these warnings.

Accelerator is a time and storage-saving feature first introduced in NetBackup 7.5. Using dedicated track log, Accelerator can determine which files have changed when the subsequent backup runs, and only send the changes to the media server. This applies to full backup as well, so imagine having a full backup at the speed of incremental backup.

If there aren’t many changes to the files, you will be laughing all the way back to your server. The backup is really that quick.

But, yes, BUT you need to know how to use it to make the most out of it. I won’t discuss the initial configuration as it can be found here.

What I’m trying to say here is: Once you have completed the initial configuration and kicked off some jobs, please check the job detail to see if there is any Accelerator related warning mentioned below. If there is, fix it, else you won’t get the full performance benefit.

Scenario 1:
Dec 17, 2018 8:21:44 PM – Info bpbkar (pid=8528) not using change journal data for <D:\>: no forced rescan schedule configured
Dec 17, 2018 8:21:44 PM – Info bpbkar (pid=8528) not using change journal data for enumeration for <D:\> but will use it for change detection

Solution: Create a full backup schedule with Accelerator forced rescan option checked.

Scenario 2:
Dec 17, 2018 8:30:10 PM – Info bpbkar32 (pid=1216) not using change journal data for <D:\>: hard link or reparse point change detected
Dec 17, 2018 8:30:10 PM – Info bpbkar32 (pid=1216) not using change journal data for enumeration for <D:\> but will use it for change detection

Solution: The chance is the data being backed up resides in an NTFS deduplicated volume. Refer to https://www.veritas.com/support/en_US/article.000127405 for further info. You can contact Veritas NetBackup support for assistance.

Scenario 3:
Dec 17, 2018 8:38:15 PM – Info bpbkar32 (pid=844) not using change journal data for <D:\>: not configured for use
Dec 17, 2018 8:38:15 PM – Info bpbkar32 (pid=844) not using change journal data for enumeration for <D:\> but will use it for change detection

Solution: For Windows client, it is best to let Accelerator tap into NTFS change journal. It can be enabled by going to NetBackup Java Admin Console > Host Properties > Clients > double click the client > Properties > Windows Client > Client Settings > tick the Use Change Journal option. Click OK to confirm.

SharePoint 2016 GRT restore fails with status code 5: the restore failed to recover the requested files. Granular restore log (ncfgre) throws error 0xE0008488:Access is denied.

I encountered this problem recently and with only access to the restore logs, it took me a while to figure out that the root cause is as simple as restore destination pointing to the wrong box!

Longer story:

I have the following evidence on hand:

NCFGRE log from SQL Server

22/06/2018 08:49:55.508 [Debug] NB 51216 ncfrai 158 PID:4960 TID:9240 File ID:352 [No context] 4 [[fsys\spsv3] ] <from Consumer::initializeResource> WNetAddConnection2 returned: 67. Either connection failed or the network path is not accessible (../BEDSContext.cpp:159)
22/06/2018 08:49:55.509 [Debug] NB 51216 ncfrai 158 PID:4960 TID:9240 File ID:352 [No context] 1 [ResourceChildBEDS::_attach()] FS_AttachToDLE() Failed! (0xE0008488:Access is denied.) (../ResourceChild.cpp:674)

Additional environment info

• SharePoint 2016 front-end server and SQL back-end server are both VMWare virtual machines.
• SharePoint farm is backed up using VMWare policy with Enable SharePoint Recovery option checked.
• Now we need to restore some granular items.

Assuming you have experience with NetBackup SharePoint agent before, do you notice anything suspicious?

Well, I didn’t initially.

NetBackup always generates NCFGRE log in the SQL Server for as long as I remember. Until… SharePoint 2016 came along.

To connect to SharePoint 2016, NetBackup doesn’t use custom API anymore, instead it uses a standardized method called SharePoint Server Object Model (SSOM). With that, the granular recovery process has been moved to the SharePoint front-end server. It is no longer on the back-end server.

If you notice, the NCFGRE log in my case was created in the SQL Server, so that should raise a suspicion. Yes, this happens if the restore destination is pointing to the SQL Server.

Just bear this in mind: As far as NetBackup is concerned, SharePoint restore destination must always point to the SharePoint front-end server. This will solve the problem I mentioned at the beginning.

 

 

 

 

 

SharePoint farm backup fails if using TLS 1.2 connection and NetBackup client is version 8.1 or 8.1.1.

Transport Secure Layer (TLS) facilitates secure computer communication over network. TLS 1.2 is the latest available version that is gaining momentum as it is much more secure than its predecessors (TLS 1.1/1.0 and SSL 1.0/2.0/3.0).

Microsoft SharePoint Server supports TLS 1.2 from version 2010 onwards. If you want to enable it, obviously, you also need to configure SQL connection to strictly use TLS 1.2.

From version 8.1 onwards, NetBackup introduces a new feature called Secure Communication. Basically it is now mandatory for all NetBackup hosts to communicate securely via TLS 1.2.

Bear in mind, however, that the above feature does not yet extend to NetBackup’s interaction with SharePoint farm. If you upgrade the NetBackup software on your SharePoint farm to 8.1 or 8.1.1, and the servers happen to use TLS 1.2, content database backup and restore will fail. SQL agent backup does work fine, however.

You will see the following error in SQL Server’s bpbkar log:

03:42:10.599 [1220.796] <2> ov_log::V_GlobalLog: INF - SQL Error Source:
Microsoft OLE DB Provider for SQL Server
03:42:10.599 [1220.796] <2> ov_log::V_GlobalLog: INF - SQL Error Description:
[DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security error.

 

PS: This log is located in \Veritas\NetBackup\logs\bpbkar\. Create the folder if it does not exist and re-run the backup to get the log.

This issue will be fixed in NetBackup 8.1.2 which should be released soon. Meanwhile, you may want to roll back the NetBackup client software on the SharePoint Servers back to 7.7.3.

The NetBackup Admin console failed to establish a secure connection with the host masterserver.domain.com. The request was terminated with eror code: VRTS-24630

Let’s consider the following scenario:

You have been using NetBackup Java Administration Console on your Windows PC to manage your NetBackup servers remotely without problem, until recently. The only changes were both the master server and Java Administration Console were upgraded to version 7.7.3.

Now you keep getting this error when logging in to the Java Administration Console:
The NetBackup Admin console failed to establish a secure connection with the host masterserver.domain.com
The request was terminated with eror code: VRTS-24630

On further investigation, you find that as long as your user account is part of local Administrators group of the Windows PC, you can log in without issue. The problem is your company’s Group Policy does not allow that.

This can be explained by a change in code from NetBackup 7.7.x onwards. Your user account needs to be part of the local Administrators group to be able to log in. If that is not possible, you will need to implement a workaround, which is disabling User Access Control.

This has to be done via registry and not via User Account Control Settings in Control Panel.
On the Windows PC from where the Java Administration Console is launched, launch Registry Editor (regedit) and go to:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\

Double click “EnableLUA” and set the value to 0.

Reboot your PC so the change takes effect. After that, you should be able to log in using a non-Administrator account.

[Error] V-58000-500-0003: Unable to open network share. An internal error occurred. Restarting the appliance may solve this issue.

There will be times when you need to transfer files in and out of NetBackup Appliance server, for instance, to install hotfixes, transfer log files for troubleshooting, copy script files and so on.

NetBackup Appliance CLISH has a convenient way to open network share so you can transfer files from and to your Windows PC (or Linux/UNIX if that’s what you have). The steps are mentioned in these technotes:
https://www.veritas.com/support/en_US/article.100023444
https://www.veritas.com/support/en_US/article.100010965

Recently I encountered an issue where opening Logs share returned the following error:

[Error] V-58000-500-0003: Unable to open network share. An internal error occurred. Restarting the appliance may solve this issue.

At that time, I narrowed down the issue to invalid entries in /etc/exports file.

nbu-appliance:/etc # cat exports

#default exports establish by NetBackup
#/inst/client *(ro,async,no_subtree_check)
#default export for logs
#/log *(ro,async,no_subtree_check)
#default export for incoming_patches
#/inst/patch/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for incoming_plugins
#/inst/plugin/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for LiveUp
#/inst/patch/client *(ro,async,no_subtree_check)
/log *(ro,async,no_subtree_check)
/inst/patch/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for logforwarding
#/inst/logforwarding *(all_squash,rw,sync,no_subtree_check)

Notice the duplicate entries in bold red. After removing them and running exportfs -ra (so the /etc/exports is re-read), I was able to open the Logs share.

MSDP performance issue after upgrading to NetBackup 8.1 / Appliance 3.1

A lot of NetBackup customers would have upgraded to version 8.1 (3.1 if appliance) or later by now. One of the hottest topic in my team at the moment is how some customers encounter MSDP performance issue soon after upgrading to this version.

The stronger fingerprint algorithm in NetBackup 8.1’s MSDP is definitely good in the long run, but for those who recently upgraded from older NetBackup, the algorithm conversion from MD5 to SHA-2 can be a pain point – especially for the first few days post upgrade.

The issue may manifest in different ways: NetBackup Deduplication Engine (spoold) crashing, replication and/or duplication hanging and general slowness for MSDP-related operations.

If you think you are affected, download and install EEB 3935219 attached to technote: https://www.veritas.com/support/en_US/article.100041057

Note that if you already have EEB 3935219 installed, check the revision. The above technote contains the latest revision which is 11. Anything lower than 11 should be uninstalled and replaced by revision 11.

[9 May 2018] Update:

Some users may still notice performance issue even after data conversion has finished. If you are one of them, first, try lowering the amount of memory used by NetBackup deduplication engine (spoold) down from the default value of 75% to 50%.

Second, tune the OS kernel a little bit.

The steps are:

1. Open /msdp/data/dp1/pdvol/etc/puredisk/contentrouter.cfg file and search for MaxCacheSize. If the value is not 50%, set it to 50%. Save the file and restart NetBackup services. In theory you just need to restart spoold, but if you can afford to restart all NetBackup services, you will have a cleaner start.

2. Edit /etc/sysctl.conf file and ensure the 3 parameters below are set to the values given here (if these parameters don’t exist, just create them).

vm.overcommit_ratio = 100
kernel.numa_balancing = 0

Save the file and run sysctl -p to make the change effective.

Oracle Intelligent Policy backup fails with status code 239 when initiated from OpsCenter web GUI

Depending on your role, you may need to initiate backup job from OpsCenter web GUI instead of the regular NetBackup Java Administration Console.

If your OpsCenter is version 7.7.x or 8.0, there is a known issue where Oracle Intelligent Policy backup fails with status code 239 when initiated from OpsCenter web GUI (Monitor > Policies > Manual Backup). The same backup works fine when initiated from the NetBackup Java Administration Console.

This issue is fixed in OpsCenter version 8.1, but if you cannot upgrade yet, the only option is to contact Veritas Technical Support.

For OpsCenter version 7.7.3, simply mention Etrack 3907949 and ask for an EEB (Hotfix). For version 8.0, mention Etrack 3917921.

Please note the following restrictions currently apply for Oracle Intelligent Policy backup launched from OpsCenter:

• It is hard coded to back up all instances for all clients within that Oracle Intelligent Policy (granular selection of instances and/or clients is not allowed).
• It does not work for script-based Oracle backup.
• Currently only Firefox browser is supported (if you need to use other browsers, just open a case with Technical Support).

May 3, 2018: [Update] My colleague told me Internet Explorer 11 with latest update may also work. I haven’t tested it myself though.

Veritas Flex 5340 Appliance: Container, meet NetBackup.

Launched in November last year, the NetBackup 5340 appliance is a beast. It sports 2x 20-core Intel Xeon 6138 CPUs and 768GB DDR4 memory, with usable storage capacity of up to 1.92 PB. This month, we are introducing a product that looks similar to the 5340 appliance but runs NetBackup in a totally different way.

Unlike traditional 5340 appliance, the Flex 5340 is built from the ground up to run NetBackup server in Docker container. You can mix and match up to 6 NetBackup master and media servers, each running in its own container.  The beauty of container is it uses computing power more efficiently than Hypervisor-based Virtual Machine.

Theoretically we can run many NetBackup instances inside the Flex 5340 box, but for now, Engineering limits the number of instances to six. This allows performance to be comparable to a typical dedicated NetBackup server. In the future, you can deploy other Veritas applications as containers once they are available.

The Flex 5340 is cluster ready too. You can add a second Flex 5340 and cluster them together in approximately 15 minutes. If one node goes down, the containers can fail over and continue running.

 

 

 

NetBackup 8.1.1 is now available!

Admittedly, I should have written this post earlier. NetBackup 8.1.1 is already out since February 16th, 2018. Despite the version increasing by “0.0.1”, it is not a maintenance pack. It is a full release.

You can check out the list of new features, enhancements and changes here.  One of the most exciting new features (to me, anyway) is the option to choose either fixed-length deduplication or variable-length deduplication (VLD). VLD is disabled by default and can be granularly enabled for a particular backup policy or client.

There is an endless debate on which method is better, but I feel that each one has its own merit. Most backup software is stuck with only one method.

User is unable to log in to NetBackup Java Administration Console after upgrading to NetBackup 8.0

After upgrading NetBackup Master Server to version 8.0, you may encounter the following error when logging in to the Java Administration Console:

Could not connect to NetBackup Service Layer. You may not be able to perform the functions in the Administration Console that depends on connectivity to this service. Please ensure the nbsl service is up and running.

There can be various reasons why it happens. I have compiled a list of things to look at based on past experience. Hopefully one of them can resolve your issue.

Checklist:

1. NetBackup EMM database must be running. Verify by running below command:
Linux/UNIX: /usr/openv/db/bin/nbdb_ping
Windows: install_path\Veritas\NetBackup\bin\nbdb_ping

2. Master server’s Client_Name, EMMSERVER and the first entry in Server list must be identical with the host name mentioned in NetBackup’s CA certificate.

Linux/UNIX:
* Get the CA cert detail by running:
/usr/openv/netbackup/bin/nbcertcmd -listcacertdetails
* Compare the hostname with the above parameters stored in /usr/openv/netbackup/bp.conf

Windows:
* Get the CA cert detail by running:
install_path\NetBackup\bin\nbcertcmd -listcacertdetails
* Launch registry editor and go to HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config.

For example:

C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -listcacertdetails
Subject Name : /CN=nbatd/OU=root@nbumaster /O=vx
Start Date : Jun 30 13:55:39 2015 GMT
Expiry Date : Jun 25 15:10:39 2035 GMT
SHA1 Fingerprint : C4:81:6C:B6:66:25:C1:DA:82:E3:06:F3:23:26:4F:51:85:3B:B8:71

In this case, make sure Client_Name, EMMSERVER and the first Server entry are all listed as nbumaster.

3. In databases.conf file, NBDB database must be on the first line.
Reference: https://www.veritas.com/support/en_US/article.000126459

Good example:
Linux/UNIX:

# cat /usr/openv/var/global/databases.conf
"/usr/openv/db/data/NBDB.db" -n NBDB
"/usr/openv/db/data/NBAZDB.db" -n NBAZDB

Windows:

type C:\Program Files\Veritas\NetBackupDB\conf\databases.conf
"C:\Program Files\Veritas\NetBackupDB\data\NBDB.db" -n NBDB
"C:\Program Files\Veritas\NetBackupDB\data\NBAZDB.db" -n NBAZDB

4. The following processes must be running for NetBackup Java Administration Console authentication:

• nbatd
• nbwmc
• nbsl

You can run this command to verify:
Linux/UNIX: /usr/openv/netbackup/bin/bpps
Windows: install_path\NetBackup\bin\bpps

TIPS: If nbsl is not running, try (re)starting it by following: https://www.veritas.com/support/en_US/article.100033680

If step 1-4 are good and nbatd and nbwmc are not running, I suggest to contact NetBackup Technical Support as it may require more in-depth troubleshooting.

5. If the 3 processes above are running and you are still getting the same error, check whether the Tomcat certificate has expired.

A) First, enable NBCURL logging.
Linux/UNIX: Add below line in /usr/openv/netbackup/bp.conf
ENABLE_NBCURL_VERBOSE = 1

Windows: Add a DWORD (32-bit) key in HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config, called: ENABLE_NBCURL_VERBOSE
Double click the key and put a value of: 1

B) Second, verify the Tomcat certificate.
Linux/UNIX: /usr/openv/netbackup/bin/nbcertcmd -ping
Windows: install_path\NetBackup\bin\nbcertcmd -ping

If the Tomcat certificate has expired, you will see an entry similar to this:

* Server certificate:
* subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
* start date: 2017-01-31 21:59:12 GMT
* expire date: 2018-01-31 23:14:12 GMT
* issuer: CN=broker; OU=root@nbumaster; O=vx
* SSL certificate verify result: certificate has expired (10), continuing anyway.

If you try below command, it will fail as well:
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getcertificate -force
nbcertcmd: The -getCertificate operation failed for server winref.
EXIT STATUS 8506: The certificate has expired.

C) In that case, we need to renew the Tomcat certificate. Below steps are the same as this technote, with additional precautionary measure.

Linux:

• Stop NetBackup services first and make a copy of the following directory: /usr/openv/var
• Start NetBackup services again.
• Run: export WEBSVC_PASSWORD=web_service_user_password
• Run: /usr/openv/netbackup/bin/admincmd/nbcertconfig -t -user netbackup_web_service_user
• And: /usr/openv/wmc/bin/install/configureCerts

Windows:

• • Stop NetBackup services first and make a copy of the following directory: install_path\NetBackup\var\
  • Start NetBackup services again.
  • Run: set WEBSVC_PASSWORD=web_service_user_password
  • And: install_path\NetBackup\bin\admincmd\nbcertconfig -t -user netbackup_web_service_user
  • And: install_path\NetBackup\wmc\bin\install\configureCerts.bat
  • Note: netbackup_web_service_user is usually nbwebsvc.

D) Verify the Tomcat certificate again.
Linux/UNIX: /usr/openv/netbackup/bin/nbcertcmd -ping
Windows: install_path\NetBackup\bin\nbcertcmd -ping

The Tomcat certificate should be good now. For example:

* Server certificate:
* subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
* start date: 2018-03-06 22:52:27 GMT
* expire date: 2019-03-07 00:07:27 GMT
* issuer: CN=broker; OU=root@nbumaster; O=vx
* SSL certificate verify ok.

E) At this point, you can retrieve/update the rest of the certificates as required by NetBackup.

Linux/UNIX, run:

/usr/openv/netbackup/bin/nbcertcmd -getCaCertificate
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
/usr/openv/netbackup/bin/admincmd/bpnbaz -ProvisionCert

Windows, run:

install_path\NetBackup\bin\nbcertcmd -getCaCertificate
install_path\NetBackup\bin\nbcertcmd -getCertificate -force
install_path\NetBackup\bin\admincmd\bpnbaz -ProvisionCert

F) Try logging in to the NetBackup Java Administration Console again.

If issue persists, I strongly recommend to contact NetBackup Technical Support.