Category: NetBackup Appliance

[Error] V-58000-500-0003: Unable to open network share. An internal error occurred. Restarting the appliance may solve this issue.

There will be times when you need to transfer files in and out of NetBackup Appliance server, for instance, to install hotfixes, transfer log files for troubleshooting, copy script files and so on.

NetBackup Appliance CLISH has a convenient way to open network share so you can transfer files from and to your Windows PC (or Linux/UNIX if that’s what you have). The steps are mentioned in these technotes:
https://www.veritas.com/support/en_US/article.100023444
https://www.veritas.com/support/en_US/article.100010965

Recently I encountered an issue where opening Logs share returned the following error:

[Error] V-58000-500-0003: Unable to open network share. An internal error occurred. Restarting the appliance may solve this issue.

At that time, I narrowed down the issue to invalid entries in /etc/exports file.

nbu-appliance:/etc # cat exports

#default exports establish by NetBackup
#/inst/client *(ro,async,no_subtree_check)
#default export for logs
#/log *(ro,async,no_subtree_check)
#default export for incoming_patches
#/inst/patch/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for incoming_plugins
#/inst/plugin/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for LiveUp
#/inst/patch/client *(ro,async,no_subtree_check)
/log *(ro,async,no_subtree_check)
/inst/patch/incoming *(all_squash,rw,sync,no_subtree_check)
#default export for logforwarding
#/inst/logforwarding *(all_squash,rw,sync,no_subtree_check)

Notice the duplicate entries in bold red. After removing them and running exportfs -ra (so the /etc/exports is re-read), I was able to open the Logs share.

MSDP performance issue after upgrading to NetBackup 8.1 / Appliance 3.1

A lot of NetBackup customers would have upgraded to version 8.1 (3.1 if appliance) or later by now. One of the hottest topic in my team at the moment is how some customers encounter MSDP performance issue soon after upgrading to this version.

The stronger fingerprint algorithm in NetBackup 8.1’s MSDP is definitely good in the long run, but for those who recently upgraded from older NetBackup, the algorithm conversion from MD5 to SHA-2 can be a pain point – especially for the first few days post upgrade.

The issue may manifest in different ways: NetBackup Deduplication Engine (spoold) crashing, replication and/or duplication hanging and general slowness for MSDP-related operations.

If you think you are affected, download and install EEB 3935219 attached to technote: https://www.veritas.com/support/en_US/article.100041057

Note that if you already have EEB 3935219 installed, check the revision. The above technote contains the latest revision which is 11. Anything lower than 11 should be uninstalled and replaced by revision 11.

[9 May 2018] Update:

Some users may still notice performance issue even after data conversion has finished. If you are one of them, first, try lowering the amount of memory used by NetBackup deduplication engine (spoold) down from the default value of 75% to 50%.

Second, tune the OS kernel a little bit.

The steps are:

1. Open /msdp/data/dp1/pdvol/etc/puredisk/contentrouter.cfg file and search for MaxCacheSize. If the value is not 50%, set it to 50%. Save the file and restart NetBackup services. In theory you just need to restart spoold, but if you can afford to restart all NetBackup services, you will have a cleaner start.

2. Edit /etc/sysctl.conf file and ensure the 3 parameters below are set to the values given here (if these parameters don’t exist, just create them).

vm.overcommit_ratio = 100
kernel.numa_balancing = 0

Save the file and run sysctl -p to make the change effective.

User is unable to log in to NetBackup Java Administration Console after upgrading to NetBackup 8.0

After upgrading NetBackup Master Server to version 8.0, you may encounter the following error when logging in to the Java Administration Console:

Could not connect to NetBackup Service Layer. You may not be able to perform the functions in the Administration Console that depends on connectivity to this service. Please ensure the nbsl service is up and running.

There can be various reasons why it happens. I have compiled a list of things to look at based on past experience. Hopefully one of them can resolve your issue.

Checklist:

1. NetBackup EMM database must be running. Verify by running below command:
Linux/UNIX: /usr/openv/db/bin/nbdb_ping
Windows: install_path\Veritas\NetBackup\bin\nbdb_ping

2. Master server’s Client_Name, EMMSERVER and the first entry in Server list must be identical with the host name mentioned in NetBackup’s CA certificate.

Linux/UNIX:
* Get the CA cert detail by running:
/usr/openv/netbackup/bin/nbcertcmd -listcacertdetails
* Compare the hostname with the above parameters stored in /usr/openv/netbackup/bp.conf

Windows:
* Get the CA cert detail by running:
install_path\NetBackup\bin\nbcertcmd -listcacertdetails
* Launch registry editor and go to HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config.

For example:

C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -listcacertdetails
Subject Name : /CN=nbatd/OU=root@nbumaster /O=vx
Start Date : Jun 30 13:55:39 2015 GMT
Expiry Date : Jun 25 15:10:39 2035 GMT
SHA1 Fingerprint : C4:81:6C:B6:66:25:C1:DA:82:E3:06:F3:23:26:4F:51:85:3B:B8:71

In this case, make sure Client_Name, EMMSERVER and the first Server entry are all listed as nbumaster.

3. In databases.conf file, NBDB database must be on the first line.
Reference: https://www.veritas.com/support/en_US/article.000126459

Good example:
Linux/UNIX:

# cat /usr/openv/var/global/databases.conf
"/usr/openv/db/data/NBDB.db" -n NBDB
"/usr/openv/db/data/NBAZDB.db" -n NBAZDB

Windows:

type C:\Program Files\Veritas\NetBackupDB\conf\databases.conf
"C:\Program Files\Veritas\NetBackupDB\data\NBDB.db" -n NBDB
"C:\Program Files\Veritas\NetBackupDB\data\NBAZDB.db" -n NBAZDB

4. The following processes must be running for NetBackup Java Administration Console authentication:

  • nbatd
  • nbwmc
  • nbsl

You can run this command to verify:
Linux/UNIX: /usr/openv/netbackup/bin/bpps
Windows: install_path\NetBackup\bin\bpps

TIPS: If nbsl is not running, try (re)starting it by following: https://www.veritas.com/support/en_US/article.100033680

If step 1-4 are good and nbatd and nbwmc are not running, I suggest to contact NetBackup Technical Support as it may require more in-depth troubleshooting.

5. If the 3 processes above are running and you are still getting the same error, check whether the Tomcat certificate has expired.

A) First, enable NBCURL logging.
Linux/UNIX: Add below line in /usr/openv/netbackup/bp.conf
ENABLE_NBCURL_VERBOSE = 1

Windows: Add a DWORD (32-bit) key in HKEY_LOCAL_MACHINE \ SOFTWARE \ Veritas \ NetBackup \ CurrentVersion \ Config, called: ENABLE_NBCURL_VERBOSE
Double click the key and put a value of: 1

B) Second, verify the Tomcat certificate.
Linux/UNIX: /usr/openv/netbackup/bin/nbcertcmd -ping
Windows: install_path\NetBackup\bin\nbcertcmd -ping

If the Tomcat certificate has expired, you will see an entry similar to this:

* Server certificate:
* subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
* start date: 2017-01-31 21:59:12 GMT
* expire date: 2018-01-31 23:14:12 GMT
* issuer: CN=broker; OU=root@nbumaster; O=vx
* SSL certificate verify result: certificate has expired (10), continuing anyway.

If you try below command, it will fail as well:
C:\Program Files\Veritas\NetBackup\bin>nbcertcmd -getcertificate -force
nbcertcmd: The -getCertificate operation failed for server winref.
EXIT STATUS 8506: The certificate has expired.

C) In that case, we need to renew the Tomcat certificate. Below steps are the same as this technote, with additional precautionary measure.

Linux:

  • Stop NetBackup services first and make a copy of the following directory: /usr/openv/var
  • Start NetBackup services again.
  • Run: export WEBSVC_PASSWORD=web_service_user_password
  • Run: /usr/openv/netbackup/bin/admincmd/nbcertconfig -t -user netbackup_web_service_user
  • And: /usr/openv/wmc/bin/install/configureCerts

Windows:

    • Stop NetBackup services first and make a copy of the following directory: install_path\NetBackup\var\
    • Start NetBackup services again.
    • Run: set WEBSVC_PASSWORD=web_service_user_password
    • And: install_path\NetBackup\bin\admincmd\nbcertconfig -t -user netbackup_web_service_user
    • And: install_path\NetBackup\wmc\bin\install\configureCerts.bat
    • Note: netbackup_web_service_user is usually nbwebsvc.

D) Verify the Tomcat certificate again.
Linux/UNIX: /usr/openv/netbackup/bin/nbcertcmd -ping
Windows: install_path\NetBackup\bin\nbcertcmd -ping

The Tomcat certificate should be good now. For example:

* Server certificate:
* subject: CN=nbumaster; OU=TOMCAT@nbumaster; O=vx
* start date: 2018-03-06 22:52:27 GMT
* expire date: 2019-03-07 00:07:27 GMT
* issuer: CN=broker; OU=root@nbumaster; O=vx
* SSL certificate verify ok.

E) At this point, you can retrieve/update the rest of the certificates as required by NetBackup.

Linux/UNIX, run:

/usr/openv/netbackup/bin/nbcertcmd -getCaCertificate
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
/usr/openv/netbackup/bin/admincmd/bpnbaz -ProvisionCert

Windows, run:

install_path\NetBackup\bin\nbcertcmd -getCaCertificate
install_path\NetBackup\bin\nbcertcmd -getCertificate -force
install_path\NetBackup\bin\admincmd\bpnbaz -ProvisionCert

F) Try logging in to the NetBackup Java Administration Console again.

If issue persists, I strongly recommend to contact NetBackup Technical Support.

nbcertcmd -createtoken fails with EXIT STATUS 8000: User does not have permission(s) to perform the requested operation

I wanted to deploy a NetBackup 8.1 client the other day, and one of the steps involved entering a token from master server. My master server was a NetBackup Appliance running software version 3.1.

As per the procedure, I went to the master server and logged in to the NetBackup Web Management Console first:

bpnbat -login -logintype WEB

Afterwards, I ran the following command to generate the token:

nbcertcmd -createtoken -tokenname token_name

But it failed with: EXIT STATUS 8000: User does not have permission(s) to perform the requested operation, despite using “admin” account which had root privileges.

My colleague Hoai (who is a brilliant troubleshooter, by the way) suggested to create a NetBackupCLI account:

  • Go back to CLISH, and navigate to Main_Menu > Manage > NetBackupCLI
  • Run: Create myUser
  • You can change myUser to your user name.
  • Enter the password for your user twice.
  • Once created, log out and log back into the Appliance using the above account.
  • Re-run bpnbat and nbcertcmd commands, they should work.

Clean way to reconfigure Fiber Transport Media Server (FTMS) on NetBackup Appliance 52xx and 53xx

NOTE: Please refer to my other post for FTMS on a BYO NetBackup Media Server.

While you can theoretically run the same FTMS commands on a NetBackup Appliance as its BYO siblings, you need to be careful with the former because it contains additional monitoring databases, scripts and reports.

The rule of thumb is if you can use CLISH to perform a task, utilize it instead of doing it manually through command line. It ensures the databases, scripts and reports stay in sync. This is especially true for FTMS configuration.

If you accidentally ran commands on your NetBackup Appliance that broke its FTMS configuration, you can follow below steps to reset it again.

NOTE: These steps will require reboots, so please allocate a maintenance window first.

1. Unplug all fiber channel cables from your Appliance. However, confirm beforehand that you only use the connections for FTMS, and not for any other purpose like disk array. If you have disk array connected, arrange with your SAN Admin to gracefully disconnect/unmount the disk array. If you have tape libraries attached, as long as no backups run then they are not used.

Don’t unplug the SAS cables that connect to the disk shelf if you have one.

2. Go to elevated prompt and run the following script to reset the FTMS setting (don’t forget the number 4 at the end):
sh /opt/NBUAppliance/scripts/fcr/clear_san.sh 4

** The appliance will reboot automatically once this step is complete **

3. Configure the SAN client again.
Log in to Appliance CLISH and go to Settings.

Run: FibreTransport SANClient Enable 4

** Once the wizard is completed, you will be prompted to reboot again **

5. After appliance is back, verify the FTM setting. Log back in to the CLISH and go to Settings.

Run: FibreTransport SANClient Show

Check the status. It should show: [Info] Fibre Transport Sever enabled.

Then go to Manage > FibreChannel

Run: Show

Check if there is any error. If all look good, you have your FTMS back. If not, it is best to contact NetBackup Technical Support.