Event IDs and error messages

This issue occurs if Windows fails to create a VSS snapshot. To resolve this issue, do the following:

• Check UDSAgent.log
• Check disk space on protected volumes. 300MB may not be enough.
• Check Windows Event Logs for VSS related errors.
• vssadmin list writers may show writers in a bad state.

Usually these errors are accompanied by VSS errors reported in the logs such as:

VSS_E_VOLUME_NOT_SUPPORTED_BY_PROVIDER VSS_E_UNEXPECTED_PROVIDER_ERROR

First check if all the VSS writers are in a stable state by going to the command line and issuing the command as below.

# vssadmin list writers

Check output to confirm that all the writers are in a stable state.

Restart VSS service and check if the writers are stable. If not you may have to reboot the machine.

This issue occurs if there is insufficient disk space to process a snapshot.

1. Ensure the drive being backed up is not full.
2. Check if all the VSS writers are in a stable state From the Windows command line, run:
vssadmin list providers vssadmin list writers
3. If these services are not running, start them and re-run the job. If the writer's State is Not Stable, restart the VSS service. If the problem continues after restarting the service, reboot the host.

Sometimes the message appears when internal VSS errors occur.

Check the Windows Event Logs for VSS related errors. For errors related to VSS, search for related Microsoft patches. Additional VSS troubleshooting details can be found on Microsoft TechNet.

Microsoft recommends at least 320MB on devices specified for saving the created VSS snapshot, plus change data that is stored there.

Actifio recommends the shadow storage space be set to unbounded (unlimited) using these commands:

To change the storage area size in the Windows UI, refer to - Configuring Volume Shadow Copy on Windows Server 2008

Re-run the backup once the VSS state is stable and shadow storage is set to unbounded.

This issue occurs if the staging LUN is not visible to the UDSAgent on the application's host and the host is unable to detect the staging LUN from the backup/recovery appliance.

Actifio Connector could not see the staging LUN. This can be caused by a bad connection or by trouble on the LUN.

Verify that FC/iSCSI connectivity is good, then make sure it works by mapping the VDisk, partitioning it, formatting it, copying files to it, etc. The steps for partitioning and formatting are OS specific.

Jobs fail if a file that was modified in the source disk is copied to the staging disk, but the file is larger than the free space available in the staging disk.

Anti-virus programs or third party drivers may have applied file locks that cannot be overridden.

Check the UDSAgent.log to see which file could not be accessed. Attempt to find which process is locking the file using lsof on Unix/ Linux, or fltmc on Windows. Exclude the file from the antivirus or capture job and re-try the capture.

The current processes known to Microsoft are listed at: Allocated filter altitudes.

These errors are rarely found on Unix or Linux, but it is possible that a process such as database maintenance or patch install / update has created an exclusive lock on a file.

Install the latest Actifio Connector.

A file system limitation or inconsistency was detected by the host operating system.

Run the Windows Disk defragmenter on the staging disk.

Low I/O throughput from the hosts disks or transport medium, iSCSI or FC.

Ensure there are no I/O issues in the host's disks or transport medium. The transport medium will either be iSCSI or Fibre Channel depending on out of band configuration. Consult storage and network administrators as needed.

To resolve this, enable "Do not forcefully unload the user registry at user logoff", see User Profile Service functionality.

This is because the ODBC login for the database is failing. Fixing the ODBC login resolves the issue.

Check /act/logs/UDSAgent.log for details and contact Google support for the resolution for the issue.

Invalid username and password being parsed from the control file. On the source, review the UDSAgent.log to see if the source is configured with the correct username/password under Advanced Settings in the connector properties.

Actifio Connector failed to backup the archive log using RMAN archive backup commands. The likely causes for this failure are:

• Connector failed to establish connection to database
• The archive logs were purged by another application
• TNS Service name is configured incorrectly, causing backup command to be sent to a node where the staging disk isn't mounted

Search for ORA- or RMAN- errors in the RMAN log. This is the error received from Oracle. Use the preferred Oracle resource as these are not Backup and DR Service conditions, and hence cannot be resolved within Backup and DR Service.

• Actifio Connector logs: /var/act/log/UDSAgent.log
• Oracle RMAN logs: /var/act/log/********_rman.log

To reduce consumption of the snapshot pool, do the following:

• Move VMware VMs from a snapshot to a Direct-to-OnVault backup plan. Then expire all snapshots to release the space used by the staging disks and last snap. This only works for VMware VMs; other application types still use some snapshot pool space if protected by a Direct-to-OnVault policy.
• Reduce the number of snaps kept for an application by changing the policy template. Applications that have high change rates create larger snapshots, so this has the highest benefit for high change-rate applications. This does not necessarily lead to a different RPO, an OnVault images of each snap can be created before they are expired.
• Delete mounts, clones, and live-clones if they are not needed.

To immediately reduce VDisk consumption, do the following:

• Ensure expirations are enabled, both at the global and individual application level.
• Group databases from a single host together into a Consistency Group. For example, if a host has 9 databases, create one Consistency Group for all 9 databases, then protect that consistency group rather than the individual databases.
• Reduce the number of snapshots kept for an application by changing the policy template used by a backup plan.
• Delete unwanted mounts, clones, and live-clones images.
• Move VMware VMs from a snapshot to a Direct- OnVault backup plan. You need to expire all snapshots to release the VDisks used by the staging disks. This only lowers the VDisk count for VMware VMs, and still uses VDisks when protected by a direct-to-OnVault policy.
• Change VMware VMDKs that do not need to be protected to independent mode as these cannot be protected by VMware snapshots.

If this alert repeats daily but the appliance does not reach the maximum VDisks, then modify the policies as above to reduce the number of VDisks used, or increase the alert threshold. During a daily snapshot window the VDisk count can fluctuate while new VDisks are created for snapshots before the old VDisks are removed as a part of snapshot expirations. The daily fluctuations varies depending on the number of applications protected.

A heartbeat ping to monitored storage has failed due to hardware failure or network issue. Check the network to resolve the issue.

Review the backup plan best practices and optimize your policies. The are common causes for backup plan violations.

• Job scheduler is not enabled. See to enable the scheduler.
• The first jobs for new applications can often take a long time: Long job times can occur during the first snapshot or dedup job for an application. On-ramp settings can be used to prevent ingest jobs from locking up slots and locking out ingested applications. See Set priorities for the first new applications.
• Applications are inaccessible due to network issues.
• Policy windows are too small or job run times are too long: While you cannot control how long each job takes to run, you can control the schedule time for applications that are running. Jobs that run for many hours occupy job slots that could be used by other applications. Review the backup plan best practices and adjust polices accordingly.
• Replication process sends the data to a remote backup/recovery appliance. Ensure that the bandwidth & utilization of your replication link is not saturated.

To reduce consumption of the snapshot pool, do the following:

• Move VMware VMs from a snapshot to a Direct-to-OnVault backup plan. Then expire all snapshots to release the space used by the staging disks and last snap. This only works for VMware VMs; other application types still use some snapshot pool space if protected by a Direct-to-OnVault policy.
• Reduce the number of snaps kept for an application by changing the policy template. Applications that have high change rates create larger snapshots, so this has the highest benefit for high change-rate applications. This does not necessarily lead to a different RPO, an OnVault images of each snap can be created before they are expired.
• Delete mounts, clones, and live-clones if they are not needed.

Each backup/recovery appliance checks the remote appliance hourly for possible remote protection issues. The appliance communication fails due to the following issues:

• Network error (temporary or permanent). Temporary network error does not mean job to fail; jobs are retried, but the hourly check is not updated.
• Certificate error. To fix the certificate error, you need to re-exchange the certificate.

Review the backup plan best practices and optimize your policies. The are common causes for backup plan violations.

• Job scheduler is not enabled. See to enable the scheduler.
• The first jobs for new applications can often take a long time: Long job times can occur during the first snapshot or dedup job for an application. On-ramp settings can be used to prevent ingest jobs from locking up slots and locking out ingested applications. See Set priorities for the first new applications.
• Applications are inaccessible due to network issues.
• Policy windows are too small or job run times are too long: While you cannot control how long each job takes to run, you can control the schedule time for applications that are running. Jobs that run for many hours occupy job slots that could be used by other applications. Review the backup plan best practices and adjust polices accordingly.
• Replication process sends the data to a remote backup/recovery appliance. Ensure that the bandwidth & utilization of your replication link is not saturated.

Review the backup plan best practices and optimize your policies. The are common causes for backup plan violations.

• Job scheduler is not enabled. See to enable the scheduler.
• The first jobs for new applications can often take a long time: Long job times can occur during the first snapshot or dedup job for an application. On-ramp settings can be used to prevent ingest jobs from locking up slots and locking out ingested applications. See Set priorities for the first new applications.
• Applications are inaccessible due to network issues.
• Policy windows are too small or job run times are too long: While you cannot control how long each job takes to run, you can control the schedule time for applications that are running. Jobs that run for many hours occupy job slots that could be used by other applications. Review the backup plan best practices and adjust polices accordingly.
• Replication process sends the data to a remote backup/recovery appliance. Ensure that the bandwidth & utilization of your replication link is not saturated.

Adding a raw device mapping to a VM "stuns" the VM until ESX has had a chance to add the new resource. To find out why the raw device mapping could not be added, look at the ESX logs for the VM in question (vmware.log).

Refer to the VMware documentation and knowledge base for assistance on reviewing the logs for error messages. Also, review the VMware article for more information on collecting VMware logs.

This is a VMware issue; for additional information, refer to VMware KB article - 1015180.

Virtual machine quiesce issues are dependent on the OS type. Additional investigation, further VMware KBA searches or contact VMware support to resolve this issue.

This usually means that the SCSI device you are trying to add to the VM is already in use by another VM.

This issue occurs if the VM's CTK files are locked, unreadable, or are being committed. To fix this issue, remove and re-create these CTK files. Refer to the KB article - 2013520 for more information.

There are two options for formatting a VMware datastore: NFS and VMFS. With NFS, there are some limitations like not being able to do RDM (Raw Disk Mapping). This means that you cannot mount from the backup/recovery appliance to an NFS datastore. Refer to the following KB article - 1001856 for additional information.

The Actifio Connector stops responding between the appliance and a host with Backup and DR agent is installed.

1. Restart the UDSAgent Backup and DR agent service on the specified host.
2. Telnet to tcp port 5106 (UDSAgent communication port)
# telnet 5106
Expected output:
Trying 10.50.100.67...
Connected to dresx2.accu.local.
Escape character is '^]'.
Connection closed by foreign host.
3. Verify network connectivity between appliance and host doesn't drop. If the problem persists, network analysis will be required.

This occurs when an inconsistency is found between the source and target data. Contact Google support to get the resolution for this issue.

This issue occurs if the backup/recovery appliance is not configured with iSCSI connection to the target host.

Ensure that the network ports are open for iSCSI and the target host has discovered the backup/recovery appliances.

The backup/recovery appliance is unable to reach the ESX host over the network or resolve the ESX host name using DNS. Contact Google support to get the resolution for this issue.

This is an alert condition generated by Backup and DR Service and is caused by leftover delta files in the VM's datastore. Normally, the delta files are removed after Backup and DR snapshot is consolidated. In some instances, these can be left behind by the VMware consolidation, and Backup and DR begins failing jobs to prevent exacerbating the issue.

This issue is caused by VMware, refer to the knowledge base article - 1002310.

This happens when the ESX server cannot be reached by the controller, usually because of a physical connection or DNS problem. To fix this issue, do the following:

• Ensure port 902 is open between the backup/recovery appliance and the ESX host.
• Check the current DNS server and ensure it is current and valid.
• If the vCenter is virtualized, attempt a backup after migrating the vCenter to a different ESX host.
• Ensure SSL required is set to True on the ESX host in the advanced settings.

There are two possible solutions for this situation:

• If consolidation is required for some disks on VM, size is reported as zero. To fix this issue, creating and deleting a snapshot of the VM.
• See if the VMDK can be restored from a backup image.

This issue occurs when the remaining space on the datastore is less than the critical threshold. If more storage is not made available soon, then jobs start to fail when the remaining space is inadequate to store them.

For more information, refer to the VMware knowledge base article - 1003412.

To initiate backup, the Actifio Connector service must be reachable by the backup/recovery appliance. This issue occurs, when the required ports are not open, the incorrect host IP is configured, the Backup and DR agent service not running, or the host is out of physical resources. To fix this issue, do the following:

• Ensure that the port in use between the host, backup/recovery appliance, and Actifio Connector is open. By default, the Backup and DR agent uses port 5106 for bi-directional communication from the backup/recovery appliance. Make sure your firewall permits bi-directional communication through this port.
• Ensure that the correct IP is configured for the host Manage > Appliance > Configure Appliance Networking.
• Ensure that the Backup and DR agent service is running on the target host and restart, if necessary.
• On Windows, find the UDS Host Agent service in services.msc and click Restart.
• On Linux, run the command /etc/init.d/udsagent restart

GEN-DEBUG [4400] UDSAgent starting up ... GEN-INFO [4400] Locale is initialized to C GEN-WARN [4400] VdsServiceObject::initialize - LoadService for Vds failed with error 0x80080005 GEN-WARN [4400] initialize - Failed to initialize Microsoft Disk Management Services: Server execution failed [0x80080005] GEN-WARN [4400] Failed initializing VDSMgr, err = -1, exiting... GEN-INFO [4400] Couldn't connect to namespace: root\mscluster GEN-INFO [4400] This host is not part of cluster GEN-WARN [4400] Failed initializing connectors,exiting -1

During a mount job, the backup/recovery appliance is unable to connect to the OnVault pool. This issue can be occurred due to any of the following reasons.

• No bucket name is specified for the OnVault pool.
• Invalid credentials-access ID or access key not specified or wrong ID for the OnVault pool
• Invalid bucket in the OnVault pool
• General authentication issues for the OnVault pool.
• DNS server in clusters /etc/resolv.conf is either different or the forward and reverse DNS zones files are changed.

VM snapshot fails if the ESX server is unable to quiesce the virtual machine - either because of too much I/O, or because VMware tools cannot quiesce the application using VSS in time. Check the event logs on the host and check the VM's ESX log (vmware.log).

Crash-consistent snapshots and connector-based backups show this behavior less often. For more information, refer to the VMware knowledge base articles - 1018194 and 1007696.

This issue occurs if the VM's UUID is modified. To fix this issue, rediscover the VM and check if it was discovered as a new UUID. You can confirm this in the management console by comparing the UUID of the newly discovered VM and that of the previously discovered VM. If the UUIDs do not match, the VM might have been cloned.

You can also see this error, if a large number of Backup and DR managed VMs are removed from the vCenter.

An OnVault job needed to be retried. Possible issues could include: The Service Account being used has the wrong role. The Service Account does not have authority to write to the bucket. The Cloud Storage bucket no longer exists.

Backup of a SQL Server Instance found some databases were offline and could not be backed up. This commonly occurs when the database has been deleted on the server side, but is still included on the Backup/DR side. The error message contains the names of the offline databases that should be investigated.

Metadata writes to an OnVault bucket failed. Possible issues could include: The Service Account being used has the wrong role. The Service Account does not have authority to write to the bucket. The Cloud Storage bucket no longer exists.

This is is an internal event and can be ignored.

This is is an internal event and can be ignored.