Summary/Purpose

The purpose of this article is to explain a few instances of how an “Object Server Not Found Error” may occur and to show how to modify your Veeam database to fix your jobs in the event that:

• A vSphere server has crashed beyond a recoverable state
• A server has been removed and added back into the inventory in vSphere

Overview

In have a handful of occasions where I have had Backups and Replicas that were running successfully, stop working. In 3 separate instances on a VMware environment, I have experienced an error in Veeam denoting that “Task failed Error: Object not found.”

In all 3 instances, the backup files (.vbk and .vrb files) were still usable in Veeam, i.e. components such as the Restore Guest File (Windows), however every time I tried to run the backup or replica job, the job would immediately fail. This article explains how to fix the backup job to function correctly in the following scenarios:

1. Migrated a server to a new host (ESX/ESXi)
2. Loss of VM ID via loss of VMware vSphere Server (root of the 2nd and 3rd issue are the same)
3. Removing a server from your inventory then adding it back into vCenter. (root of the 2nd and 3rd issue are the same)

Veeam Job Configured with ESX/ESXi 

When you initially create your job(s), you want to make sure that you add all your servers from the vSphere Server in the ADD OBJECTS screen. 

If you have multiple ESX/ESXi hosts, this is important because if you ever migrate to another host, your Veeam backup will still try to point to the original mapping on the ESX/ESXi host. Thus, the server will appear unavailable to Veeam. However, if you have a vSphere server, the path will be mapped via the vSphere server.

Virtual Machine ID was Changed

It is important to note that VMware assigns a unique ID to each VM. If a vSphere server is lost or if a VM is removed and added back to the inventory, the ID number of the VM will change. Although you can use Veeam to create a backup a vSphere server, I have seen instances where the backups did not span back far enough and the server could not be recovered. In addtion to Veeam’s backups, I would suggest that VMware’s recommended backup for a vSphere server be considered, which can be found here: http://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-1-overview.html.

How to Lose/Reassign your VM ID

The most common ways I have seen this occur are:

• Losing the vSphere server and rebuilding from scratch. This will cause all the VM-ID’s to change.
• Someone with administrative access to vSphere removes a VM in the vSphere inventory. At a later date, that virutal machine is added back into the inventory.

The Fix: Modifying Your Veeam Database

STEP 1: Identify which VM’s are failing and in what Veeam job(s).

This can be done by looking at the job logs.

STEP 2: Open up PowerCLI on the vSphere server.

Once you have identified the VM’s, open up vSphere PowerCLI.

Type in connect-viserver

STEP 3: Pull all the ID’s associated with your VM’s.

Next, you’ll want to run the command get-vm | select name, Id

Note: If you have hundreds of VM’s in your library, you can individually pull the problem VM instead of the list of every VM in your inventory.

In the example I have shown below, the name of the problem server is Legoland. Note that the ID is vm-5583.

STEP 4: Pulling the Old ID’s from your Existing Veeam Job

In the example below, I used Microsoft SQL Server Management Studio Express (this is free and can be downloaded at http://www.microsoft.com/download/en/details.aspx?id=8961) to open up my Veeam database.

I created a new SQL query that will pull all the object_id’s (this will be the equivalent of the vm-id) from an existing job.

SELECT bo.object_name, bo.object_id, bo.path
FROM bjobs bj
     INNER JOIN ObjectsInJobs oij ON bj.id = oij.job_id
     INNER JOIN BObjects bo ON bo.id = oij.object_id
WHERE bj.Name = ‘‘ and bj.type=0

 

This will pull a list of servers in a job of my choosing. In my example, the old ID for Legoland is listed as vm-228 in Veeam’s database.

STEP 5: Adjust the ID’s in your Veeam Job

Here I will show how to adjust the old ID, vm-228, to the new ID pulled from STEP 3, vm-5583.

CAUTION: Be VERY careful when executing the next command. Syntax is VERY important. If you do not feel confident in your ability to execute queries, chat with a database administrator or someone that has knowledge on SQL queries.

You will want to make a new query or execute the following:

UPDATE bobjects
SET [object_id] = ‘new-id’
WHERE [object_id] = ‘old-id’

With my example, this will be:

STEP 7: Verifying the ID Changed Successfully

Next, you can verify that the new ID is in place by executing the following command again:

SELECT bo.object_name, bo.object_id, bo.path
FROM bjobs bj
     INNER JOIN ObjectsInJobs oij ON bj.id = oij.job_id
     INNER JOIN BObjects bo ON bo.id = oij.object_id
WHERE bj.Name = ‘‘ and bj.type=0

Now adjust all the VM’s your having issues with and your Veeam Backup job will start working again.

 

Edit: As noted in one of the comments below, CBT has been known to stop working after applying this fix. See the following Veeam KB article http://www.veeam.com/KB1113 to fix CBT.

 

If you would like to learn more about Veeam Backup & Replication, visit http://www.veeam.com/vmware-esx-backup/vmware-esx-esxi-vsphere-vcenter.html.