How to consolidate a snapshot in VMware ESX when there are virtual machine errors

This is a compiled article from YouTube & VMWare KB, I thought of doing it when i ran into problems with Snapshots.

Videos are great !! and the KB's are so many. I was a bit lost and was required to compile my own material.

There are series of command being shown in this video which i am going to write in my blogs in steps

( KB1007849 and KB1004232 )

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1007849&sliceId=2&docTypeID=DT_KB_1_1&dialogID=262958161&stateId=0%200%20262960208

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1004232&sliceId=1&docTypeID=DT_KB_1_1&dialogID=262958148&stateId=0%200%20262960165

THIS ARTICLE IS APPLICABLE FOR FOLLOWING SCENARIOS

• Virtual machines with snapshots or delta disks stop responding or fail during a storage migration.
• When powering on a virtual machine, you see this error message:
  
  The parent virtual disk has been modified since the child was created.
  
  
• A virtual machine unexpectedly powers off and you see this error when you attempt to power it back on:
  
  A general system error occurred: internal error.
  
  
• Virtual machine log contains the entry:
  
  vcpu-0| Msg_Question: msg.hbacommon.outofspace reply=0
vcpu-0| Msg_Question:
vcpu-0| msg.hbacommon.outofspace There is no more space for the redo log of W2003Sbs1-000001.vmdk.
vcpu-0| You may be able to continue this session by freeing disk space on
vcpu-0| the relevant partition, and clicking Retry. Otherwise,
vcpu-0| click Abort to terminate this session.
  
  
• You are unable to remove snapshots for virtual machines on one or more datastores.

This error can occur if:

• The base disk of the virtual machine has been modified after a virtual machine snapshot or delta disk has been created.
• You have run out of space on a datastore containing the snapshot or delta disk.
• The guest operating system suffered an exception or kernel panic when a virtual machine snapshot was taken.

Overview

A VMware ESX virtual machine disk consists of a descriptor (ends in .vmdk) and an extent (ends in -flat.vmdk). When a virtual machine has been snapshotted, the attached virtual hard disks (otherwise known as the base disks) no longer receive guest OS modifications or writes; they are stored in separate virtual machine delta disk files (ending in -delta.vmdk). There is one delta file per base disk that is snapshotted.

With exception to thin-provisioned virtual machine disks, virtual machine disks typically have a reserved or set space requirement on a datastore. Snapshots, however, require additional space and consideration. They increase in size as further modifications or writes are stored.

For example:

1. If 10GB of changes were completed on one snapshotted virtual machine, its delta disk file size will increase proportionally by 10GB.
2. Creating another snapshot causes the existing snapshot delta disk to retain its current size, however the next delta disk will begin to store all forthcoming modifications and writes. The base disk is still left unmodified since inception of the first snapshot.
3. If another 15GB of changes were performed by the guest operating system, a total of 25GB of snapshot delta has now been recorded over the respective virtual machine's two snapshot delta disk files.
4. For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove All Snapshots task) causes unique changes stored only in the second snapshot delta disk to be copied upwards through the snapshot chain and into the first snapshot, or its "parent."
5. This effect is recursive for each preceding parent file. As a result, the first snapshot delta disk file will grow by up to 15GB, accommodating all new blocks. Any common changes stored in both snapshot levels does not require additional space, however.
6. The end result is a datastore requiring 40GB, or 25GB + 15GB.

Notes:

• Additional space is required if the virtual machine is running during a Remove All Snapshots task. The amount of space consumed is dependent the amount of snapshot delta information the virtual machine has to temporarily store while its consolidation process completes.
• If the same 10GB were changed in the second snapshot, and just 5GB of unique block changes occurred, then the first snapshot will increase by 5GB during the consolidation, not the complete 15GB.
• Opting to save virtual machine memory contents with each Create Snapshot task requires additional space from the VMFS datastore. The amount of space required is equivalent to the amount of memory assigned to the virtual machine. This requirement can apply multiple times over, if multiple snapshots including virtual machine memory contents are created. 

If a datastore fills up due to delta files accumulating too much space, or the datastore runs out of space during a snapshot consolidation effort, the symptoms outlined previously will occur. Migration or removal of stored data and snapshot consolidation will be required to resolve the issue.

1.     SSH to your ESXi machine

2.     vmware-cmd -l      ( to list virtual machines in this host)

3.     cd <virtual machine directory name>       ( to change to the vm directory )

4.     ls -ltrh   ( to view the contents in this directory )

a)     .VMX is the vm's own configuration file

b)     xxxxx-flat.vmdk is the base disk

c)     each xxxxx-flat vmdk file has a smaller .vmdk descriptor file

d)     xxxxx-0000??-delta.vmdk are snaphot files

e)     each xxxxx-0000??-delta.vmdk file has smaller .vmdk descriptor file

5.  Find out all the disks associated with the virtual machine. The disk maybe located on different datastore

            grep –i filename myVM.VMX | grep –i vmdk

this command will show the current disk in use

6. Find out the associated VM’s and the space which they are taking

      ls –lah *.vmdk

this should show you the myVM-flat.vmdk  base disk and the associated –delta.vmdk files

7.  Ensure the virtual machine is powered off. You can check by the following command.

        vmware-cmd  myVM.vmx getstate

8.  Check that there is enough free space on the datastore.

       vdf –h

            make sure that there is enough space on the drive.

     9.   Verify that the snapshot chain is intact

CID is a hexadecimal number identifying the disk.

           grep –i filename myVM.VMX | grep –i vmdk

            (get the current snapshot filename)

10.  you have to now verify manually if the current snapshot files CID matches parent CID

         less myVM-000003.vmdk | egrep –i “cid|parentcid|vmdk” –color

verify that myVM-000003.vmdk is referencing myVM-000002.vmdk and if all okay then goto the next step.

         less myVM-000002.vmdk | egrep –i “cid|parentcid|vmdk” –color

verify that myVM-000002.vmdk is referencing myVM-000001.vmdk and if all okay then goto the next step.

         less myVM-000001.vmdk | egrep –i “cid|parentcid|vmdk” –color

this should reference the base disk myVM.VMDK

         less myVM.VMDK  | egrep –i “cid|parentcid|vmdk” –color

this is to the check the CID of the base disk.

If the CID of the base disk does not match with the parent CID in the myVM-000001.vmdk than we have to manually edit the descriptor file of myVM-000001.vmdk to update the parent CID value to match

Vi myVM-000001.vmdk

Change the parentCID value to match the base disk value and save the file.

Verify that myVM-000001.vmdk file has the updated reference for the base disk.

         less myVM-000001.vmdk | egrep –i “cid|parentcid|vmdk” –color

Once we have verified the SNAPSHOT chain we will use VMKFSTOOL –i  tool to copy out and consolidate the snapshots all in one step.

The clone operation may take an extended amount of time depending on the number and size of the snapshot delta disks.

Create a new directory for the new consolidated VM.

Mkdir  /vmfs/volumes/NewDatastore/recover

Pwd    (get current directory name)

Vmkfstools  –i myVM-000003.vmdk  /vmfs/volumes/NewDatastore/recover/myVM_new.vmdk

Here we are committing all snapshots in the chain into new disk, myVM_new.vmdk

This will start the clone process.

If the vmkfstools  command fails then

a)     This could be the reason of corrupt snapshot or

b)     You maybe running out of diskspace.

c)     Run the vmkfstools command with the next higher snapshot up in the chain.

Now at this point you have cloned the original vm disks into new vm into a new datastore. Now we have to point the existing VM to point to the new VM and new datastore.

1)     Go to ESX host

2)     Detach the existing Hard Disk from VM (don’t delete it)

3)     Attach the newly created disk myVM_new.VMDK which has all snapshots consolidated into the base disk.

4)     Browse to the new datastore and select the new VMDK file.

5)     Power on the virtual machine

Once the virtual machine powers up okay, we can go back to clear space used by the original disks and snapshot files.

     *** Thanks To Vmware For Showing Excellent Video ***
** Golden IT Rule : take good backups before doing anything **