Determining the file at a specific VMDK offset

| September 12, 2013 | Programming | 1 Comment

So whilst writing up that previous blog post, I started getting I/O errors on one of my virtual machines. This tends to start happening more as we move into Summer, since Brisbane is bloody hot at times, which sometimes causes my spinning plates of rust to get a case of the vapours.

At any rate, I did have backups of the virtual disk (.vmdk) files, and was considering rolling back to the previous backup, but since

  • the backup was from a week ago, and
  • the errors were limited to only a 4k region within a 60GB disk,

I thought it’d be easier just to make a copy of the existing dodgy disk and just pull whatever file(s) were affected from backup.

This blog post describes determining a file contained within an ext4 partition at a specific VMDK offset. For a description of recovering data from an NTFS partition, see the following post instead.
Edit: Got another dodgy sector, so I’m going to try this again using the offline method. Which you probably should try first.

Online method

The online method involves booting into the virtual machine itself and performing recovery operations from there. If this isn’t possible, you might want to look at the offline method below.

So this is what the bnedev03.vmdk file looks like, as viewed from the hypervisor containing it (called bnehyp03). As you can see the data stored on the disk is split into 31 separate files, named bnedev03-s001.vmdk to bnedev03-s031.vmdk (the files referenced under the “Extent description” line):

knoxg@bnehyp03:/var/vmware/bnedev03$ cat bnedev03.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a0689571
parentCID=ffffffff
isNativeSnapshot="no"
createType="twoGbMaxExtentSparse"

# Extent description
RW 4192256 SPARSE "bnedev03-s001.vmdk"
RW 4192256 SPARSE "bnedev03-s002.vmdk"
RW 4192256 SPARSE "bnedev03-s003.vmdk"
RW 4192256 SPARSE "bnedev03-s004.vmdk"
RW 4192256 SPARSE "bnedev03-s005.vmdk"
RW 4192256 SPARSE "bnedev03-s006.vmdk"
RW 4192256 SPARSE "bnedev03-s007.vmdk"
RW 4192256 SPARSE "bnedev03-s008.vmdk"
RW 4192256 SPARSE "bnedev03-s009.vmdk"
RW 4192256 SPARSE "bnedev03-s010.vmdk"
RW 4192256 SPARSE "bnedev03-s011.vmdk"
RW 4192256 SPARSE "bnedev03-s012.vmdk"
RW 4192256 SPARSE "bnedev03-s013.vmdk"
RW 4192256 SPARSE "bnedev03-s014.vmdk"
RW 4192256 SPARSE "bnedev03-s015.vmdk"
RW 4192256 SPARSE "bnedev03-s016.vmdk"
RW 4192256 SPARSE "bnedev03-s017.vmdk"
RW 4192256 SPARSE "bnedev03-s018.vmdk"
RW 4192256 SPARSE "bnedev03-s019.vmdk"
RW 4192256 SPARSE "bnedev03-s020.vmdk"
RW 4192256 SPARSE "bnedev03-s021.vmdk"
RW 4192256 SPARSE "bnedev03-s022.vmdk"
RW 4192256 SPARSE "bnedev03-s023.vmdk"
RW 4192256 SPARSE "bnedev03-s024.vmdk"
RW 4192256 SPARSE "bnedev03-s025.vmdk"
RW 4192256 SPARSE "bnedev03-s026.vmdk"
RW 4192256 SPARSE "bnedev03-s027.vmdk"
RW 4192256 SPARSE "bnedev03-s028.vmdk"
RW 4192256 SPARSE "bnedev03-s029.vmdk"
RW 4192256 SPARSE "bnedev03-s030.vmdk"
RW 61440 SPARSE "bnedev03-s031.vmdk"

# The Disk Data Base
#DDB

ddb.toolsVersion = "8450"
ddb.adapterType = "lsilogic"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "7832"
ddb.uuid = "60 00 C2 9b de e9 2e 10-02 6e 8e 08 8f b7 bd 1b"
ddb.longContentID = "0343fb05b2103a6a5bb68d45a0689571"
ddb.virtualHWVersion = "8"

So first thing I did was make a copy of the entire virtual machine, which triggered the I/O error again on bnedev03-s029.vmdk. To attempt get as much of the file as possible, I used the dd command with the noerror parameter (the old VM has been moved to /var/vmware/x/bnedev03-corrupt, and the new replacement is being copied into /var/vmware/bnedev03):

knoxg@bnehyp03:/var/vmware/bnedev03$ dd if=../x/bnedev03-corrupt/bnedev03-s029.vmdk of=./bnedev03-s029.vmdk bs=512 conv=noerror,sync
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+0 records in
834800+0 records out
427417600 bytes (427 MB) copied, 18.8726 s, 22.6 MB/s

dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+1 records in
834801+0 records out
427418112 bytes (427 MB) copied, 36.2722 s, 11.8 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+2 records in
834802+0 records out
427418624 bytes (427 MB) copied, 53.5822 s, 8.0 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+3 records in
834803+0 records out
427419136 bytes (427 MB) copied, 70.8422 s, 6.0 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+4 records in
834804+0 records out
427419648 bytes (427 MB) copied, 88.3722 s, 4.8 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+5 records in
834805+0 records out
427420160 bytes (427 MB) copied, 105.682 s, 4.0 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+6 records in
834806+0 records out
427420672 bytes (427 MB) copied, 123.062 s, 3.5 MB/s
dd: reading `../x/bnedev03-corrupt/bnedev03-s029.vmdk': Input/output error
834800+7 records in
834807+0 records out
427421184 bytes (427 MB) copied, 141.662 s, 3.0 MB/s
4146552+8 records in
4146560+0 records out
2123038720 bytes (2.1 GB) copied, 259.626 s, 8.2 MB/s
knoxg@bnehyp03:/var/vmware/bnedev03$

Which copied fine except for the blocks 834800 to 834807 (where the block size has been specified on the command line as 512 bytes). This corresponds to byte locations 427417600 to 427421695 of bnedev03-s029.vmdk.

Each file comprising the virtual disk is described by lines with this format in the bnedev03.vmdk descriptor:

...
RW 4192256 SPARSE "bnedev03-s001.vmdk"
...

Each VMDK extent (except for the last one) is 4192256 sectors in length, where the sector size is fixed as 512 bytes, making each extent 2146435072 bytes long (around 2GB).

So relative to the first byte of the vmdk, this error is located at

  • the sum of all the extent sizes bnedev03-s001.vmdk to bnedev03-s028.vmdk, plus 427417600,
  • = 2146435072*28 + 427417600
  • = 60100182016 + 427417600
  • = byte offset 60527599616.

So what I want to do is determine what file is at byte position 60527599616. Luckily, when I rebooted the duplicated VM it started OK, and a filesystem check via fsck found no fault in the filesystem. The 4k that didn’t transfer properly is probably garbage though, so I should probably update or delete the file(s) that have been corrupted by the I/O error.

There is, conveniently, tools to do this. The xxd command can convert bytes into hexadecimal and can be supplied byte offset and dump sizes. So once logged into the virtual machine, I first tried checking this offset within /dev/sda on the VM, in chunks of 100, 1000 and 10000 bytes:

knoxg@bnedev03:~$ xxd -l 100 -s 60527599616 /dev/sda
xxd: /dev/sda: Permission denied
knoxg@bnedev03:~$ sudo xxd -l 100 -s 60527599616 /dev/sda
[sudo] password for knoxg: *******
e17b9e000dae9 ed5b 47a3 17b0 d80d 347a 7370 7011  ...[G.....4zspp.
e17b9e0108f5f 43a1 804c a068 658c 02c0 1804 fcbf  ._C..L.he.......
e17b9e020c5ff 3338 000a e0f6 0487 8684 8486 8487  ..38............
e17b9e0308547 8485 8784 8602 49e0 1f0b b38b 02c6  .G......I.......
e17b9e0400748 7838 242c 0c12 165e 1a1c 5211 1a8a  .Hx8$,...^..R...
e17b9e050ac2a 2723 7b87 d188 3102 7a04 8d18 8441  .*'#{...1.z....A
e17b9e060db33 338a                                .33.
knoxg@bnedev03:~$ sudo xxd -l 1000 -s 60527599000 /dev/sda | less
knoxg@bnedev03:~$ sudo xxd -l 10000 -s 60527590000 /dev/sda | less

this just looked like binary garbage, so instead I tried to find out the filename containing those bytes.

First of all I need to check what kind of filesystem is running on the partition containing this byte offset, which turns out is ext4:

knoxg@bnedev03:~$ sudo fdisk -l
[sudo] password for knoxg: ********

Disk /dev/sda: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00070330

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        7508    60300288   83  Linux
/dev/sda2            7508        7833     2611201    5  Extended
/dev/sda5            7508        7833     2611200   82  Linux swap / Solaris
knoxg@bnedev03:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              57G   41G   14G  76% /
...
knoxg@bnedev03:~$ mount -l
/dev/sda1 on / type ext4 (rw,noatime,errors=remount-ro)
...

then determine the block size of the file system (4096 bytes) and the starting offset of the partition (1048576 bytes)

knoxg@bnedev03:~$ sudo tune2fs -l /dev/sda1  | grep -i 'block size'
Block size:               4096
knoxg@bnedev03:~$ sudo udisks --show-info /dev/sda1 | grep -i 'offset'
    offset:                    1048576
    alignment offset:          0

Byte offset 60527599000 of the disk /dev/sda is therefore at byte offset 60527599000 – 1048576 = 60526550424 of the partition /dev/sda1

Byte offset 60526550424 of /dev/sda1 is at the file system block offset 60527598488 / 4096 = 14776989. Because this is an ext4 partition, I need to get the inode number containing block 14776989:

knoxg@bnedev03:~$ sudo debugfs -R "icheck 14776989" /dev/sda1
debugfs 1.41.11 (14-Mar-2010)
Block   Inode number
14776989        1583922

and now the filename for inode 1583922:

knoxg@bnedev03:~$ sudo debugfs -R "ncheck 1583922" /dev/sda1
debugfs 1.41.11 (14-Mar-2010)
Inode   Pathname
1583922 /opt/sonatype-work/nexus/storage/snapshots/com/randomnoun/appnav/appnav-web/0.0.4-SNAPSHOT/appnav-web-0.0.4-20130104.224635-2.war

it turns out that this file (appnav-web-0.0.4-20130104.224635-2.war) was a not terribly important snapshot of a web application from 5 months ago, so I ended up just deleting the whole directory rather than pulling the old file from backup.

So there you go.

Offline method

The steps above assume that the VM is bootable and functioning, which may not necessarily be the case. I got another dodgy sector on this VMDK a few months later, so here are some alternate steps if you want to detect corrupted data from outside the virtual machine.

I was alerted to the dodgy sector when I attempted to backup the VM:

knoxg@bnehyp03:/var/vmware$ sudo /bin/bash -c 'for X in bnedev01 bnedev02 bnedev03 ; do { echo $X ; cp -r $X /media/storage/vmware/$X-20140117 ; } ; done'
bnedev01
bnedev02
bnedev03
cp: reading `bnedev03/bnedev03-s010.vmdk': Input/output error

The error appears to be in the 10th VMDK extent of bnedev03.vmdk, so let’s find out the bytes affected. The commands below create a bnedev03-s010.clean file with the bad sectors zeroed out, and then swaps that file into the existing VMDK fileset.

knoxg@bnehyp03:/var/vmware/bnedev03$ dd if=bnedev03-s010.vmdk of=bnedev03-s010.clean bs=512 conv=noerror,sync
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+0 records in
1774728+0 records out
908660736 bytes (909 MB) copied, 47.6521 s, 19.1 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+1 records in
1774729+0 records out
908661248 bytes (909 MB) copied, 65.1016 s, 14.0 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+2 records in
1774730+0 records out
908661760 bytes (909 MB) copied, 82.5716 s, 11.0 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+3 records in
1774731+0 records out
908662272 bytes (909 MB) copied, 100.042 s, 9.1 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+4 records in
1774732+0 records out
908662784 bytes (909 MB) copied, 117.372 s, 7.7 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+5 records in
1774733+0 records out
908663296 bytes (909 MB) copied, 134.752 s, 6.7 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+6 records in
1774734+0 records out
908663808 bytes (909 MB) copied, 152.032 s, 6.0 MB/s
dd: reading `bnedev03-s010.vmdk': Input/output error
1774728+7 records in
1774735+0 records out
908664320 bytes (909 MB) copied, 174.532 s, 5.2 MB/s
4135544+8 records in
4135552+0 records out
2117402624 bytes (2.1 GB) copied, 233.903 s, 9.1 MB/s
knoxg@bnehyp03:/var/vmware/bnedev03$ mv bnedev03-s010.vmdk /var/vmware/x/bnedev03-corrupt/bnedev03-s010.vmdk
knoxg@bnehyp03:/var/vmware/bnedev03$ sudo mv bnedev03-s010.clean bnedev03-s010.vmdk

I then mount sda of the cleaned VMDK so I can take a squiz at the partition table:

knoxg@bnehyp03:/var/vmware/bnedev03$ sudo vmware-mount -f bnedev03.vmdk /mnt/bnedev03-sda
Failed to open disk: The specified virtual disk needs repair (60129558150)
Failed to mount disk 'bnedev03.vmdk': Cannot open the virtual disk
knoxg@bnehyp03:/var/vmware/bnedev03$ sudo vmware-vdiskmanager -R bnedev03.vmdk
The virtual disk, 'bnedev03.vmdk', was corrupted and has been successfully repaired.
knoxg@bnehyp03:/var/vmware/bnedev03$ sudo vmware-mount -f bnedev03.vmdk /mnt/bnedev03-sda
knoxg@bnehyp03:/var/vmware/bnedev03$ fdisk -lu /mnt/bnedev03-sda/flat
You must set cylinders.
You can do this from the extra functions menu.

Disk /mnt/bnedev03-sda/flat: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00070330

                 Device Boot      Start         End      Blocks   Id  System
/mnt/bnedev03-sda/flat1   *        2048   120602623    60300288   83  Linux
Partition 1 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(7507, 42, 23)
/mnt/bnedev03-sda/flat2       120604670   125827071     2611201    5  Extended
Partition 2 has different physical/logical beginnings (non-Linux?):
     phys=(1023, 254, 63) logical=(7507, 74, 54)
Partition 2 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(7832, 95, 7)
/mnt/bnedev03-sda/flat5       120604672   125827071     2611200   82  Linux swap / Solaris

So sda1 (/mnt/bnedev03-sda/flat1) starts at sector offset 2048 of sda, which is byte offset 2048*512 = 1048576 of sda.

Let’s mount sda1 using a loopback device, and determine the block size:

knoxg@bnehyp03:/var/vmware/bnedev03$ sudo losetup -o 1048576 /dev/loop0 /mnt/bnedev03-sda/flat
knoxg@bnehyp03:/var/vmware/bnedev03$ sudo tune2fs -l /dev/loop0 | grep -i 'block size'
Block size:               4096

From the dd output, the bad block ranges are

  • dd blocks 1774728 to 1774735 and 4135552
  • = byte offset within bnedev03-s010.vmdk of (1774728 * 512) to (1774735 * 512 + 511) and (4135552 * 512) to (4135552 * 512 + 511)
  • = byte offset within bnedev03-s010.vmdk of 908660736 to 908664831 and 2117402624 to 2117403135
  • = vmdk offset bytes (908660736 + 9 * 4192256 * 512) to (908664831 + 9 * 4192256 * 512) and (2117402624 + 9 * 4192256 * 512) to (2117403135+ 9 * 4192256 * 512)
    (+ size of the bnedev03-s001.vmdk … bnedev03-s009.vmdk vmdk extents)
  • = vmdk offset bytes (908660736 + 19317915648) to (908664831 + 19317915648) and (908660736 + 19317915648) to (908664831 + 19317915648)
  • = vmdk offset bytes 20226576384 to 20226580479 and 21435318272 to 21435318783.
  • = partition (sda1) offset bytes (20226576384 – 1048576) to (20226580479 – 1048576) and (21435318272 – 1048576) to (21435318783 – 1048576).
    (since 1048576 is the starting byte of sda1 as determined by the fdisk output above)
  • = partition (sda1) offset bytes 20225527808 to 20225531903 and 21434269696 to 21434270207
  • = ext4 blocks (20225527808 / 4096) to (20225531903 / 4096) and (21434269696 / 4096) to (21434270207 / 4096)
    (since 4096 is the ‘Block Size’ in the output of tune2fs above)
  • = ext4 blocks 4937873 to 4937874 and 5232976

(The earlier section goes into more detail on how these offset unit changes are calculated)

The inodes for these ext4 blocks are

knoxg@bnehyp03:/var/vmware/bnedev03$ sudo debugfs -R "icheck 4937873 4937874 5232976" /dev/loop0
debugfs 1.41.11 (14-Mar-2010)
Block	Inode number
4937873	26395
4937874	26395
5232976	26418

And the files on these inodes are

knoxg@bnehyp03:/var/vmware/bnedev03$ sudo debugfs -R "ncheck 26395 26418" /dev/loop0
debugfs 1.41.11 (14-Mar-2010)
Inode	Pathname
26395	/var/lib/cvsd/repos/module1/foo.java,v
26418	/var/lib/cvsd/repos/module1/bar.c,v

and not forgetting to delete the loopback device unmount the vmdk

knoxg@bnehyp03:/var/vmware/bnedev03$ sudo losetup -d /dev/loop0
knoxg@bnehyp03:/var/vmware/bnedev03$ sudo vmware-mount -k /var/vmware/bnedev03/bnedev03.vmdk

So. The files listed above were in /var/lib/cvsd of the filesystem, so are part of my CVS repository source history. These are relatively important to me, so I’ve dredged those up from backup and replaced them. And there you go.

Update 13/9/2013: Fixed the partition offset calculation
Update 17/1/2014: Added the offline section

Tags:, ,

Related Posts

About The Author

knoxg

One Comment

Add a Comment

Your email address will not be published. Required fields are marked *