USB Devices as VMFS Datastore in vSphere ESXi 6.5
In ESXi 6.5, there are some changes concerning devices connected with USB. The legacy drivers, including xhci, ehci-hcd, usb-uhci, and usb-storage have been replaced with a single USB driver named vmkusb. The new driver has some implications if you are trying to use USB devices like USB sticks or external hard disks as VMFS formatted datastore.
Some people have reported that they have issues with USB Datastores since ESXi 6.5. I’ve tried to reproduce and fix those problems. This post explains the changes in the new version and how to create VMFS 5 or VMFS6 formatted USB devices as datastore on your ESXi host.
Please remember that using USB Datastores is not supported by VMware so if you have problems, do not open service requests for that.
Device Naming Changes in ESXi 6.5
Which I noticed first was that the devices naming in 6.5 have been changed. In ESXi 6.0, all USB devices are recognized as mpx.vmhbaXX while in 6.5 some are listed as naa.XXX. Here is an example of a Samsung M.2 flash drive connected with a USB3 adapter:
# ESXi 6.0u2 -rw------- 1 root root 128035676160 Nov 27 15:57 naa.5000000000000001 # ESXi 6.5 -rw------- 1 root root 128035676160 Nov 27 18:16 mpx.vmhba36:C0:T0:L0
However, not all devices are affected. My 16GB flash drives shows up with the old mpx.vmhba naming on both versions:
# ESXi 6.0u2 -rw------- 1 root root 16022241280 Nov 27 16:02 mpx.vmhba34:C0:T0:L0 # ESXi 6.5 -rw------- 1 root root 16022241280 Nov 27 16:18 mpx.vmhba34:C0:T0:L0
The new naming does not have an impact on USB datastores, despite it might happen that you overlook the USB device when checking the /dev/disks/ directory. You can still partition and format USB devices with VMFS like in previous versions, just use the new name.
Existing Datastores are not mounted automatically
A common problem which I’ve heard multiple times, but only seen since ESXi 6.5, is that when you connect a flash drive that already contains a VMFS datastore, the datastore does not show up. You can see the device in /dev/disks but the datastore remains offline. The vmkernel.log hints the problem:
2016-11-27T10:25:05.885Z cpu0:65593)VMWARE SCSI Id: Id for vmhba34:C0:T0:L0 0x50 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x41 0x53 0x32 0x31 0x31 0x35 2016-11-27T10:25:05.885Z cpu0:65593)ScsiDeviceIO: 9120: Get VPD 80 Inquiry for device "naa.5000000000000001" from Plugin "NMP" failed. Not supported 2016-11-27T10:25:05.885Z cpu0:65593)ScsiDeviceIO: 9167: Get VPD 86 Inquiry for device "naa.5000000000000001" from Plugin "NMP" failed. Not supported 2016-11-27T10:25:05.887Z cpu0:65593)ScsiDeviceIO: 7851: Could not detect setting of QErr for device naa.5000000000000001. Error Not supported. 2016-11-27T10:25:05.891Z cpu0:65593)ScsiDeviceIO: 8365: Could not detect setting of sitpua for device naa.5000000000000001. Error Not supported. 2016-11-27T10:25:05.898Z cpu1:65562)ScsiDeviceIO: 2948: Cmd(0x439500b60000) 0x1a, CmdSN 0x238 from world 0 to dev "naa.5000000000000001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11136: Device naa.5000000000000001:1 detected to be a snapshot: 2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11143: queried disk ID: <type 2, len 14, lun 0, devType 0, scsi 0, h(id) 4847995147196820166> 2016-11-27T10:25:05.905Z cpu0:65593)LVM: 11150: on-disk disk ID: <type 0, len 11, lun 0, devType 0, scsi 0, h(id) 8386732072863042940> 2016-11-27T10:25:05.914Z cpu0:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x43029e372700, Registered! 2016-11-27T10:25:05.914Z cpu0:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x43029e372700, Registered! 2016-11-27T10:25:05.914Z cpu0:65593)ScsiDevice: 4265: Successfully registered device "naa.5000000000000001" from plugin "NMP" of type 0
The datastore obviously gets detected as snapshot. I don’t know why and I’ve only seen this in ESXi 6.5 but it’s very easy to fix. Get a list of VMFS snapshots with ESXCLI:
# esxcli storage vmfs snapshot list 583b1a72-ade01532-55f6-f44d30649051 Volume Name: usbflash VMFS UUID: 583b1a72-ade01532-55f6-f44d30649051 Can mount: true Reason for un-mountability: Can resignature: true Reason for non-resignaturability: Unresolved Extent Count: 1
Use the VMFS UUID to mount the datastore:
# esxcli storage vmfs snapshot mount -u 583b1a72-ade01532-55f6-f44d30649051
The datastore should now show up within /vmfs/volumes and the Embedded Host Client.
Create VMFS5/6 Datastore on USB drives
Creating VMFS5 or VMFS6 datastores works just as in previous versions:
- Connect to the ESXi host with SSH
- Stop the USB arbitrator service. This service is used to passthrough USB device from an ESX/ESXi host to a virtual machine. (When disabling it, you can no longer passthrough USB devices to VMs)
~ # /etc/init.d/usbarbitrator stop
- (optional) Use this command to permanently disable the USB arbitrator service after reboot.
~ # chkconfig usbarbitrator off
- Plug in the USB device to your ESXi host. While connecting the USB device you can either watch /var/log/vmkernel.log to identify the device name or identify it within /dev/disks.
Devices are either named mpx.vmhbaXX or with an naa.X number.~ # ls /dev/disks/
- Write a GPT label to the device (Assuming that the Device ID is naa.5000000000000001)
~ # partedUtil mklabel /dev/disks/naa.5000000000000001 gpt
- To create a partition you need to know the start sector, end sector, which depends on the device size and the GUID.
The start sector is always 2048
The GUID for VMFS is AA31E02A400F11DB9590000C2911D1B8
The end sector can be calculated with the following formula (Use the numbers from getptbl):~ # partedUtil getptbl /dev/disks/naa.5000000000000001 gpt 15566 255 63 250069680
15566 * 255 * 63 – 1 = 250067789
You can also calculate the end sector with the following command:
~ # eval expr $(partedUtil getptbl /dev/disks/naa.5000000000000001 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1 250067789 - Create the VMFS partition (Replace with your end sector)
~ # partedUtil setptbl /dev/disks/naa.5000000000000001 gpt "1 2048 250067789 AA31E02A400F11DB9590000C2911D1B8 0"
- Format the partition with VMFS5 or VMFS6
# VMFS v5 ~ # vmkfstools -C vmfs5 -S USB-Datastore /dev/disks/naa.5000000000000001:1 # VMFS v6 ~ # vmkfstools -C vmfs6 -S USB-Datastore /dev/disks/naa.5000000000000001:1
The USB-Datastore should now appear in your datastores view.
This is how your command output should look like:
[root@esx4:~] partedUtil mklabel /dev/disks/naa.5000000000000001 gpt
[root@esx4:~] eval expr $(partedUtil getptbl /dev/disks/naa.5000000000000001 | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1
250067789
[root@esx4:~] partedUtil setptbl /dev/disks/naa.5000000000000001 gpt "1 2048 250067789 AA31E02A400F11DB9590000C2911D1B8 0"
gpt
0 0 0 0
1 2048 250067789 AA31E02A400F11DB9590000C2911D1B8 0
[root@esx4:~] vmkfstools -C vmfs6 -S USB-Datastore /dev/disks/naa.5000000000000001:1
create fs deviceName:'/dev/disks/naa.5000000000000001:1', fsShortName:'vmfs6', fsName:'USB-Datastore'
deviceFullPath:/dev/disks/naa.5000000000000001:1 deviceFile:naa.5000000000000001:1
ATS on device /dev/disks/naa.5000000000000001:1: not supported
.
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).
Creating vmfs6 file system on "naa.5000000000000001:1" with blockSize 1048576, unmapGranularity 1048576, unmapPriority default and volume label "USB-Datastore".
Successfully created new volume: 583b0efb-33b7ba1e-57e7-001fc69b9659
USB Datastore on USB drives with >2TB
Large drives still do not work in ESXi 6.5:
2016-11-27T10:18:38.213Z cpu4:66004)WARNING: ScsiPath: 7481: Adapter Invalid does not exist 2016-11-27T10:18:38.216Z cpu4:65939)WARNING: ScsiPath: 7481: Adapter Invalid does not exist 2016-11-27T10:18:38.216Z cpu1:65942)VMK_PCI: 393: Device 0000:00:14.0 name: vmhba32 2016-11-27T10:18:38.216Z cpu1:65942)DMA: 646: DMA Engine 'vmhba35' created using mapper 'DMANull'. 2016-11-27T10:18:38.230Z cpu2:65593)ScsiScan: 446: Path 'vmhba35:C0:T0:L0': Failed to read VPD Serial id page: Not supported 2016-11-27T10:18:38.233Z cpu2:65593)ScsiScan: 446: Path 'vmhba35:C0:T0:L1': Failed to read VPD Serial id page: Not supported 2016-11-27T10:18:38.234Z cpu2:65593)WARNING: xpt_scsi_adapter_discover:1224: unable to find target 1: No connection 2016-11-27T10:18:38.237Z cpu7:65593)ScsiScan: 446: Path 'vmhba35:C0:T0:L1': Failed to read VPD Serial id page: Not supported 2016-11-27T10:18:38.238Z cpu7:65593)StorageApdHandler: 977: APD Handle Created with lock[StorageApd-0x43029e35d910] 2016-11-27T10:18:38.238Z cpu7:65593)ScsiEvents: 501: Event Subsystem: Device Events, Created! 2016-11-27T10:18:38.239Z cpu7:65593)VMWARE SCSI Id: Could not get disk id for vmhba35:C0:T0:L1 2016-11-27T10:18:38.239Z cpu7:65593)ScsiDeviceIO: 9120: Get VPD 80 Inquiry for device "t10.WD______SES_Device______WMC1T2972871____" from Plugin "NMP" failed. Not supported 2016-11-27T10:18:38.239Z cpu7:65593)ScsiDeviceIO: 9167: Get VPD 86 Inquiry for device "t10.WD______SES_Device______WMC1T2972871____" from Plugin "NMP" failed. Not supported 2016-11-27T10:18:38.242Z cpu7:65593)ScsiDeviceIO: 7851: Could not detect setting of QErr for device t10.WD______SES_Device______WMC1T2972871____. Error Not supported. 2016-11-27T10:18:38.242Z cpu7:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 40, Parameter: 0x43029e383880, Registered! 2016-11-27T10:18:38.242Z cpu7:65593)ScsiEvents: 300: EventSubsystem: Device Events, Event Mask: 200, Parameter: 0x43029e383880, Registered! 2016-11-27T10:18:38.242Z cpu7:65593)ScsiDevice: 4265: Successfully registered device "t10.WD______SES_Device______WMC1T2972871____" from plugin "NMP" of type 13 2016-11-27T10:18:38.244Z cpu7:65593)ScsiScan: 446: Path 'vmhba35:C0:T0:L0': Failed to read VPD Serial id page: Not supported 2016-11-27T10:18:38.245Z cpu7:65593)StorageApdHandler: 977: APD Handle Created with lock[StorageApd-0x43029e213fd0] 2016-11-27T10:18:38.245Z cpu7:65593)ScsiEvents: 501: Event Subsystem: Device Events, Created! 2016-11-27T10:18:38.246Z cpu7:65593)VMWARE SCSI Id: Could not get disk id for vmhba35:C0:T0:L0 2016-11-27T10:18:38.246Z cpu7:65593)ScsiDeviceIO: 9120: Get VPD 80 Inquiry for device "t10.WD______My_Book_1140____WMC1T2972871____" from Plugin "NMP" failed. Not supported 2016-11-27T10:18:38.246Z cpu7:65593)ScsiDeviceIO: 9167: Get VPD 86 Inquiry for device "t10.WD______My_Book_1140____WMC1T2972871____" from Plugin "NMP" failed. Not supported 2016-11-27T10:18:38.248Z cpu7:65593)ScsiDeviceIO: 7851: Could not detect setting of QErr for device t10.WD______My_Book_1140____WMC1T2972871____. Error Not supported. 2016-11-27T10:18:38.251Z cpu2:65563)ScsiDeviceIO: 2948: Cmd(0x439500b1fe80) 0x1a, CmdSN 0x6ce from world 0 to dev "t10.WD______My_Book_1140____WMC1T2972871____" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. 2016-11-27T10:18:38.253Z cpu7:65593)ScsiDeviceIO: 8365: Could not detect setting of sitpua for device t10.WD______My_Book_1140____WMC1T2972871____. Error Not supported. 2016-11-27T10:18:38.253Z cpu7:65593)ScsiDevice: 3163: Failing registration of device 't10.WD______My_Book_1140____WMC1T2972871____': Unsupported sector size:4096 2016-11-27T10:18:38.253Z cpu7:65593)ScsiEvents: 545: Event Subsystem: Device Events, Destroyed! 2016-11-27T10:18:38.253Z cpu7:65593)WARNING: NMP: nmp_RegisterDevice:851: Registration of NMP device with primary uid 't10.WD______My_Book_1140____WMC1T2972871____' failed. Not supported 2016-11-27T10:18:38.253Z cpu7:65593)StorageApdHandler: 1062: Freeing APD handle 0x43029e213fd0 [] 2016-11-27T10:18:38.253Z cpu7:65593)StorageApdHandler: 1147: APD Handle freed!
