Following on from the previous post about a recent white-box VMware ESXi 5.1 build I was hoping to enable GPU pass-through to one of the VM’s. Initially this was to make use of things like Adobe Photoshop via XenApp to aid in GPU rendering, but having played around with Plex I was too tempted to divert the resources to a media based VM to output via HDMI as my existing WDTV Live is feeling a little dated.
First I tried with an old Nvidia Geforce 7300 I had lying around but this appeared a no go, it was detected by the VM but Windows couldn’t seem to start the device driver for it. Reading further it appears AMD cards have the best results for pass-through so I went ahead and purchased an ASUS 7750 1GB DDR5 card for around £75. I opted for the DirectCU model as this is a completely fanless design further aiding in a silent ESXi host.
Installation was simple with more than enough space for the cooler – I was a little worried over it’s size given the mATX board but it only takes up around 2 slots.
I haven’t tested with the onboard GPU enabled as I figured it safest to disable, again reading around I found most people had to turn any built in graphics options off to properly allow a discrete card to work. In my case on the ASRock board this was another simple process, booting to the UEFI utility under the Advanced / North Bridge Configuration there are three options I disabled. IGPU Multi-Moniter, Render Standby and Deep Render Standby. I also set the Primary Graphics Adapter to the PCI Express slot. I should also mention I disabled the On-Board audio as the AMD card will support HDMI audio
Next it was configuring ESXi to allocate the device ready for pass-through. This can be done by following the below:
- Note:This does assume you have a VT-d capable CPU and Mainboard
- Connect to vCenter (or stand-alone ESXi host) via the vSphere Client
- Select the host and go to the Configuration tab
- Select Advanced Settings, here you can now add items for passthrough
- Once added you will have to restart the ESXi host to complete
Once the device has been added we now need to add it to the appropriate VM. This is another pretty quick process, first make sure the VM is powered off. Right click the required VM and select Edit Settings, within the Hardware tab click Add… and then select PCI Device from the list. You should now see the items we enabled for pass-through, select them and click Next/Finish.
As a slight additional complication for any VM that looks to have more then 2GB of RAM assigned it will fail to Power On if we only leave the process as above, there are a few additional steps I needed to carry out in order to get the VM booted up.
Checking the VMware.log for the machine I noticed the following error, the VM would basically attempt to power on and no display would be seen. No real error would show in the tasks as it would just power off again after a few seconds.
2013-03-16T14:31:34.829Z| vcpu-0| I120: [msg.pciPassthru.mmioOutsidePCIHole] PCIPassthru 001:00.0: Guest tried to map 64 device pages (with base address of 0xb2640) to a range occupied by main memory. This is outside of the PCI Hole. Add pciHole.start = “2854” to the configuration file and then power on the VM.
Even after adding the pciHole command as stated it didn’t seem to boot, reading around I found the below to give people the best results:
pciHole.start = “1200”
pciHole.end = “2200”
To add these to your VM config just download the vmname.vmx file from the datastore folder where the VM resides. Load up in notepad (or my preference notepad++) and add them to the end of to config. Copy the config back to the datastore overwriting the original and that’s it.
As a final step we also need to ensure all the Memory is allocated to the machine, this is done via editing the VM’s settings and going to the Resources tab. Then under Memory tick the box that says “Reserve all guest memory (All Locked)”. Having completed the above I was then able to power on the VM and start installing the AMD Catalyst drivers – along with configuring for HDMI output.
For some tweaks I disabled the VMware SVGA adapter so the VM only see’s the ATI… sorry AMD card (force of habit ;)). Installing TeamViewer for remote access and testing with Plex the basics were all good to go and looking to work pretty solidly. So far no PSOD or BSODs so it’s looking very promising!
For those interested I stumbled across Unified Remote which has been awesome for remote Keyboard/Mouse and Plex use – It’s available HERE. I run this via Android on a Nexus 7 and it’s flawless, an amazing way to be super lazy and control your VM from the couch! Only initial gripe is it doesn’t appear to be a service so you can’t boot up and have it running at lock screen – basically means you just have to disable the Win8 lock screen via gpedit.msc along with UAC as these two things appear to stop it’s functionality until manual intervention kicks in. Continuing the lazy streak TeamViewer on the Nexus has helped in those instances 😉