m

Following on from the previous post about a recent white-box VMware ESXi 5.1 build I was hoping to enable GPU pass-through to one of the VM’s. Initially this was to make use of things like Adobe Photoshop via XenApp to aid in GPU rendering, but having played around with Plex I was too tempted to divert the resources to a media based VM to output via HDMI as my existing WDTV Live is feeling a little dated.

First I tried with an old Nvidia Geforce 7300 I had lying around but this appeared a no go, it was detected by the VM but Windows couldn’t seem to start the device driver for it. Reading further it appears AMD cards have the best results for pass-through so I went ahead and purchased an ASUS 7750 1GB DDR5 card for around £75. I opted for the DirectCU model as this is a completely fanless design further aiding in a silent ESXi host.

Installation was simple with more than enough space for the cooler – I was a little worried over it’s size given the mATX board but it only takes up around 2 slots.

I haven’t tested with the onboard GPU enabled as I figured it safest to disable, again reading around I found most people had to turn any built in graphics options off to properly allow a discrete card to work. In my case on the ASRock board this was another simple process, booting to the UEFI utility under the Advanced / North Bridge Configuration there are three options I disabled. IGPU Multi-Moniter, Render Standby and Deep Render Standby. I also set the Primary Graphics Adapter to the PCI Express slot. I should also mention I disabled the On-Board audio as the AMD card will support HDMI audio

Next it was configuring ESXi to allocate the device ready for pass-through. This can be done by following the below:

  • Note:This does assume you have a VT-d capable CPU and Mainboard
  • Connect to vCenter (or stand-alone ESXi host) via the vSphere Client
  • Select the host and go to the Configuration tab
  • Select Advanced Settings, here you can now add items for passthrough
  • Once added you will have to restart the ESXi host to complete

Once the device has been added we now need to add it to the appropriate VM. This is another pretty quick process, first make sure the VM is powered off. Right click the required VM and select Edit Settings, within the Hardware tab click Add… and then select PCI Device from the list. You should now see the items we enabled for pass-through, select them and click Next/Finish.

As a slight additional complication for any VM that looks to have more then 2GB of RAM assigned it will fail to Power On if we only leave the process as above, there are a few additional steps I needed to carry out in order to get the VM booted up.

Checking the VMware.log for the machine I noticed the following error, the VM would basically attempt to power on and no display would be seen. No real error would show in the tasks as it would just power off again after a few seconds.

2013-03-16T14:31:34.829Z| vcpu-0| I120: [msg.pciPassthru.mmioOutsidePCIHole] PCIPassthru 001:00.0: Guest tried to map 64 device pages (with base address of 0xb2640) to a range occupied by main memory. This is outside of the PCI Hole. Add pciHole.start = “2854” to the configuration file and then power on the VM.

Even after adding the pciHole command as stated it didn’t seem to boot, reading around I found the below to give people the best results:

pciHole.start = “1200”
pciHole.end = “2200”

To add these to your VM config just download the vmname.vmx file from the datastore folder where the VM resides. Load up in notepad (or my preference notepad++) and add them to the end of to config. Copy the config back to the datastore overwriting the original and that’s it.

As a final step we also need to ensure all the Memory is allocated to the machine, this is done via editing the VM’s settings and going to the Resources tab. Then under Memory tick the box that says “Reserve all guest memory (All Locked)”. Having completed the above I was then able to power on the VM and start installing the AMD Catalyst drivers – along with configuring for HDMI output.

For some tweaks I disabled the VMware SVGA adapter so the VM only see’s the ATI… sorry AMD card (force of habit ;)). Installing TeamViewer for remote access and testing with Plex the basics were all good to go and looking to work pretty solidly. So far no PSOD or BSODs so it’s looking very promising!

For those interested I stumbled across Unified Remote which has been awesome for remote Keyboard/Mouse and Plex use – It’s available HERE. I run this via Android on a Nexus 7 and it’s flawless, an amazing way to be super lazy and control your VM from the couch! Only initial gripe is it doesn’t appear to be a service so you can’t boot up and have it running at lock screen – basically means you just have to disable the Win8 lock screen via gpedit.msc along with UAC as these two things appear to stop it’s functionality until manual intervention kicks in. Continuing the lazy streak TeamViewer on the Nexus has helped in those instances 😉

35 comments

  1. 29/03/2013 at 8:56 AM Webbo

    Just to add as an additional note, I’ve had a few queries on how general video playback handled. A few people report tearing or judder issues – especially on 1080p video. It took me a while to tweak it to display how I prefer but the primary tearing issues were down to V-Sync, ensuring this was enabled within Plex and also having it set to “Always On” within the AMD Catalyst Control Center (under Gaming – 3D Application Settings) removed most the immediate playback glitching. It was then a case of enabling hardware acceleration and playing with the HDTV viewing profiles to get it perfect 😉

  2. 24/05/2013 at 7:58 PM vibe666

    any idea if this could work with multiple vm’s each with their own gfx cards?

    i have 3 htpc’s in a central media closet plus a nas and an esxi host, but i’d like to bring that down to just a single host running everything if possible so i was wondering if pci passthrough on multiple gfx cards could be made to work in multiple htpc vm’s?

  3. 25/05/2013 at 7:30 AM Webbo

    Afraid I couldn’t say for sure as I’ve not tried it myself, I don’t see why not but then again it’s not an officially supported option from VMware (hardware support anyway). It’s proven awkward for a lot of people to get scenario’s like this up and running smoothly as different GPUs can be hit or miss. I’ll see if I can dig out a spare card to test the concept as I’d be curious myself 🙂 stay tuned….

  4. 11/06/2013 at 7:28 PM intoran

    I am looking at this as well. I would like to run XBMC inside the VMs each with their own discrete card. Thanks for the info and keep us posted on bugs/glitches.

    • 14/06/2013 at 1:48 PM Webbo

      Still going strong, given it’s now my main media station it’s certainly getting alot of use. Haven’t tried XBMC with it but Plex / VLC run everything perfectly so far. Will update the AMD drivers soon to see if any further improvements or possible issues with the latest versions.

  5. 12/06/2013 at 1:48 PM Nathan Simon

    VMDirectPath limits in ESXi 5.1

    VMDirectPath PCI/PCIe devices per host 8
    VMDirectPath PCI/PCIe devices per virtual machine 4 (6 if 2 of them are teradici devices)

  6. Pingback: Any Gigabit Load Balancing Routers For Home Use ~ 2x 300mbps Fibre - Page 3

  7. 18/06/2013 at 8:34 AM Cristian

    I was almost convinced there was no way to get the GPU passthrough working…. No matter what setting I was changing I was always getting the nasty atikmpag.sys bsod screen when enabling the second monitor.
    Finally, I found this post, and after disabling IGPU Multi-Moniter, Render Standby and Deep Render Standby, it started to work as expected

    My setup:

    ASRock Z77Extreme6 / AMD (ATI)7750 / ESXi 5.1update1

    Thank you so much Webbo!

  8. 21/07/2013 at 7:04 PM intoran

    How did you disable the VMWare adapter? I tried disabling it in device manager and couldn’t get video on the AMD card after that. Kinda wierd behavior even in console view until I enabled it.

  9. 22/10/2013 at 3:26 PM Steve K

    I also would be interested – when I disable the VMware SVGA 3D display adapter and boot the server, display properties shows me using an unknown devices called VgaSave. I feel like this is so close! ESXi 5.5 / Guest OS Server 2008 R2.

  10. 04/11/2013 at 7:18 PM chrisiesmit93Chris Smit

    What type of motherboard are you exactly using?

  11. 28/12/2013 at 3:44 AM Hoss

    Create a service from the EXE using built-in sc.exe …

    http://support.microsoft.com/kb/251192

  12. 14/01/2014 at 4:02 PM Leon Straathof

    Hi, VMware documentation states that you should have at least 2 graphic adapters because the console still needs 1. Altough they mention it is possible to configure with just 1 they do not recommend it. So can you test with the HD4000 enabled, I want to do the same with exactly the same motherboard although I want to use a cheaper amd video card hd7750 or hd7770. Reason for the cheaper card selection is that I do not want to insert a lager power supply in my rig have a Be-Quiet 450 rated powersupply the same motherboard a i7-3770 32GB memory 2 normal sata disks 7200 rpm and the same ssd as you have. I wonder what would be the optimal card that when all hardware is under full load puts the power supply at about 85% load (about the optimal performance point and still getting long life out of my parts).

  13. 05/02/2014 at 9:26 AM crazyfool2100

    After failing to get an NVIDIA GT640 to pass-through, I purchased the ASUS 7750 1GB DDR5. After installing the VMware tools in the Windows 7 Guest and installing the ATI Catalyst drivers, pass-through worked like a charm. Using an AsRock E3C224-4L. Thank You!

  14. 05/02/2014 at 1:10 PM Peer

    Webbo can you please tell us how you did disable the VMware SVGA Adapter?

    • 07/02/2014 at 4:35 PM Hoss

      I use startup and shutdown batch scripts (configured in local Group Policy) to disable VMware SVGA at startup and enable it at shutdown using devcon.exe.

      1) Get DevCon: http://9b5.org/2011/10/getting-devcon-exe-onto-and-working-with-windows-7/

      2) Build batch scripts. Example from my startup_disable_vmware_svga.bat:

      c:
      cd \tools\devcon\
      devcon disable “PCI\VEN_15AD&DEV_0405&SUBSYS_040515AD&REV_00”
      exit

      Build a second batch script called shutdown_enable_vmware_svga.bat that does “devcon enable PCI\…….”

      3) Configure Group Policy to run the batch scripts at shutdown and startup: http://technet.microsoft.com/en-us/library/cc753404.aspx

      • 05/09/2014 at 5:39 PM ted stevens

        ok so that was the most convoluted way to try and get devcon and it doesnt even work. Just dl it from MS like a normal person: http://support.microsoft.com/kb/311272 and put the exe in your system dir

  15. 14/02/2014 at 9:18 AM Natriumatom

    I second that. Please tell us, or did you just disable it in Windows Manager?

    • 14/02/2014 at 9:49 AM Webbo

      Sorry all! Have been asked a few times about this but need some serious time to update my site, lots of new stuff has been played with to blog about 🙂

      My setup I have simply disabled it within Windows Manager. So far still going strong as a little HTPC vm!

  16. 14/02/2014 at 10:57 AM Natriumatom

    Thanks a lot 🙂

    I’m working on something like that myself at the moment with the following hardware:

    Intel i5 4430 Haswell
    AsRock B85M Pro4
    16GB Ram
    Radeon 6450
    LG Bluray Writer
    120GB SSD
    some storage HDs

    I tried it a long long time with Xen, Xenserver, KVM etc. but I could never patch through the Bluray Drive so that it works for bluray movies with AnyDVD HD or PowerDVD. Finally after I saw your blog post I tried it with ESXi 5.5 and low and behold the Bluray drive gets patched through and AnyDVD works. But I’m still struggling with the GPU Passthrough at the moment, crashes on boot when I passthrough the radeon and have the driver installed (atipmak something).

  17. 17/02/2014 at 11:10 PM Peer

    thanks
    just for the note:
    for all who what to make the ATI card as primary adapter:
    – configure vm with the gfx card
    – install OS
    – disable ATI HDMI device in the device manager (necessary?)
    – Install VMware Tools without SVGA adapter driver
    – Install ATI driver and now you can setup two monitors, activate remote desktop before reboot!
    – if you reboot then the real output on the gfx is active and ESXi shows a black screen, try remote desktop

    tested with ESXi 5.5, Supermicro X9SRA and ATI 7870 GHz

  18. 15/03/2014 at 6:46 AM Pooch

    Got an hd5450 mostly working on clean win 7 vm. Just can’t get DXVA working for any of the media foundation decoders if the radeon is set to be the primary adapter in display options (black screen with audio only). If I just extend it as a second monitor it looks like dxva works by using the svga driver. If I disable the svga then radeon stops working. Anyone have dxva working with just their radeon card enabled?

  19. 18/03/2014 at 4:26 AM alanpuccinelli

    I feel like I’m so close with my ESXI 5.5 setup but I can’t get hardware accelerated video to play correctly when I’ve got passthrough enabled.

    1st to get passthrough going on my HD5450 it looks like I have to have the Vmware SVGA driver installed and on. If its not running I get no passthrough. I can disable it after the system boots and it works until reboot.

    If the SVGA card is the Primary then HW video works but the minute I switch over to the Radeon as primary I only get black screen and audio for MPEG2 and MP4 videos. The issue appears to be with the Media Foundation decoders being assigned to the SVGA card as the primary renderer. I can’t seem to find away around this. If I disable the SVGA card I cant hardware decode DXVA with the Radeon which cripples Windows Media Center and anything playing through the default media player. I can use VLC or anything that Hardware renders with the older DirectShow stuff but I’d like to be able to use Media Center. Anyone have success getting DXVA to work this way with and HD5xxx series card on ESXI 5.5?

  20. 18/03/2014 at 11:31 AM Steve K

    Read Hoss’ reply above. That is how I got my clients vm to pass through GPU reliably. Disable svga on startup, and enable it before shutdown.

    • 19/03/2014 at 10:27 PM alanpuccinelli

      When you say reliably can you verify that DXVA is working correctly? What card are you passing through and what version of ESXi are you running?

      • 19/03/2014 at 10:31 PM alanpuccinelli

        When I disable SVGA manually after boot I can’t get DXVA to work right for media foundation decoders (black screen with audio only on MPEG2 and MP4 streams) and DXVA2 option in XBMC results in no video too. Are you saying disabling it via a script at startup will actually change this? Perhaps disabling it before the ATI drivers load is the key?

  21. 06/07/2014 at 11:08 PM Matt

    You guys are awesome btw for posting all this stuff. I feel like i am close to getting this figured out. I keep re-installing the VM in hopes this will work.

    I have an asrock 970 extreme 4
    i am passing through 2 hard drives (RAW LUN)
    USB highpoint
    4 cores and 8 g of ram.
    I also have a usb soundcard and a 5040 ATI card.

    I have done 1 item at a time and had no issues till i get to the graphics card. Once i install the drivers for it and reboot, it crashes ESXI and i have to power cycle the whole box, then i turn it back on and it boots up fine, reboot, same problem. If i disable the vmware vga card it reboots fine but the picture is gone on the ati card and only works through the vsphere client, i look in devices and it shows that the ATI card is the only one connected but it only works from the vsphere client. It seems backwards. I tried the script and it turns if on at shutdown and kills it during startup.

    Any help people, i want to try the fresh installation but want to make sure that has worked for some people.

    Thanks guys

    • 13/08/2014 at 1:39 AM Matt

      I fixed my own problem, i had to use different cards, i guess although some people use certain cards, you just have to keep trying different ones untill you find the ones that work for you. For me i have 2 ATI Raedon R7 240’s that are working great with the 3rd R9 270x. All three work together at the same time. The highpoint usb card is freaking awsome. Make sure you guys do the pci hole thing and don’t copy and paste it, write it out (That took me weeks to figure out). Also the driver install is not the auto installer but the custom driver installer looking in the unzipped folder for the driver. Also install vmware tools before you install the drivers. Good luck and thanks for this thread, it really helped me out.

  22. 20/06/2015 at 11:10 PM Brandon

    I have the same black screen with DVA2 enabled. The latest driver doesn’t include the codecs but 13.9 did, Verified using the DXVA checker.. But using DXVA2 inside KODI results in audio but black screen any help?

  23. 20/06/2015 at 11:40 PM Brandon

    Does anyone have a AMD card with a virtualized passed through KODI build WITH DXVA2 working?

  24. 25/09/2015 at 2:30 PM Denis

    Yes DXVA2 works just fine with a AMD R9 285.
    Some else has glitches with two VMs running at the same time like in this video? https://www.youtube.com/watch?v=sY2ZumCMv_4

  25. 10/10/2015 at 3:23 PM Jan

    Can tell that after switching off integrated Audio and 1394 on my Asrock Extreme4 finally I can passthrough ATI Radeon 7700

  26. Pingback: ESXi 6 setup | adrian's domain

  27. 25/12/2016 at 4:28 PM Jay Yang

    Hi,

    What os is running for your Kodi?

    How did you deal with the amd driver issue?

    How did you disable the default virtual vga card
    ?