Full KVM support working on the jetson nano ( Kernel Virtual Machine ) plus Virtio-GPU passthrough

Hi Guys,

Just wanted to let everyone know that it is possible to get full kvm virtualization to function on the jetson nano.

My setup is as follows:

Running the latest kernel release ( 4.9.140 ) with a couple patches and a small modification to the DTB, with KVM and Virtio modules enabled of course.

Custom Lubuntu 19.10 image that I built.

Mater branch of Qemu built with OpenGL, KVM and virgilrenderer enabled.

virgilrenderer built from source.

With this setup I am able to boot AOSP 9.0 with full kvm acceleration and virtio-gpu pass-through!! getting near bare metal performance it seems.

menu animations are smooth and I have played back a couple youtube videos in the built in webview browser and playback seemed fine, I have had some difficulty installing google play services with the particular image I am using but I am confident that with some tinkering it would not be to difficult to get a fully functioning android experience running within linux on the nano… which is pretty damn awesome.

I have also booted a generic Armbian image in Qemu, again with full KVM and GPU pass-through and managed to play a couple 720p youtube videos without frame drops. So again with some optimization I’m sure performance could be improved further.

I plan on providing my modified kernel and DTB for download and I will most likely provide instructions for getting things set up on your own once I have the time.

If there is anyone here with Android emulation experience through Qemu that might be able to give me some tips on how I can get google play services set up without it crashing on me that would be appreciated I would really like to work with someone here on putting together a pre-built android image that is set up for media consumption and gaming / emulation or whatever floats your boat.

If we can put that togeather we could provide a downloadable package with my modified kernel and everything that is required to allow anybody to get setup with Android on the nano.

I’m sure an android environment could be useful to people playing around with AI on the nano as well… camera pass-through should be an easy task in qemu … I already have Bluetooth pass-through working.

Also how about Windows for ARM on the nano… that could be fun too.

I’ll probably post a video Demo when I have some time later, until then please give me your thoughts.

Regards,
Chris.

Just wanted to mention that with this setup Netflix playback on the jetson nano is now possible as well. running an Armhf virtual machine with kodi 18 and the netflix addon.

Or if you wanted you could run a Rasbian VM and watch netflix through chromium browser.

Neat. Can you provide your kconfig, or a diff from it and the Tegra defconfig? I haven’t tried toying with trying to get kvm to work on Tegra, but it’s nice to know it does.

Unfortunately it won’t be that simple the modifications to the kernel that I performed weren’t tracked.

Going into detail will take some time but I can provide the modified kernel source including the modified dts.

For now I can give you a rough idea of what was done.

First the kernel was patched with the following patches ( which don’t apply very cleanly so some manual modification is required ).

Patch Set: https://patchwork.kernel.org/patch/9358817/

Next the jetson nano’s DTB is modified using the following sources as a reference, however I believe the most important change is to the interrupt controller section but keep in mind you can’t use the same numbers they used, you have to follow the numbers the nano is using in that section to extrapolate the additions ( if that makes sense )

Again the following pdf and patch were only used as a reference for the changes I made to the nano’s source.

https://github.com/BaoqianWang/VirtualizationOnJetsonTX2/blob/master/KVM%20on%20Jetson%20TX2.pdf

http://mc.pp.se/l4t-sources-28-1.patch

lastly the appropriate kernel modules are enabled here are some sources that I referenced.

https://www.collabora.com/news-and-blog/blog/2018/02/12/virtualizing-gpu-access/

http://www.shincbm.com/linux/kvm/arm64/2019/03/22/arm64-kvm-armbian.html

http://www.linux-kvm.org/page/Tuning_Kernel

Unfortunately all of the above kernel modification are required to fully support kvm acceleration on the nano within qemu, anything less and I was faced with errors regarding irqchip / pmu.

I think the best thing to do would be to provide my kernel source and then you can take a look at the various modifications that were done and compare it with the stock source.

sorry I can’t be more specific but I’m totally willing to share my source as well a pre-built kernel for ppl to try out. I probably should have taken more care to track my progress.

The modifications that I made to the dtb were a last ditch effort to get things working because all of my previous attempts had failed … it just so happened to be the thing that was needed to get it working properly.

Any chance you could generate working patches based on your work?

That would be nice. I won’t have time to do anything with it until April at least, but I’m sure others might find it useful.

Have a question, why 0x0100 size and not 0x2000 in the device tree? According to the TRM for TX1 (=Nano), this should be the right values:

intc: interrupt-controller {
	compatible = "arm,cortex-a15-gic";
	interrupt-parent = <&intc>;
	#interrupt-cells = <3>;
	interrupt-controller;
	reg = <0x0 0x50041000 0x0 0x1000
	       0x0 0x50042000 0x0 0x2000
	       0x0 0x50044000 0x0 0x2000
	       0x0 0x50046000 0x0 0x2000
	interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
	status = "disabled";
};

Could you please make tutorial how to set up VM at your Xubuntu system? I want to test box86 performance but can’t figure out how to do this on my own.

Hi @mr.chrismitchells,

Nice work. 👍
I was wondering if it’s actually more effort to get Andorid to run directly on Nano…

Greate Work! Thanks @mr.chrismitchells, I am also very interested in repeat your work, will it be available for you to post the tutorial and resource in github, looking forward to follow you and contribute as well.

Follow: Finally worked on KVM with GPU virtualization.

hi mr.chrismitchells，
Thank you for your share~
could you tell me where I can down load the pre-built android9.0 image that showed in your video？

Is it possible to run Win10 ARM on jetson nano using qemu and kvm ?

but he disappeared ? I didn’t understand if it works or not…

@rreddy78
seems folks were able to do so
https://pbs.twimg.com/media/EXVi8WzUcAAD8dC?format=jpg&name=large
Xubuntu 20.04 Focal Fossa L4T R32.3.1 - Custom Image for the Jetson Nano - #21 by astr0b4by?

that’s so nice ! but using a different ubuntu version,the nvidia jetpack will work anyway ? why not to using ubuntu 1804 instead of 20,to preserve the funcionability of the nvidia jetpack.

Do u know if Astroboy wrote a detailed tutorial for learn how to install Windows10 on Nvidia Jetson Nano under qemu-system-aarch64 + KVM ? Can u give me the link ? I wanna try to replicate.

Is it possible to use nvidia-docker with a nvidia NGC tensor flow container that uses nvidia GPU inside the kvm virtual machine?

So rather than a virtio-gpu, is it possible to just passthrough host GPU to vm?