This guide provides a step-by-step procedure for virtualizing Intel® Software Guard Extensions (Intel® SGX) using the Kernel-based Virtual Machine (KVM) virtualization module in the Linux* kernel with the QEMU* virtual machine monitor. This creates a virtual machine on your Intel SGX capable hardware, which can use Intel SGX in the guest operating system.

Requirements

In order for us to use QEMU we need to install it and that can be done with a package manager. In my case, here on Ubuntu, I'll write apt install qemu and qemu-kvm. This will install QEMU and some other software that it needs to run. The version of QEMU that I'm installing has a graphical user interface component. QEMU full system emulation has the following features: QEMU uses a full software MMU for maximum portability. QEMU can optionally use an in-kernel accelerator, like kvm. The accelerators execute most of the guest code natively, while continuing to emulate the rest of the machine. I have installed KVM using the instructions given on the wiki. However, when I try to check its version using kvm -version what I get as output is this QEMU emulator version 1.2.0 (qemu-kvm-1.2.0+. GitHub is where people build software. More than 40 million people use GitHub to discover, fork, and contribute to over 100 million projects. Your kernel has to be 2.6.29 or newer to run any version of qemu-kvm (kernel 2.6.27/2.6.28 with kvm-kmod 2.6.29 will also work) your kernel has to be 2.6.25 or newer to run the kvm 76 userspace (or any newer kvm-XX release) the modules provided by Linux 2.6.22 or later require kvm-22 or any later version.

To use Intel SGX in a virtual machine, you must meet the following requirements:

• The host system must support Intel SGX.
• Intel SGX must be enabled, either explicitly in the BIOS or via the software enabling procedure.
• If you want to use Flexible Launch Control in guest systems, the hardware must also support the feature.

Note: Intel SGX support in KVM and QEMU is still under active development. Significant changes to the software can and do occur during each revision. This software should not be used in production deployments at the current time.

Installation Procedure

This guide assumes you are starting from a clean installation of Ubuntu* 18.04.2 LTS Server, but the steps here can be adapted to other Linux distributions.

Note that the current release of KVM-SGX is based on the Linux 5.x kernel. You will be building and installing the 5.0.0 kernel as part of this process.

Install Prerequisites

Install the software packages that are needed to build the kernel:

Building KVM-SGX

Clone the kvm-sgx repository. This takes several minutes, even over a high-speed/high-bandwidth connection.

Prepare for the kernel build. Setting a shell variable that defines the build area makes this process less error-prone.

The O parameter specifies the output location of your kernel configuration and build. This allows you to build multiple kernels from the same source tree without them interfering with one another, and keeps your source tree clean.

With this convenience shell variable set, you are ready to configure the kernel.

To enable Intel SGX support in KVM guests, you must enable the core functionality in the kernel from the Processor type and features menu. Scroll down to Intel SGX core functionality and select it. It will be off by default if you are building from a fresh source tree. When this item is selected, the menu expands and auto selects Intel SGX Driver (this cannot be changed).

Next, you need to ensure that KVM virtualization is enabled. From the top-level menu, go to the Virtualization menu and ensure the following features are selected. These should already be enabled as loadable modules (an “M” instead of an asterisk “*”), so you should not have to change anything here.

• Kernel-based virtual machine (KVM) support
• KVM for Intel® processors support
• The virtio options

The other options are not required, but most of them may already be set.

Save your settings and Exit the kernel configuration screen.

The next step is to compile the kernel. A first-time build from the source tree is a lengthy process that can take a couple of hours, depending on your hardware.

Note that you can use the -j option with make for a parallel build that speeds up the compilation, but being too aggressive with this parameter can lead to build failures with cryptic error messages.

When the build is finished, install the new kernel. You’ll need to be at the root level for this.

The installation procedure runs grub to update your boot configuration so that the new kernel boots by default. At this point, you are ready to reboot.

After rebooting, log in to verify the new kernel and Intel SGX support. First, the kernel version should be 5.0.0+, which you can check with uname.

Verify that the Intel SGX driver has loaded by examining the kernel log. You should see a log entry for sgx, which prints the address range for the enclave page cache (EPC) in memory.

If you do not see an “sgx: EPC section” line, then your Intel SGX driver did not load. Possible causes are:

• Intel SGX is not enabled in the system BIOS
• Intel SGX is not supported by your processor
• Intel SGX core functionality was not selected in the kernel configuration
• An old kernel without the Intel SGX driver booted instead of the new one

If you see this line:

then your system supports Intel SGX, but not Launch Control. You’ll still be able to use KVM in your guest systems, but launch control will not be available.

Building QEMU-SGX

Clone the qemu-sgx repository.

Check out the 3.1.0 tag. This is the release that is compatible with the 5.0.0 kvm-sgx kernel.

The README file from the kvm-sgx repository states which qemu-sgx releases are compatible with a given kvm-sgx kernel.

The build procedure for QEMU is slightly different than that for most Linux applications. You need to create a build directory, then run QEMU’s configure script from within. The following configure options are recommended (and in some cases, required):

The --enable-kvm option is not necessary if you are booted into your KVM-enabled kernel, but is provided here for completeness and clarity.

The --enable-spice option will let you use the libvirt package and virt-viewer to reach the console.

The --disable-git-update option prevents the QEMU build system from trying to pull in updated sources from the QEMU Git* repository. For the Intel SGX build it's recommended that you freeze the source code repository to ensure compatibility in the event a rebuild is necessary. The QEMU-SGX repository is an out-of-tree set of patches to the QEMU source code, which means the original QEMU package maintainers are not validating their updates against the changes needed by Intel SGX.

The --enable-curses, --enable-gtk, and --enable-vnc options provide additional flexibility for connecting to the console.

If you enable other options, you may need to install additional prerequisite packages.

A quick functionality test is to run an empty guest virtual machine (VM) with minimal arguments, and force Intel SGX support on. If there are errors, the VM will refuse to start. (If everything is correct, the VM will start, but there is no OS so it will not have anything to boot. This is normal.)

If Intel SGX is properly enabled on the host, you should see output similar to the following when the virtual machine starts up:

Hit Ctrl-A, then C to get the monitor’s command prompt, then type quit to exit.

If the virtual machine ran properly, you are ready to install QEMU. It will go into /usr/local/bin unless you changed the install prefix when you ran configure.

Using Intel SGX in the QEMU Virtual Machine

There are two parts to enabling Intel SGX in a guest VM.

• The SGX feature must be enabled.
• An enclave page cache (EPC) must be defined.

Enabling Intel SGX in the VM

Adding +sgx to the -cpu option to QEMU enables Intel SGX in the VM. Depending on what CPU model you choose for your VM, QEMU may auto-enable SGX, but there’s less confusion if you explicitly add the sgx parameter.

An easy way to enable Intel SGX support is to pass the host CPU through to the VM rather than define a specific CPU model. This is the default behavior, but again, being explicit with QEMU options helps to eliminate confusion.

If your hardware supports Intel SGX Launch Control, then you can enable (or disable) Launch Control with the sgxlc parameter. By default, the virtual machine inherits the host machine’s configuration, but continuing with the theme of clarity, it is recommended that you explicitly define it.

The following options enable Intel SGX in the VM, but disable Launch Control:

To enable both Intel SGX and Launch Control:

If for some reason you want to disable SGX in the VM entirely, use the following:

Allocating an Enclave Page Cache

The current version of the sgx-kvm kernel divides the EPC among the guest hosts. You specify how much of the EPC to expose to the guest, which reduces the total amount of EPC available to other guests (and to the host system itself, if you choose to run SGX applications on the host as well). If you have 96 MB of EPC on your host and you assign 16 MB to each virtual machine, then you’ll only be able to run (at most) six VMs at one time. This behavior may change in future releases of the kvm-sgx kernel.

To define an EPC range, you must allocate a custom QEMU memory object and assign it a unique ID, then provide the memory ID to the -sgx-epc option. The following QEMU options create and assign an 8-MB EPC to the VM:

You can define multiple EPC segments in this manner. See the README file for the qemu-sgx repository for more information on defining EPC segments.

Integrating with libvirt

While it is possible to run qemu-system-x86_64 directly, the lengthy QEMU command lines and complex options make that unwieldy. Libvirt provides a complete management suite that greatly simplifies virtual machine creation, execution, and maintenance. A usage guide to libvirt is beyond the scope of this tutorial.

Run the following to install libvirt:

The installation process should add you to the libvirt group. At this point you will need to log out and then log in again so that the new group membership takes effect. The default configuration for libvirt requires that users be in the libvirt group to create and manage global VMs.

This command also installs the native QEMU package into /usr/bin. To use the qemu-system-x86_64 binary that you installed into /usr/local/bin, you’ll need to specify the path to the QEMU emulator manually after creating a virtual machine. It’s not recommended that you overwrite or replace the distribution-managed package, as any package updates will overwrite your custom-built binary.

Configuring AppArmor

AppArmor is a mandatory access control (MAC) system installed by Ubuntu 18.04 by default. MACs constrain application capabilities at a finer level of granularity than the traditional Unix* file permissions. The default configuration for libvirt does not allow execution of programs in /usr/local/bin.

If you are using AppArmor, you’ll need to add the following line to /etc/apparmor.d/local/usr.sbin.libvirtd:

Reload the AppArmor profile for ibvirtd by running apparmor_parser:

Configuring QEMU in libvirt

The Intel SGX build of QEMU requires access to /dev/sgx_virt to assign EPC memory pages, and this access will be denied by libvirt’s cgroup controllers, which are enabled by default. You need to add /div/sgx_virt to the list of devices required by virtual machines. Edit /etc/libvirt/qemu.conf and change the cgroup_device_acl list to include /dev/sgx_virt:

Finally, QEMU needs to read and write to the /dev/sgx_virt device. Unfortunately, this device gets created with file mode 600 at boot time, and it’s owned by root, so QEMU has to launch as root. In /etc/libvirt/qemu.conf, set the runtime user to uid 0:

By default, libvirt also tries to use a security driver for QEMU, and it chooses either Security-Enhanced Linux (SELinux) or AppArmor, whichever is available. This is set by the security_driver parameter. If you don’t want to use a security driver, set this parameter to “none”. Configuring SELinux profiles for libvirt’s security driver is outside the scope of this document.

If you are using AppArmor for the security driver, you’ll also need to modify /etc/apparmor.d/libvirt/TEMPLATE.qemu to read:

After making these changes, you’ll need to restart the libvirtd service:

Install a Test VM

Verify that your login session includes the libvirt group by running the groups command:

If you want to be able to execute virt-manager or use the graphics consoles for your virtual machines, you’ll need a graphical desktop as well. If you haven’t already, install the desktop environment for Ubuntu, and (optionally) a VNC server, if you’ll be working remotely. Note that this step is strictly one of convenience; you can create and manage virtual machines in a command-line environment.

Unfortunately, libvirt does not have direct support for the Intel SGX enabled version of QEMU. To add Intel SGX support to a virtual machine, you’ll first need to create the VM and then edit the XML by hand.

You have three main options for creating a VM.

Option 1: Create a blank VM on the command line

This method works entirely from the command line, though it does require a few manual steps. First, generate a unique UUID with the uuidgen command.

Next, create a disk image to hold the VM. Using the QEMU disk image format is arguably the most versatile, and allows features such as VM snapshots and encryption. The following creates a 20GB disk image.

Finally, create an XML file, which we’ll call testvm.xml, that defines a minimal VM. This configuration is for 4 GB RAM and a single virtual disk, and does not include a graphical console (you can add this later in virt-manager). Note that you’ll need to set the UUID and the source file path to the disk image you created above. This must be an absolute path.

Next, define the VM in libvirt using the XML source file:

You can verify that the domain has been created by listing the known libvirt domains.

Run it as a final validation step. This definition does not provide a console, nor is there an OS loaded, so it will simply start and run in the background.

Stop the VM by running the following:

Option 2: Create a blank VM using virt-manager

Once your desktop environment is running, you can run virt-manager and create a virtual machine. This method is arguably the easiest, as the GUI steps you through the options.

Option 3: Create and install an OS in the VM with virt-install

Qemu Kvm Version Windows 7

This method allows you to create and install a VM from a source image such as an ISO image. The virt-install command is part of the virtinst package.

Adding Intel SGX to the Test VM

Qemu

Unfortunately, libvirt does not have direct support for the Intel SGX enabled version of QEMU. To add Intel SGX support to your newly-created virtual machine, you’ll need to edit the XML for the newly created VM (or domain, using libvirt’s terminology) by hand.

The first step is to edit the opening <domain> stanza as follows:

This allows you to encode custom QEMU arguments in the XML definition for the domain.

Next, you need to delete the <cpu> stanza if your XML definition includes one. This is necessary because you’ll be providing custom arguments to QEMU’s -cpu option.

Qemu Kvm Version 7

Inside the <devices> stanza, you’ll need to add the path to your qemu binary in /usr/local/bin.

The required arguments for QEMU are encoded via a <qemu:commandline> stanza, with each argument getting its own <qemu:arg> definition. Place this anywhere inside the <domain> stanza.

This XML corresponds to the following QEMU command-line options:

Test the Intel SGX Enabled VM

Test that Intel SGX functionality is present by starting the virtual machine. If everything is correct, the VM should launch without errors.

If your VM does not start, see the following table for a list of common errors and their possible causes:

Assuming the VM launches properly, you are now ready to create a real guest image and start using Intel SGX in your virtualized environment.

Summary

With the Intel SGX enabled KVM and QEMU distributions it is possible to virtualize Intel SGX on Intel SGX capable hardware. Each guest operating system gains access to the Intel SGX hardware features, and they can be configured independently of one another, whether that be EPC size, access to flexible launch control, or even access to Intel SGX as a whole. Further, these virtual machines can be managed through libvirt, making it possible to integrate Intel SGX enabled guests into existing deployments.

This is still an evolving technology, however, and it is under active development. Virtualizing Intel SGX is appropriate for development systems, testing, and personal workgroup environments, but it is not appropriate for production solutions at the current time.

Further Reading

See the Intel / kvm-sgx and Intel / qemu-sgx project pages on GitHub* for technical information and the latest updates on the Intel SGX virtualization project.

TWEET: Virtualizing Intel® SGX using KVM in the Linux* kernel with QEMU virtual machine monitor to create a VM on your Intel SGX capable hardware that can use Intel SGX in the guest OS.

SUMMARY: Virtualizing Intel® SGX using KVM in the Linux* kernel with QEMU virtual machine monitor to create a VM on your Intel SGX capable hardware that can use Intel SGX in the guest OS.

Centos 7 Qemu-kvm Version