Steps to setup VMware ESX within a KVM VM
Install Linux on a physical system (I used latest Ubuntu 14.04)
Install the qemu/kvm package
apt-get install qemu-kvm
Edit /etc/modprobe.d/kvm-intel (or kvm-amd for AMD cpus) and add
options kvm ignore_msrs=1
options kvm-intel nested=y ept=y
Download qemu-2.0.0 from http://wiki.qemu.org/Download and apply the following patch to hw/i386/pc_piix4.c line 206. Change
pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, xen_enabled(),
pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, TRUE,
Configure the qemu 2.0.0 source with the following command:
./configure enable-kvm target-list=x86_64-linux-user,x86_64-softmmu
Install the bridge-utils package
apt-get install bridge-utils
Configure a network bridge in /etc/network/interfaces as following:
iface br0 inet static
Please notice that your ip-configuration may be different
Choose a directory for the KVM virtual machine and create the virtual machine harddisk with the following qemu-img command:
qemu-img create -f raw esxi00.img 16G
Download VMware ESX Hypervisor 5.5 from the VMware website and put the install iso file (VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso) into the same directory as the harddisk file was created a step before
Start the KVM virtual machine with:
/usr/local/bin/qemu-system-x86_64 -enable-kvm -cpu host -m 4096 -smp 2 -cdrom VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso -netdev tap,helper=/usr/local/libexec/qemu-bridge-helper,id=hostnet0 -device vmxnet3,netdev=hostnet0,id=net0 -hda esxi00.img
Please notice the "vmxnet3" virtual network device type!
Go through the (normal) VMware ESX installation procedure
After the installation enable the SSH service on the ESX Host with is now running with the KVM VM. Then ssh into the ESX Host and set the following two parameters in /etc/vmware/config:
vhv.allow = "TRUE"
Now please reboot the ESX Host system to activate the new configuration!
When creating a virtual machine on the (nested virtualized) ESX Host please make sure to add the following two parameters to the VM's vmx configuration file:
vhv.enable = "TRUE"
vmx.allowNested = "TRUE"
You have successfully virtualized a VMware ESX Hypervisor with KVM!
I hope you now enjoy your fully functional VMware ESX Host
Additional informations on the VMware parameters used in this howto
vhv.allow = "TRUE" does nothing on ESX 5.1 and later
vhv.enable = "TRUE" is only going to be useful if you want to run VMs under the VMs running under ESXi running under kvm
You can add vmx.allowNested = "TRUE" to /etc/vmware/config in the ESXi VM to avoid having to put it in every nested VM's configuration file.
This quick howto was created with informations from https://communities.vmware.com/thread/451412