Install VMWare ESXi in KVM

Steps to setup VMware ESX within a KVM VM

Install Linux on a physical system (I used latest Ubuntu 14.04)
Install the qemu/kvm package

apt-get install qemu-kvm

Edit /etc/modprobe.d/kvm-intel (or kvm-amd for AMD cpus) and add

options kvm ignore_msrs=1
options kvm-intel nested=y ept=y

Download qemu-2.0.0 from http://wiki.qemu.org/Download and apply the following patch to hw/i386/pc_piix4.c line 206. Change

pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, xen_enabled(),
0x4);
to
pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, TRUE,
0x4);

Configure the qemu 2.0.0 source with the following command:

./configure enable-kvm target-list=x86_64-linux-user,x86_64-softmmu

Create /usr/local/etc/qemu/bridge.conf

allow all

Install the bridge-utils package

apt-get install bridge-utils

Configure a network bridge in /etc/network/interfaces as following:

auto br0
iface br0 inet static
address 192.168.88.135
netmask 255.255.255.0
network 192.168.88.0
broadcast 192.168.88.255
gateway 192.168.88.1
dns-nameservers 192.168.88.1
bridge_ports eth0
bridge_fd 0
bridge_hello 2
bridge_maxage 12
bridge_stp off

Please notice that your ip-configuration may be different

Choose a directory for the KVM virtual machine and create the virtual machine harddisk with the following qemu-img command:

qemu-img create -f raw esxi00.img 16G

Download VMware ESX Hypervisor 5.5 from the VMware website and put the install iso file (VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso) into the same directory as the harddisk file was created a step before

Start the KVM virtual machine with:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -cpu host -m 4096 -smp 2 -cdrom VMware-VMvisor-Installer-5.5.0.update01-1623387.x86_64.iso -netdev tap,helper=/usr/local/libexec/qemu-bridge-helper,id=hostnet0 -device vmxnet3,netdev=hostnet0,id=net0 -hda esxi00.img

Please notice the "vmxnet3" virtual network device type!

Go through the (normal) VMware ESX installation procedure
After the installation enable the SSH service on the ESX Host with is now running with the KVM VM. Then ssh into the ESX Host and set the following two parameters in /etc/vmware/config:

vhv.allow = "TRUE"
hv.assumeEnabled="TRUE"

Now please reboot the ESX Host system to activate the new configuration!

When creating a virtual machine on the (nested virtualized) ESX Host please make sure to add the following two parameters to the VM's vmx configuration file:

vhv.enable = "TRUE"
vmx.allowNested = "TRUE"

Congratulations!
You have successfully virtualized a VMware ESX Hypervisor with KVM!

I hope you now enjoy your fully functional VMware ESX Host

Additional informations on the VMware parameters used in this howto

vhv.allow = "TRUE" does nothing on ESX 5.1 and later
vhv.enable = "TRUE" is only going to be useful if you want to run VMs under the VMs running under ESXi running under kvm
You can add vmx.allowNested = "TRUE" to /etc/vmware/config in the ESXi VM to avoid having to put it in every nested VM's configuration file.

This quick howto was created with informations from https://communities.vmware.com/thread/451412