Infiniband Drivers OpenNebula 3.4
- Libvirt must be installed and used. - The driver parses KVM specific XML files, only KVM has been tested. - IP tables with NAT support is required. - The driver has only been tested with shared and qcow2 transfer managers. - The driver assumes the default installation path, /var/lib/one/ - Configuration only tested on RHEL6/CENTOS6 and Ubuntu 11.10/12.04 - Requires Open Nebula 3.4.X - Requires passwordless sudo and ssh equivalency between hypervisor nodes for oneadmin (admittedly large security risk but I am not aware of an alternate solution at the moment)
1. Unzip and copy files to /var/lib/one/remotes/vmm/kvm-ib
2. Update oned.conf and add the following
VM_MAD = [ name = "vmm_kvm_ib", executable = "one_vmm_exec", arguments = "-t 15 -r 0 kvm-ib", default = "vmm_exec/vmm_exec_kvm.conf", type = "kvm" ]
3. Configure host network and NAT rules.
First make sure the IB device is configured as connected mode with mtu set to 64K, to provides the highest throughput.
Its best to show this by example.
Example configuration, a /29 network on a different subnet than the IB network
1-to-1 NAT host range
172.16.100.90 -> 192.168.10.90 172.16.100.91 -> 192.168.10.91 172.16.100.92 -> 192.168.10.92 172.16.100.93 -> 192.168.10.93 172.16.100.94 -> 192.168.10.94
IPoIB network = 192.168.10.0/24 Guest IB range = 126.96.36.199/29 (this must be within the IB network range) IB device = ib0 VM bridge = virbr1
Create a host only network in libvirtd with 172.16.100.89 as gateway on all the hypervisors
Libvirt XML file for this host only network example
<network> <name>ibnat0</name> <uuid>ddd423b1-0317-ad2b-3b50-a144b73d9a3a</uuid> <bridge name='virbr1' stp='on' delay='0' /> <mac address='52:54:00:E8:D3:25'/> <ip address='172.16.100.89' netmask='255.255.255.248'> </ip> </network>
IP tables rules configuration.
Rules can be added and removed with the drivers along with the IP aliases,
Example Script for /29 network pool with virbr1 bridge, first IB tables is cleaned out and MTU is set to 64K for the bridge and dummy device. Bridge MTU cannot be configured without having atleast one device connected to it which is why libvirt creates a TAP device virbr1-nic and assigns it to the mac bridge virbr1.
iptables -F iptables -t nat -F ip link set virbr1-nic mtu 65520 ip link set virbr1 mtu 65520
One important note, it isn't strictly required to have a rule for every IP address. The nat rule below is perfectly legal; however the behavior wont be as expected, 172.16.100.90 may nat to 192.168.100.91 instead of 192.168.100.90. It will still be one-to-one nat because the number of source ips is less than or equal to the nat ip pool. We found to have maximum application compatibility its best create a rule for every IP address through open nebula drivers.
(perfectly legal NAT rules but don't use it if you want VM to VM connectivity over IB)
iptables -t nat -A POSTROUTING -s 172.16.100.88/29 -o ib0 -j SNAT --to-source 192.168.10.90-192.168.10.95 iptables -t nat -A PREROUTING -d 192.168.10.88/29 -j DNAT --to-destination 172.16.100.90-172.16.100.95
Create an open nebula network within the range of the guest IB range
4. Network Configuration guest side
5. IB Configuration host side