Configure VM Network access to IPoIB Network using 1-to-1 NAT
This guide explains how to configure VM access to an Infiniband Network from within Open Nebula using a 1-to-1 NAT. VMs generally do not play very well with IPoIB because the IPoIB driver do not support MAC bridging; one alternative is to use IP routing. IP routing requires complicated routing rules to support VM migration and access from other hosts on the Infiniband network. 1-to-many NAT can also be used to provide VMs access to the IB network however a lot of protocols do not play well with 1-to-many NAT and Destination NAT configuration is complicated. 1-to-1 NAT provides a single external address for every internal address. Open Nebula vmm driver architecture allowed us to control iptables and ip alias rules during the vm life cycle. The drivers will add or remove ip aliases on the IPoIB links and iptables nat rules for the VMs. The guide assumes users have extensive knowledge on how vmm drivers work. The drivers make a lot of assumptions about the environment.
Prerequises and Limitations
1. Unzip and copy files to /var/lib/one/remotes/vmm/kvm-ib (files are not linked to the wiki yet, waiting for a procedure)
2. Update oned.conf and add the following
VM_MAD = [ name = "vmm_kvm_ib", executable = "one_vmm_exec", arguments = "-t 15 -r 0 kvm-ib", default = "vmm_exec/vmm_exec_kvm.conf", type = "kvm" ]
3. Configure host network and NAT rules.
First make sure the IB device is configured as connected mode with mtu set to 64K, to provides the highest throughput.
Its best to show this by example.
Example configuration, a /29 network on a different subnet than the IB network
IPoIB network = 192.168.10.0/24 Guest IB range = 18.104.22.168/29 (this must be within the IB network range) IB device = ib0 VM bridge = virbr1
Create a host only network in libvirtd with 172.16.100.89 as gateway on all the hypervisors. The IP range would be split into the following..
1-to-1 NAT host range
Libvirt XML file for this 'host only' network example (libvirt uses the term 'host only' for networks it creates without a route entry)
<network> <name>ibnat0</name> <uuid>ddd423b1-0317-ad2b-3b50-a144b73d9a3a</uuid> // replace with unique uuid <bridge name='virbr1' stp='on' delay='0' /> <mac address='52:54:00:E8:D3:25'/> // replace with unique mac address <ip address='172.16.100.89' netmask='255.255.255.248'> </ip> </network>
IP tables rules configuration.
Rules can be added and removed with the drivers along with the IP aliases,
Example Script for /29 network pool with virbr1 bridge, first IB tables is cleaned out and MTU is set to 64K for the bridge and dummy device. Bridge MTU cannot be configured without having atleast one device connected to it which is why libvirt creates a TAP device virbr1-nic and assigns it to the mac bridge virbr1.
iptables -F iptables -t nat -F ip link set virbr1-nic mtu 65520 ip link set virbr1 mtu 65520
One important note, it isn't strictly required to have a rule for every IP address. The nat rule below is perfectly legal; however the behavior wont be as expected, 172.16.100.90 may nat to 192.168.100.91 instead of 192.168.100.90. It will still be one-to-one nat because the number of source ips is less than or equal to the nat ip pool. We found to have maximum application compatibility its best create a rule for every IP address through open nebula drivers.
(perfectly legal NAT rules but don't use it if you want VM to VM connectivity over IB)
iptables -t nat -A POSTROUTING -s 172.16.100.88/29 -o ib0 -j SNAT --to-source 192.168.10.90-192.168.10.95 iptables -t nat -A PREROUTING -d 192.168.10.88/29 -j DNAT --to-destination 172.16.100.90-172.16.100.95
Create an open nebula network within the range of the guest IB range
4. Network Configuration guest side
5. IB Configuration host side