L2 network over L3 Networks - Xen, Openvswitch & Vyatta

I had been in hibernation while embracing the virtualization trend. In order to understand things better me and my friend designed a little experiment. We had a requirement to fulfill!

Requirement was – 

• On demand connectivity to home network from outside world. 
• Always on connectivity between 2 sites geographically apart
• Connectivity between application virtual machines across 2 sites. There are 2 applications, which run on different operating systems, which are in different networks. 
• Minimum cost

We picked few tools to solve this problem -

- A decent tower PC with reasonable hardware, i3 dual core processor with virtualization support, few gigs of RAM, harddisk and 1 ethernet card.
- Xen hypervisor running on top of Ubuntu Server.
- Openvswitch on Xen hypervisor to emulate switch environment.
- Vyatta - A network operating system for doing NAT, IPSec and routing.
- Few sleepless nights.

I installed Ubuntu server on the PC. Then installed Xen hypervisor over it. Xen provides environment to create, start and stop virtual machines. Note that Xen is a bare metal hypervisor, which means it can run directly over a given hardware without the need of an operating system. However installing Xen over Ubuntu server gave a better control.

Next step was to install Openvswitch. Openvswitch is a virtual switch that controls the network functions of virtual machines. I picked up the latest version of Openvswitch. Thanks to Nicira for providing VxLan support in openvswitch. I had only 1 NIC card, but I needed 2 different networks. So I created 2 dummy bridges, along with 1 bridge attached to eth interface of hypervisor ubuntu machine.

Untill now I have a PC, running ubuntu server, with Xen hypervisor and openvswitch installed. The base was set. Time to spin some VM’s!

I brought up the Vyatta VM. Once the VM was up it had 3 network interfaces. 1 network interface used for management (tied to xen bridge which in turn was tied to eth1 of hypervisor), other 2 for applications (tied to 2 dummy bridges).  Then I created another linux VM. Even the linux VM has 3 networks interfaces, 1 for management and other for applications.

The below picture shows the OVS config on my box.  So eth1 on linux VM, eth1 on Vyatta VM and xenbr1 were able to ping each other. Similarly eth2 on linux VM, eth2 on Vyatta VM  and gre-group bridge were able to ping other. Similarly eth3 on linux VM, eth3 on vyatta vm and vx-lan group were able to ping each other. This means there are three different switches with hosts connected to them.

After that, we configured Vytatta for IPSec. We both had public IP’s from service provider, so IP sec wasn’t a problem. We had to exchange the remote and local endpoints in the IPSec, his remote endpoints were 192.168.200.0/24 & 192.168.202.0/24. The dummy brides on my side were given IP addresses and remote GRE/VxLan end points were configured so as to spawn the L2 network over  different geographies/locations.   However note that there is only 1 physical link coming out of the box. So the IPSec was configured on Xen Bridge, routes were configured on hypervisor to point to Vyatta VM.

My friend created a similar setup at his end.   Once everything was done, the IPSec tunnel was Up. If I ping, from eth2 of linux VM on my setup, eth2 of linux vm on remote setup, I see that ping packet goes out of VM to hypervisor, hypervisor encapsulates it in GRE tunnel and sends it to Vyatta VM, Vyatta VM encapsulates the packet in IPSec tunnel and sends it remote end point. The packet is de-capsulated in reverse order at remote site. Same is the case for VMs eth interfaces hosted on VxLan group.

This shows that an Virtual L2 network, in different geographic locations, can be spawned over L3 network. So,

- I have a home router that I can connect to from any where. I run another IPSec instance to which I connect to when I am connected to un-protected wifi network.
- Site to site IPSec ready between 2 sites.
- VMs in my network can talk to VM's in my friends network. Applications can now communicate.
- All that costs is an cheap tower PC. Thank you Open Source.