L2 network over L3 Networks - Xen, Openvswitch & Vyatta

I had been in hibernation while embracing the virtualization trend. In order to understand things better me and my friend designed a little experiment. We had a requirement to fulfill!

Requirement was – 

• On demand connectivity to home network from outside world. 
• Always on connectivity between 2 sites geographically apart
• Connectivity between application virtual machines across 2 sites. There are 2 applications, which run on different operating systems, which are in different networks. 
• Minimum cost

We picked few tools to solve this problem -

- A decent tower PC with reasonable hardware, i3 dual core processor with virtualization support, few gigs of RAM, harddisk and 1 ethernet card.
- Xen hypervisor running on top of Ubuntu Server.
- Openvswitch on Xen hypervisor to emulate switch environment.
- Vyatta - A network operating system for doing NAT, IPSec and routing.
- Few sleepless nights.

I installed Ubuntu server on the PC. Then installed Xen hypervisor over it. Xen provides environment to create, start and stop virtual machines. Note that Xen is a bare metal hypervisor, which means it can run directly over a given hardware without the need of an operating system. However installing Xen over Ubuntu server gave a better control.

Next step was to install Openvswitch. Openvswitch is a virtual switch that controls the network functions of virtual machines. I picked up the latest version of Openvswitch. Thanks to Nicira for providing VxLan support in openvswitch. I had only 1 NIC card, but I needed 2 different networks. So I created 2 dummy bridges, along with 1 bridge attached to eth interface of hypervisor ubuntu machine.

Untill now I have a PC, running ubuntu server, with Xen hypervisor and openvswitch installed. The base was set. Time to spin some VM’s!

I brought up the Vyatta VM. Once the VM was up it had 3 network interfaces. 1 network interface used for management (tied to xen bridge which in turn was tied to eth1 of hypervisor), other 2 for applications (tied to 2 dummy bridges).  Then I created another linux VM. Even the linux VM has 3 networks interfaces, 1 for management and other for applications.

The below picture shows the OVS config on my box.  So eth1 on linux VM, eth1 on Vyatta VM and xenbr1 were able to ping each other. Similarly eth2 on linux VM, eth2 on Vyatta VM  and gre-group bridge were able to ping other. Similarly eth3 on linux VM, eth3 on vyatta vm and vx-lan group were able to ping each other. This means there are three different switches with hosts connected to them.

After that, we configured Vytatta for IPSec. We both had public IP’s from service provider, so IP sec wasn’t a problem. We had to exchange the remote and local endpoints in the IPSec, his remote endpoints were 192.168.200.0/24 & 192.168.202.0/24. The dummy brides on my side were given IP addresses and remote GRE/VxLan end points were configured so as to spawn the L2 network over  different geographies/locations.   However note that there is only 1 physical link coming out of the box. So the IPSec was configured on Xen Bridge, routes were configured on hypervisor to point to Vyatta VM.

My friend created a similar setup at his end.   Once everything was done, the IPSec tunnel was Up. If I ping, from eth2 of linux VM on my setup, eth2 of linux vm on remote setup, I see that ping packet goes out of VM to hypervisor, hypervisor encapsulates it in GRE tunnel and sends it to Vyatta VM, Vyatta VM encapsulates the packet in IPSec tunnel and sends it remote end point. The packet is de-capsulated in reverse order at remote site. Same is the case for VMs eth interfaces hosted on VxLan group.

This shows that an Virtual L2 network, in different geographic locations, can be spawned over L3 network. So,

- I have a home router that I can connect to from any where. I run another IPSec instance to which I connect to when I am connected to un-protected wifi network.
- Site to site IPSec ready between 2 sites.
- VMs in my network can talk to VM's in my friends network. Applications can now communicate.
- All that costs is an cheap tower PC. Thank you Open Source.

Mobile Packet Core – Cloud and Virtualization

Off late there has been a lot of buzz around network virtualization. Software Define networks, Openflow, Nicira acquisition by VMware, Xsigo acquisition by Oracle, Wi-Fi virtualization by AnyFi are few instances.

In the similar lines I was wondering if mobile packet core can be placed in cloud and whether EPC can run on virtual servers rather than on custom built hardware.  As far as I know there is quite some work going on in that direction in the industry. To understand better I tried a little exercise to see if the idea is remotely possible.

One of my colleagues has registered to Amazon Web Services and got an instance of Ubuntu in cloud. That means he owns a virtual machine running Ubuntu which can reached from anywhere in the world by an IP address and a set of encrypted key’s. It seems it is very easy to setup an instance of linux using Amazon web services and machine is up and running in matter of minutes. I, myself, have seen over past few years the physical linux boxes in labs were being replaced with virtual machines. So virtualization is not a new thing after all. But providing a virtually instance in cloud, dynamically, and resources (Memory, network cards, processors) being allocated based on the need is something new and interesting. So my colleagues Ubuntu instance was up and running.

I took my GTP code, SGW, that I wrote years ago and ran it on the server.  MME was running on my local machine. Couple of firewall exceptions on server and boom, my local machine and ubuntu on cloud was nicely exchanging the GTP packets. Well, it’s a client server communication you don’t need to be an Einstein to make it work; as long as IP’s are reachable everything is supposed to work. So, here it is my first GTPv2 call over the cloud onto a virtual server.

It does make me believe that mobile packet core can be pushed to cloud and onto virtual machines. There are definite challenges, but over the years it seems to be possible. Imagine bringing up more virtual instances of EPC components during the peak hours and shutting them down during the nights. If there are bunch of machines that are going to send one time information during a particular time of day, we can get the virtual instances ready to take the explosion of messages and bring down the instances once the information is exchanged. Resource can be dynamically allocated based on need.

What do you think of this? Does this seem to be a viable solution?