Using the Brocade Vyatta Controller EMS App

As part of our continuing work for NetDEF, we continue to install, setup and test the latest SDN controller tools from companies such as Brocade.

The Brocade Vyatta Controller (BVC) v1.2 is based on OpenDaylight Helium SR2 (the current release is SR3) and comes enabled with certain features that are not enabled by default in OpenDaylight such as OpenFlow and a GUI interface.

For our testing we utilized BVC, a pair of Brocade Vyatta vRouter 5600s, and the Brocade vRouter 5600 EMS App for BVC.

In the diagram you can see that we have attached a 5600 to each of the OpenFlow switches.

We followed the installation documentation from Brocade without issue, first installing BVC 1.2, then adding the Path Explorer app and finally the 5600 EMS app.

Once installed you can log into the controller on port 9000 and you should see the following toolbar on the lefthand side


Note the addition of the Vyatta vRouter 5600 EMS link.  Clicking on the link gives you the following display:

Screenshot 2015-04-27 09.05.39

Here you can see that we have already added the two vRouter 5600’s show in our diagram.  The main configuration change we needed to make on the vRouters was to enable netconf and add the netconf-ssh port of 830 to the configuration

We then selected the two vRouters (R1 and R2) and clicked on the “Create Tunnel” box and waited for about two minutes while the system built the tunnels.

Screenshot 2015-04-26 19.09.57


Screenshot 2015-04-26 19.16.38


Looking on the vRouters we saw the following configuration had been added

Obviously the pre-shared-secret being “vyatta” is a bit concerning but since we are aware of it, we can fix it manually.

The EMS app does exactly what it says it will do, configure tunnels between multiple vRouters.

One thing to note is that the EMS app is still limited, for example it does not allow us to configure which interfaces are used for the tunnel interfaces, configure a pre-shared-secret, etc.  We found the EMS app useful for creating the configurations for the vRouters that can be modified to fit your needs.

Also, just like the standard OpenDaylight dlux GUI, the Brocade GUI appears to still use the D3 gravity javascript code to display the network topology, which is pretty but can be hard to work with.

Thank you as always to Brocade and especially Lisa Caywood for being our contact and providing the software (BVC, vRouter 5600’s) necessary to do the testing.

Testing Vyatta 6.5 R1 Under VMWare – Preliminary Results

Testing Vyatta Subscription Edition 6.5 R1

We here at SDN Testing have been working with Router Analysis, Inc. our parent company doing testing of Vyatta Subscription Edition 6.5 R1 under VMWare.  Testing of Vyatta on hardware is located on Router Analysis.

For the VMWare setup we ran VMWare Hypervisor v5.1 on our spec setup as built by IXSystems:

SuperMicro X9SAE-V
Intel I7-3770 / 32G ECC RAM
Four Intel I340-T2 NICS (8 Ports Total)
Intel 520 Series 240GB SSD

We setup one of the Intel I340-T2 NICs using VMDirectPassThrough and utilized it for the upstream ports to the network.  The other 3 NICs were setup using VMWares default vSwitch configuration, each port was in it’s own vSwitch.  Those six ports were connected to another system generating packets while two VMs were created on the local machine to finish the total of eight.

Vyatta was given a VM with 2vCPUs and 4GB of RAM.

The following diagram shows the setup.

From previous tests, which will be included in the full report, we knew that each vSwitch port can forward 23% Line-Rate IMIX traffic (Tri-Modal: 7 x 64 bytes, 4 x 512 bytes, 1 x 1518) when 8 ports are in use.

Therefore each tenant was configured to send 234 Mbps of traffic outbound through the two uplink ports for a total of 1.9 Gbps.

The next steps were to configure features:

uRPF, ACLs and QoS (Shaping and Policing)

There was no impact from the features as the traffic limitation appeared to have nothing to do with Vyatta and more to do with the VMWare setup.  We were able to send IMIX traffic without issue out the two uplink interfaces at 1.9 Gbps total.

From our testing, we have concluded that Vyatta Subscription Edition 6.5 R1 behaves as expected when used as a multi-tenant virtual router.  Easily supporting the traffic and features needed in the role.

Note: We are planning to test the different vSwitches available to VMWare in the future, if we find one that behaves better we will re-run these tests.

What is Commodity Hardware?

Commodity Hardware is not the old machine you found under a desk..

You will often see “runs on commodity hardware” on a SDN product website.  What does it really mean?

Sometimes when Router Analysis gets a new piece of SDN software to test the results come out much better or worse than what the vendor claims.  The most recent occurrence of lower performance happened as we started testing Vyatta’s R6.5 in our lab.

When testing you need to have an initial benchmark that you can use to tell you what effect features and other changes you make to the system configuration have.  For Vyatta we took a simple approach.  Fire up the LiveCD image and run traffic across 2 GbE ports.

Router Analysis previously interviewed Vyatta spokesman Scott Sneddon and found out that the performance of a Vyatta system could be as high as 2Mpps (2 Million packets per second) at 64 bytes.  Our first run produced 1.78Mpps.  After a short Twitter discussion, it was determined that the number was expected with the hardware we are using (Intel i7-3770 CPU, 32G ram, Intel c216 chipset and Intel Ethernet Interface cards).

We decided to do a little testing to see what the bottleneck was and how we could reduce it.  As we debugged the issues and changed the hardware setup we got the Vyatta system up to 2Mpps.  Adding in another pair of ports and doing more tweaking we got to 2.9Mpps then early this morning 3.6Mpps.  3.6Mpps is a really good number for a software based forwarding router.  To compare, the Cisco 7200 VXR with a NPE-G2 can do 2Mpps and the 7200 VXR is a very specifically designed system with a proprietary OS designed for one thing, routing.

Besting the Cisco 7200 with PC hardware is pretty good, but doing it with PC hardware that is forwarding at almost 2x the pps and at the time of this writing 4x the bandwidth (we have tested up to 6Gbps on the Vyatta, the 7200 supports 1.8) is impressive.

Cost wise, the test system we are using is not that expensive.  Custom built for us by IXSystems the system is based on a Supermicro motherboard with the Intel c216 chipset, a QuadCore Intel i7-3770 CPU and 6 PCie slots.  Since we outfit the machines with 32GB of ECC ram, it is easy to take the difference in memory cost and apply it towards the necessary network cards.  Fully configured the system would come out well below $2000 US (or less if you source the NICs on the used market).

What is commodity hardware?  It’s still new hardware and it’s still good quality.  If you want to just throw together something from parts you find around the office, you can.  But if you intend to have a reliable and functional system, you need to do the hardware design properly.

Evaluating Midokura’s MidoNet Solution

This article is the first in a set of articles that will walk through the evaluation of Midokura’s MidoNet product.  In the first article we will discuss Midokura’s solution, what it is made of, how it works and what expectations have been set with regard to performance and the solution it aims to solve.

Midokura Logo

An Overview of Midokura:

SDN Startup Midokura launched this week at the OpenStack conference in San Diego and has gained a lot of attention in the last week or two.  If you are looking for a good write up on Midokura’s Midonet solution check out Shamus McGillicuddy‘s article on TechTarget called “Midokura network virtualization

Ivan Pepelnjak did a nice technical write up back in August titled “Midokura’s Midonet Layer 2-4 Virtual Network Solution

The Testing View:

In order to build a complete test plan we must first understand the parts of the solution.

By reading articles and looking at the Midokura website, we came up with a good idea of what Midokura was doing.

Midokura Midonet offers a way for companies solve complex network scaling issues using commodity PC hardware.  Midonet creates a fully-meshed overlay network on top of an existing IP network.

MidoNet is a collection of different components:

  • The MidoNet Agent, which runs on each node, fully processing packets as they enter the network and making sure they are delivered to the right host.
  • A routing component based to handle L3 packets (something Nicira’s solution does not offer)
  • A fast, distributed database to keep all of the flow, forwarding, filtering and other data needed to create the virtual topology.
  • Tools to keep everything in synchronization and flowing.

Once we finished the paper evaluation, we reached out to Midokura to get more specific information.

Discussion with Midokura:

We were lucky to be able to grab about 30 minutes of Ben Cherian’s (CSO of Midokura) time on Friday night, only two days before the 2012 OpenStack conference.  As our ultimate goal is to test the MidoNet product, we need a good understanding of MidoNet.  From our discussion we learned the following:

Lab tests have shown a fully utilized 10GE Interface driven by a single core from a multi-core processor.

In MidoNet, Midokura uses some well known Open Source products.  For routing, Midokura uses Quagga.  For the databases Apache Cassandra.  The underlying vSwitch is Open vSwitch and Apache Zookeeper is used to keep everything in sync.

GRE was given as an example in the virtual network overlay as it has been tested and is known to work.  Other protocols could and probably are supported but have not had enough lab time to be called supported.

Evaluating MidoNet Part 1:

As we break down MidoNet’s design we can surmise that as new flows enter the network, the lookup on the first packet should take the longest. Once the first packets have been processed and a flow created, other packets matching the same flow should have a lower latency.

As all inbound packets must pass through the edge, the test plan should look at the time it takes for new packet flows to be inspected, looked up and installed into the flow table. This can be done by measuring the latency of packets crossing the system and comparing the first packet to the later ones once the latency has stabilized. Initially no features should be enabled.

Once a base measurement has been taken and verified, features such as packet filtering should be enabled.  The features should be applied and extended, for example with packet filtering it would start with simple IP destination address filtering.  IP source address, source port, destination port, window size and other knobs should be enabled and measurements taken to see how each new addition affects the lookup time and possibly forwarding performance.

After the base and IP filtering tests are complete you can evaluate the results to build more specific test cases that push MidoNet.

In the next article we will talk more about the MidoNet design what we know about its underlying components and how to test scale.

Network Hardware and SDN

How Does SDN Fit Into The Virtual Data Center?

One thing that needs to be cleared up is the definition of  Network or Networking Hardware.  In the definition of SDN from ONF they discuss the decoupling of the Control and Data Plane with the Data Plane being defined as Network Hardware.  Here is where things can get confusing.

What is Network Hardware?

Wikipedia says the following :

Networking hardware or networking equipment typically refers to devices facilitating the use of a computer network. Typically, this includes gatewaysroutersnetwork bridgesswitcheshubs, and repeaters. Also, hybrid network devices such as multilayer switchesprotocol convertersbridge routersproxy serversfirewallsnetwork address translatorsmultiplexersnetwork interface controllerswireless network interface controllersmodemsISDN terminal adaptersline driverswireless access pointsnetworking cables and other related hardware.”

Essentially anything that is not a end system is network hardware.  The current reality of SDN is that it tends to mean Programmable Switches when it says Network Hardware.  Switches are generally made built on fabrics that allow ports to transmit traffic to and from other ports.

If we were to be true to the current SDN message, we would only look at Programmable Switches.  Reality is there are other ways you can create a Data Plane i.e. Network Hardware.  One of these ways is using a Network Processor (which I covered in the earlier article on Vyatta).

A Computer with a few (or many) Interfaces and a Network Processor is what a Router is.  We can easily see this by looking at the design of Juniper Networks Routers.

This is one of the interesting things I see in the SDN space: Companies that can take advantage of generic hardware and add value.   These companies will create more tools for the architects and operators to use when pushing packets.

We plan to cover as many SDN related topics as possible here at SDN Testing, those that exist today and those that will come in the future.

The Vyatta Cloud Router Story

Vyatta and their approach to Cloud Routers

A few weeks ago I had the pleasure of speaking with Scott Sneddon, Cloud Solutions Architect at Vyatta Inc.  I’ve known Scott since the late 1990’s when he and I both worked for Exodus Communications.

Vyatta is one of the few full featured software based routing vendors in the market today.  Their product is a mix of OpenSource and proprietary software combined together creating a router that can not only live in the cloud but will in the future be able to utilize some of the hardware such as Intel’s Sandy Bridge (and later generation Ivy Bridge) processors as Network Processors.

Network Processors are key to hardware forwarding routers such as the Juniper T series and the Cisco Carrier Routing System allowing them to perform forwarding and features at line rate, something that routers using software based forwarding struggle with.  To get a better picture of software vs hardware forwarding you can read Router Analysis’ Enterprise Edge Router Upgrade Guide where I discuss the Cisco 7200 which uses a software forwarding engine and compare it with higher performance routers with hardware forwarding capabilities.

Vyatta offers a full featured router solution by including VPN, Firewall and other features normally found in hardware locked solutions in their software product.  I feel that Vyatta has a jump on other vendors in the True Virtual Data Center space.  One of the most important parts of the Virtual Data Center is the router and it’s ability to perform equal to or greater than the hardware based router it is replacing.  Using software forwarding alone Vyatta claims to be able to handle up to 2Mpps, which depending on packet size can easily be multiple gigabits of traffic.  In testing Vyatta is seeing up to 11Mpps using an Intel Sandy Bridge processor as a network processor.

A quick note about integrated firewalls: While software firewalls contained within the same hardware as the routers, switches and/or hosts are very useful, they are not a replacement for hardware firewalls.  In security (which I do not claim to be an expert at) the separation of networks using physical links is key.  There is some great information available in this thread on the Cisco support forums where they are discussing the ASA 1000V.

Vyatta keeps a tight relationship with the OpenSource community by hosting where you can find free versions of Vyatta’s Core Software along with community support, documentation and forums.

SDN Testing, the software defined side of Router Analysis plans to put the Vyatta product through rigorous testing in the coming weeks.

The Virtual Data Center Reality

Virtual Data Centers start to become reality.

Previously posted on Router Analysis

With the recent announcement of the CSR 1000v from Cisco, there are now two commercial Virtual Data Center stories (three if we look at the VMWare vCNS products and use one of the other vendors products for a router) Cisco and Vyatta.

What is a Virtual Data Center?  There will be a lot of different answers but in my view it consists of the following:

  • A pair of redundant Routers with multiple provider uplinks
  • A pair of redundant Firewalls
  • A pair of Load Balancers
  • Front and Backend Servers

In my previous life designing and building Internet Data Centers we would have build this entire setup out of separate parts taking up an entire rack or two.  Now it could be done in a single blade server with multiple redundant power supplies or a pair of highly spec’d servers.

Now, I want to be clear here: I don’t think that the software based Firewalls are up to the task of the hardware based ones.  I think most security companies/consultants would agree that there is a danger when you host both your servers and your firewall on the same shared hardware.  You could design the setup in a way that the ASA is only hosted on it’s own blade(s) but there is still the inherent risk of a misconfiguration or privilege escalation hack allowing someone to bypass the firewall.

Sadly the way around the security issue is to put a physical firewall in the line.  This can be easily done, so it’s mainly just a CapEx issue.

For routers at this time, we only have Cisco and Vyatta commercially.  They both are offering strong products but Cisco’s CSR 1000v is more feature rich supporting many protocols and features that come from using the previously designed and coded Cisco code base.

In the coming weeks and months I am going to be writing about the products available in the space and what limitations still need to be overcome.  I will be working with Cisco, Vyatta, VMWare and others to try and compile as much information as possible.

Summary: The Virtual Datacenter is here.  It’s not perfect, but all of the parts exist from multiple vendors.  The world of Virtualization just got a lot more interesting.

What are your thoughts?

What is SDN and What are we Testing?

SDN Testing

SDN stands for “Software Defined Network”, a simple name with thousands of different meanings.  As defined by WikiPedia “SDN separates the control plane from the data plane in network switches and routers. Under SDN, the control plane is implemented in software in servers separate from the network equipment and the data plane is implemented in commodity network equipment.”

The most important aspect of SDN is the separation of the control and data planes.  This decoupling allows end users (Service Providers, Enterprises) to use commodity hardware to build and expand their networks.

Some of the major players in the SDN space are Vyatta, Cisco, Big Switch and Nicira.  Nicira was recently purchased by VMWare and is being merged into VMWares core product offerings.

Vyatta offers a Quagga based software router with firewall and VPN support.  I recently talked with Vyatta and found their vision and commitment to the Open Source community great.  Vyatta is currently the top player when it comes to software defined routers.

Cisco offers many of the parts needed to create SDNs but some parts have not been released yet.  Cisco has released the Nexus 1000v software switch, the ASA 1000v software firewall and has announced the CSR 1000v IOS XE based software router.  Once Cisco gets the full solution out, they have a chance to leapfrog over the competition due to their history and ability to re-use their current software features.

Big Switch offers an SDN Controller.  Currently they are offering Floodlight, a Open Source version of their product with promises of a commercial version coming soon.

The last company I will cover is Nicira.  Nicira provides what they call NVP, or the Nicira Virtualization Platform.  They combine their software with the Open Source OpenVSwitch to provide a fully software controlled and forwarded network.