Building a SDN Based Private Cloud in an Hour

UPDATE: This post has been tremendously popular but as OpenStack has progressed the instructions no longer work. It is still worth a read because it gives you an idea of how OpenStack and Floodlight interact. I’ll be posting new instructions once OpenStack Havana is released (mid-October)

There is a lot of interest in building private cloud based on OpenStack and software defined networking (SDN).  Unfortunately, getting started can be expensive and painful,  requiring serious time and equipment.  Getting your executive team to sign off on a pilot can be difficult without showing flashing lights and whirling gizmos, which can be tough to do without the time or equipment.  I’ll show you how to get OpenStack and OpenFlow based SDN up and running in an hour using just your laptop, so you can get your organization excited about private cloud.

Gather Your Gear 

The hardware and software you need to get going is fairly basic.  I’ll be using my 13″ Macbook Pro using VMware Fusion (yes, I see the irony here) to host my OpenStack and SDN controller VMs.  You can use whatever hypervisor you like, so VMware is by no means a requirement.

To get OpenStack up and running I’m going to use the DevStack distribution.  DevStack is an integrated OpenStack distribution designed for developers to get OpenStack up and running quickly.  For the SDN controller I’ll be using Floodlight.  Floodlight is an open source Java based OpenFlow controller that also has a plugin for OpenStack’s Quantum API.  This will enable networks on demand, which is one of the core promises of SDN.  Both DevStack and Floodlight are free to download and easy to install, making them the perfect cloud playground.

First things first; lets grab Ubuntu to install OpenStack on.  I highly recommend Ubuntu Server 12.04 64-bit, mainly because that is what I have tested on, but also because I’m going to be using some very handy install scripts that were designed for Ubuntu.  In addition to Ubuntu, you will need to download  Floodlight.  I recommend the prebuilt VM form factor (v0.85 is fine) because you can simply import it into your hypervisor with no fuss.

Once you have the Ubuntu Server ISO, go ahead and create a new VM in your hypervisor.  I recommend giving it as much memory as you can spare with 4GB good for getting a few VMs going.  Knowing that disk is a premium on a laptop a 20GB thin provisioned disk is more than enough.  Before you install Ubuntu on the newly created VM you will need to modify the VMs .vmx file.  You will be running a hypervisor (KVM) inside of another hypervisor (VMware Fusion) and Fusion needs the appropriate config to support this.  Your .vmx file is stored with your VM (usually Document/Virtual Machines/.  Modify the .vmx file with the following:

add the line:   vhv.enable = “TRUE”
modify the line: vcpu.hotadd = “TRUE” to vcpu.hotadd = “FALSE”

If you are using a hypervisor other than VMware Fusion I suggest you check if there are any restrictions to running KVM within that hypervisor before you move to the next step.

Install Ubuntu will the default setting.  There is no need to install any of the optional packages that the Ubuntu installer offers.  All of the packages you need will be automatically installed later.  Ubuntu will need to download OpenStack so ensure the IP you assign to Ubuntu has Internet connectivity.

Your Own Private Cloud  

Before you start the installation and setup of OpenStack and Floodlight, it is worth taking a few minutes to understand how OpenStack and Floodlight will interact.

OpenStack has the concept of projects and each project has one or more associated networks.  OpenStack has two models for provisioning networks; Nova and Quantum.  Quantum is the future direction of OpenStack networking and Nova networking is maintained for legacy support but will be deprecated over time.  In the case of Floodlight OpenStack will provision networks by calling Quantum.  Quantum, through a plugin, then calls Floodlight, which programs the Open vSwitch using the OpenFlow protocol.  Clear as mud!

This architecture is designed so that OpenStack can make basic, high level network calls (add, delete, modify, etc) without making any assumptions on how to achieve the end result.  This is perfect because many of the networking concepts in SDN based solutions are different than traditional networks (OpenFlow solutions even more so), removing many of the barriers presented by existing networks.  This is one of the many reasons SDN is so attractive for cloud based deployments.

With that out of the way, let’s get started.

Power on the Floodlight VM and login using the user name ‘floodlight’.  By default the Floodlight VM will use DHCP.  If you need to change to a static IP now is a great time.  With Floodlight running you will need to enable the Quantum support.  There are two components; the Floodlight OpenStack plugin called RestProxy  (we will install this later with OpenStack), and the Floodlight Quantum API, which we must enable on Floodlight itself.

On the Floodlight VM issue the following command to enable Quantum support and reboot for the change to take effect.

$ touch /opt/floodlight/floodlight/feature/quantum
$ sudo reboot now

Once Floodlight reboots you can verify that the Quantum API is working by calling against the API.  You should see a status ‘ok’ return, which indicates the API is enabled.

$ curl

Before you start the install of OpenStack on Ubuntu you should verify that the Ubuntu VM can at least ping the Floodlight VM.  If there is a problem at this point it is most likely due to incorrect IP assignment or problems in your hypervisor networking configuration.  The two VMs need to communicate so make sure there are no issues here before you proceed.

Installing OpenStack and the rest of its dependancies is dramatically simplified because of some great work by the DevStack team and a handy little script written by the Floodlight team.  DevStack itself makes installing OpenStack a simple process but the Floodlight team has written a comprehensive script which installs DevStack, the Open vSwitch (an OpenFlow enabled virtual switch), Quantum, and the RestProxy plugin.  On top of that the script configures them all to work together.  This alone can save you hours of manual work and troubleshooting.

From your Ubuntu VM download the Floodlight teams DevStack install script and start the install.  Because the script is downloading/installing DevStack and a number of other components it can take 20-30 minutes for this to complete.  Make sure to enter the IP address of your Floodlight controller as shown below when executing the script.  The script will use this IP and port number to automatically configure OpenStack to talk to Floodlight.

$ wget
$ sudo chmod +x
$ sudo ./ <Floodlight IP> 8080

Once the OpenStack install is complete, do a quick reboot.

$ sudo shutdown -r now

Now that OpenStack is installed you are almost ready to go.  One important thing you should know about the DevStack distribution is that it was purposely designed to be non-persistant. Every time you reset your Ubuntu VM OpenStack will reset back to default.  VMs, tenants, virtual networks, etc will all be lost.  This helps developers test, reset, and test some more very easily.  It’s also great for doing demos of private cloud and resetting after your done, without undoing everything manually.

Starting OpenStack is straight forward.  Simply issue the follow command on your Ubuntu VM

$ cd devstack
$ sudo ./

You will see a flurry of activity as OpenStack is configured.  It should only take a couple of minutes to start and you should see a message saying Horizon (the OpenStack web portal) is running when it is finished.

OpenStack is Running

Now that OpenStack is running you can login to Horizon, with the demo/nova credentials, by browsing to the IP of your Ubuntu VM.  Congrats, your private cloud is now up and running!

OpenStack Horizon Login Screen

A Walk in the Cloud

In the DevStack default configuration, there are two users (‘admin‘ and ‘demo‘) both with the password ‘nova‘.  The admin user can create new projects, spin up new VMs for existing projects and overall manage the OpenStack implementation.  The demo user can spin up VMs within two projects to which it has access to.

To create a new VM login as the ‘demo’ user and select the ‘demo’ project from the left hand side of the screen.  Click on ‘Images and Snapshots’ and click the ‘Launch’ button net to the ‘cirros-0.3.0-x86_64-uec’ image.  Assign the instance a name and make the quantity two.  This will spin up two VMs in the same network.

Two new VMs

Now that two VMs are running clicking on the first VM should enable you to VNC into the VM itself.  From that VM start a ping to the second VM.

$ ping

If the pings are working Floodlight has correctly pushed the appropriate flows into the Open vSWitch using OpenFlow.  We can see the exact details by making an API call to Floodlight.  You can run the API call from either the Floodlight VM or your Ubuntu VM.  Floodlight should return back two flows ( I’ll summarize the output here because it’s long and ugly)

$ curl

{“actions”:[{“port”:3,”maxLength”:0,”length”:8,”type”:”OUTPUT”,”lengthU”:8}], “priority”:0,”cookie”:9007199254740992,”idleTimeout”:5,”hardTimeout”:0,”match”:{“dataLayerDestination”:”fa:16:3e:7c:64:18″,”dataLayerSource”:”fa:16:3e:13:a5:ea”,}

{“actions”:[{“port”:2,”maxLength”:0,”length”:8,”type”:”OUTPUT”,”lengthU”:8}], “priority”:0,”cookie”:9007199254740992,”idleTimeout”:5,”hardTimeout”:0,”match”:{“dataLayerDestination”:”fa:16:3e:13:a5:ea”,”dataLayerSource”:”fa:16:3e:7c:64:18″,}

Voila!  An SDN enabled private cloud based on OpenStack and OpenFlow.

Let’s Find Some Customers

Now that the basics are out of the way it’s time to do some more complex configuration and get some additional tenants going.  Out of the box DevStack includes a single network that every project is attached to.  Unfortunately the Horizon portal doesn’t expose a mechanism to define new networks or attach projects to existing networks.  So to stand up a new tenant in your private cloud you will need to do some CLI work.

When working with OpenStack via CLI you need to authenticate before you can issue commands.  The easiest way to do this is to export a few variables to tell OpenStack who are you.  On your OpenStack VM run the following:

$ export OS_TENANT_NAME=admin
$ export OS_USERNAME=admin
$ export OS_PASSWORD=nova
$ export OS_AUTH_URL=”http://localhost:5000/v2.0/&#8221;

Create a new tenant (project):

$ keystone tenant-create –name MyFirstCustomer

|    Property|                                  Value
|    description
|    enabled                                         True
|    id                      0335fc201366460486076b693875be4b
|    name                                MyFirstCustomer

Take note of the ‘id’ that was assigned to the tenant you just created.  You will use this from this point on to reference the tenant.  Now we need to create a user (bob) and assign it to the tenant you just created.

$ keystone user-create name bob pass nova

|   Property|                                   Value
|    email      
|    enabled                                         True
|    id                       3d2f30587fc24f468b24e6e0186eb8e5
|    name                                                bob
|    password         nova  (you will see a hash here)
|    tenantId   

Now you will need to assign ‘bob’ to the tenant you created will a specific role, in this case  the ‘member’ role.  To do this you will need the ID of the ‘member’ role.

$ keystone role-list | grep ‘Member’
| e27ca8edb75145c99874c44c591b5f58 | Member |

$ keystone user-role-add –user-id 3d2f30587fc24f468b24e6e0186eb8e5 –role-id e27ca8edb75145c99874c44c591b5f58 –tenant-id 0335fc201366460486076b693875be4b

You now have a new tenant and a user that can login to that tenant.  The last step is to create the networks that each of the tenants machines will connect to.  Let’s create two networks for each VM.  You will again need the tenant ID for this.  You will see a bunch of output from Quantum upon the creation of each network.  Again, I’ll summerize the output:

$ nova-manage network create –label=web –fixed_range_v4= –project=0335fc201366460486076b693875be4b

$ nova-manage network create –label=internal –fixed_range_v4= –project=0335fc201366460486076b693875be4b

Done!  Now login as ‘bob’ and check to see that things are working correctly.  When you create a new VM it should have three NICs, one labelled ‘web’ and the other labeled ‘internal’.  The third NIC will be labelled ‘public’  This NIC is the default network that is assigned to all projects.  The reason this network is attached to the VM is that it is actually assigned to no project, thus it is attached to all projects.  If you want to prevent this network from showing up on your new tenants simply assign the network to your ‘demo’ tenant.

A tenants VM with multiple networks.

There you have it.  Cloud computing, OpenFlow based SDN, multi-tenancy, and OpenStack in 60 minutes.  Now, think of a few creative ways to impress your executive team with what you have created.  To help out, I’ve created a script that you can run each time you reset DevStack.  It will create three tenants (each with their own networks) and two users.  Enjoy!

——–Copy below this line———-

# Set up authentication

export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=nova
export OS_AUTH_URL=”http://localhost:5000/v2.0/&#8221;


set -e -x

# Create tenants

COKE_TENANT_WEB=$(keystone tenant-create –name coke-web | grep ‘ id ‘ | awk ‘{print $4}’)
COKE_TENANT_DB=$(keystone tenant-create –name coke-db | grep ‘ id ‘ | awk ‘{print $4}’)
PEPSI_TENANT=$(keystone tenant-create –name pepsi | grep ‘ id ‘ | awk ‘{print $4}’)

# Create users

COKE_USER=$(keystone user-create –name coke –pass nova –email | grep ‘ id ‘ | awk ‘{print $4}’)
PEPSI_USER=$(keystone user-create –name pepsi –pass nova –email | grep ‘ id ‘ | awk ‘{print $4}’)

# Find UUID for Member role

MEMBER_UID=$(keystone role-list | grep ‘Member’ | awk ‘{print $2}’)

# Add users to tenants

keystone user-role-add –user-id $COKE_USER –role-id $MEMBER_UID –tenant-id $COKE_TENANT_WEB
keystone user-role-add –user-id $COKE_USER –role-id $MEMBER_UID –tenant-id $COKE_TENANT_DB
keystone user-role-add –user-id $PEPSI_USER –role-id $MEMBER_UID –tenant-id $PEPSI_TENANT

# Create networks for tenants

nova-manage network create –label=coke-web –fixed_range_v4= –project=$COKE_TENANT_WEB
nova-manage network create –label=coke-db –fixed_range_v4= –project=$COKE_TENANT_DB
nova-manage network create –label=pepsi –fixed_range_v4= –project=$PEPSI_TENANT

# Reassign the default network to Demo project

DEMO_TENANT=$(keystone tenant-list | grep ‘demo’ | awk ‘{print $2}’)
nova-manage network modify –fixed_range= –project=$DEMO_TENANT

——–Why are you still copying?  Stop above this line!———-

Categories: Cloud Computing, Data Center, Networking, SDN & OpenFlow

25 replies

  1. Dan,

    This is a really cool setup. I really like it and also I agree with you, one of the biggest barrier to entry of most new technologies into the enterprise data centers is being able to convince the exec team.

    This setup makes it so easy!

    ~ Aydin

  2. Dan,
    This is awesome! Curious to know if you can recommend openstack installer that doesn’t reset my config. Its a nightmare to restart everything after reset…

    • DevStack is nice for developers, giving them a quick reset button but a pain if you want to deploy a small persistent cloud service.

      There are a number of free distributions of OpenStack you may want to take a look at. StackOps comes to mind. It is a little more involved to install it and get it talking to Floodlight but it’s worth the effort.

  3. is there any other alternative to DevStack? I’d prefer persistent config. Thanks

  4. Hi Dan,
    It is very nice description. iam doing my masters in cloud computing. In coming project i choose SDN open flow. can u help me how it will work in laptop? did we have to purchase any hardware? I have to show live demo to my team.I need to install. can u tell the starting process, I already work in bulding private cloud.

    • There is no hardware involved in getting this up and running. I put this together on my laptop using VMware Fusion to run an Ubuntu VM for OpenStack and Floodlight. I’m planning on updating this shortly so stay tuned.

  5. Hi Dan,

    Just a question regarding OpenStack with floodlight.In the setup you mentioned floodlightVM.Is that a separate virtual machine(controller) running floodlight and the OpenStack(devstack) is installed on another VM?

    • Yes, I ran both as independent VMs although technically you can install Floodlight on the same VM as OpenStack. If you are trying to reduce complexity or don’t have the horse power on your machine to run two VMs then I would recommend installing Floodlight in the same machine as OpenStack.

      • Hello. Just a comment.
        I think you mean $ curl and not
        $ curl
        By the way, very good post 🙂

  6. Hi,
    When i run i gt an exception in the in openstack_dashboard: Versioning for this project requires either an sdist tarball or access to an upstream git repository

  7. Hi Dan,

    Is there a video for this setup on youtube or something my any chance ? that would be of great help for people like me.


  8. the link is no longer valid. Any similar script to get floodlight working with grizzly or upcoming havana release. I have been trying to get floodlight working with devstack for a few weeks now; but just can’t get my head around it. My floodlight can see the br-int as an ovs node (and the tap interfaces as nodes hanging off it). However, trying to create the cirros vm’s (included by default) gives errors. No idea what it may be.

  9. Hi Dan,

    I have successfully created SDN based cloud using latest OpenStack Havana release and wrote application on Floodlight. As alternative I tried even with Ryu too.

    Shankar Ganesh.P.J

    • Hi Dan, just found this thread as I’m starting with OpenFlow and SDN. The download link to floodlight no longer seems to work nor does the “apt-get install floodlight”command for ubuntu server. Do you know why this is? Have bigswitch changed their support of the floodlight project ?
      great thread though and very clearly written for a newcomer to follow


  1. Technology Short Take #27 - - The weblog of an IT pro specializing in virtualization, storage, and servers
  2. Technology Short Take #27 | Strategic HR

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: