Open vSwitch 1.9.0 on Red Hat Enterprise Linux (RHEL) 6.4

I’ve been using Open vSwitch as a replacement for the stock bridge module in Linux for a few months now.

Open vSwitch is basically a open source virtual switch for Linux. It gives you much greater flexibility than the stock bridge module, effectively turning it into a managed, virtual layer 2 switch.

Open vSwitch has a very long list of features, but I chose to use it instead of the stock bridging module because Open vSwitch offers much greater flexibility with VLANing on Virtual Machines than what is possible with the stock Linux bridge module.

As my KVM servers are running an older version of Open vSwitch (1.4.6), I decided to upgrade to the latest version, which is 1.9.0 at time of writing this post.

RedHat actually provide RPMs for Open vSwitch as part of a tech preview in the Red Hat OpenStack Folsom Preview repository. They also include the Open vSwitch kernel module in their kernel, but they are using version 1.7.1, I wanted 1.9.0, so I decided to write this blog post.

EDIT: 10/04/2013 – Looking closer, it looks like RedHat also have an RPM for 1.9.0, but they do not include the brcompat module. If you need this module, then you’ll have to build your own RPMs.

RedHat have actually back-ported a number of functions from newer kernels into the kernel provided with RHEL. This causes a problem when compiling the Open vSwitch kernel module as the OVS guys have also back-ported those functions and were using kernel version checks to apply those backports.

The OVS guys have pushed a patch into the OVS git repo which fixes this problem, so I won’t be using the tarball provided on the OVS site, but rather building from the OVS 1.9 branch of the git repository.

When using the git version of Open vSwitch, we need to run the bootstrap script to create the configure script etc, but this requires a newer version of autoconf. You can either compile autoconf yourself, or I’m sure someone has create a RHEL6 RPM for a newer version of autoconf somewhere, but I just done this part on a Fedora machine instead as it was easier:
git clone git://openvswitch.org/openvswitch
git checkout -b branch-1.9 origin/branch-1.9
./boot.sh
./configure
make dist

Now you’ll have a shiny new tarball: openvswitch-1.9.1.tar.gz

I moved this over to my dedicated RPM building virtual machine and extracted it:
tar -xf openvswitch-1.9.1.tar.gz
cd openvswitch-1.9.1

I got a compilation error when trying to build the Open vSwitch tools inside mock as openssl-devel isn’t listed as a requirement in the spec file so mock doesn’t pull it in. It’s an easy fix, I edited the spec file and added openssl/openssl-devel to it:
--- openvswitch.spec.orig 2013-04-01 18:43:50.337000000 +0100
+++ openvswitch.spec 2013-04-01 18:44:10.612000000 +0100
@@ -19,7 +19,8 @@ License: ASL 2.0
Release: 1
Source: openvswitch-%{version}.tar.gz
Buildroot: /tmp/openvswitch-rpm
-Requires: openvswitch-kmod, logrotate, python
+Requires: openvswitch-kmod, logrotate, python, openssl
+BuildRequires: openssl-devel

%description
Open vSwitch provides standard network bridging functions and

Next, I created the SRPMs using mock:

mock -r epel-6-x86_64 --sources ../ --spec rhel/openvswitch.spec --buildsrpm
mv /var/lib/mock/epel-6-x86_64/result/*.rpm ./


mock -r epel-6-x86_64 --sources ../ --spec rhel/openvswitch-kmod-rhel6.spec --buildsrpm
mv /var/lib/mock/epel-6-x86_64/result/*.rpm ./

Then, actually build the RPMs:

mkdir ~/openvswitch-rpms/

mock -r epel-6-x86_64 --rebuild openvswitch-1.9.1-1.src.rpm
mv /var/lib/mock/epel-6-x86_64/result/*.rpm ~/openvswitch-rpms/

mock -r epel-6-x86_64 --rebuild openvswitch-kmod-1.9.1-1.el6.src.rpm
mv /var/lib/mock/epel-6-x86_64/result/*.rpm ~/openvswitch-rpms/

All done! Next either sign and dump the freshly built RPMs from ~/openvswitch-rpms/ into into your yum repository, or scp them over to each host you will be installing them on, and use yum to install:
yum localinstall openvswitch-1.9.1-1.x86_64.rpm kmod-openvswitch-1.9.1-1.el6.x86_64.rpm

I won’t go into configuration of Open vSwitch in this post, but it’s not too difficult, and there are many posts elsewhere that go into this.

Connecting to Usenet via Two Internet Connections

As I mentioned in a earlier post, I have two connections from Virgin Media at home and I wanted to use them both to grab content from usenet.

My Usenet provider is Supernews, I’ve used them for a couple of months, and from what I understand they are actually just a product of Giganews.

Supernews only actually allow you to connect to their servers from one IP per account, so even if I had set up load balancing to split connections over both my connections, it would not have worked very well for usenet as I will be going out via two IP addresses! So for this reason I decided to take another route.

I have a dedicated server with OVH which has a 100mbit line, my two lines with Virgin Media are 60mbit and 30mbit, so I figured if I route my traffic out via my dedicated server, I should be able to saturate my line when usenetting. šŸ™‚

So the way I done this was to create two tunnels on my Cisco 2821 Integrated Services Router going to my dedicated server, one tunnel per WAN connection and basically “port forwarding” port 119 and 443 coming over the tunnels to go to Supernews. It’s working great so far and saturating both lines fully!

So the way I done this was as follows. First I setup the tunnels on my trusty Cisco 2821 ISR:

interface Tunnel1
description Tunnel to Dedi via WAN1
ip address 10.42.42.1 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1420
tunnel source GigabitEthernet0/0.10
tunnel mode ipip
tunnel destination 123.123.123.123

interface Tunnel2
description Tunnel to Dedi via WAN2
ip address 10.42.42.5 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1420
tunnel source GigabitEthernet0/1.11
tunnel mode ipip
tunnel destination 123.123.123.123

That isn’t the complete configuration, I also decided to NAT my home network to the IPs of the two tunnels. This was just in order to do it quickly. If I had not used NAT on the two tunnels, I would have to put a route on my dedicated server for my home network’s private IP range. Although this is easy, I was mainly doing this out of curiosity to see if it would work, and to do it without NAT on the tunnels I would have had to figure out how to do policy based routing in order to overcome asymmetric routing on Linux. That can be a project for another day. šŸ™‚

My dedicated is running RHEL6, so to set up the tunnel on the dedicated server I created the relevant ifcfg-tunl* files:

[root@moka ~]# cat /etc/sysconfig/network-scripts/ifcfg-tunl1
DEVICE="tunl1"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="IPIP"
PEER_OUTER_IPADDR="IP_OF_WAN_1"
PEER_INNER_IPADDR="10.42.42.1"
MY_OUTER_IPADDR="123.123.123.123"
MY_INNER_IPADDR="10.42.42.2"

[root@moka ~]# cat /etc/sysconfig/network-scripts/ifcfg-tunl2
DEVICE="tunl2"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="IPIP"
PEER_OUTER_IPADDR="IP_OF_WAN_2"
PEER_INNER_IPADDR="10.42.42.5"
MY_OUTER_IPADDR="123.123.123.123"
MY_INNER_IPADDR="10.42.42.6"

I don’t really want to go into detail on how configure netfilter rules using IPtables, so I will only paste the relevant lines of my firewall script:

# This rule masquerades all packets going out of eth0 to the IP of eth0
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Forward packets coming in from tunl1 with the destination IP of 10.42.42.2 and a source port of either 119 or 443 (Supernews use 443 for NNTP SSL port) to Supernews' server IP
iptables -t nat -A PREROUTING -p tcp -i tunl1 -d 10.42.42.2 --dport 119 -j DNAT --to 138.199.67.30
iptables -t nat -A PREROUTING -p tcp -i tunl1 -d 10.42.42.2 --dport 443 -j DNAT --to 138.199.67.30

# Forward packets coming in from tunl2 with the destination IP of 10.42.42.6 and a source port of either 119 or 443 (Supernews use 443 for NNTP SSL port) to Supernews' server IP
iptables -t nat -A PREROUTING -p tcp -i tunl2 -d 10.42.42.6 --dport 119 -j DNAT --to 138.199.67.30
iptables -t nat -A PREROUTING -p tcp -i tunl2 -d 10.42.42.6 --dport 443 -j DNAT --to 138.199.67.30

That’s all there is to it really! Of course I have a more complete rule set, but I don’t really want to go into that in this post!

Next, I just added two servers in my usenet client, one pointing at 10.42.42.2 and the other at 10.42.42.6. And magic! Now both lines will be used when my usenet client is doing its thing!

Note: If you got to the end of this post, I apologize if I make no sense, I was pretty tired while writing this post, and really just wanted to go to sleep. If you have any questions or suggestions on how to do this better, I’d be very interested in hearing them.  :~)

Dead Cisco Catalyst 3560

I’ve been trying toĀ acquire a Cisco Catalyst 3560 as it provides features which are not supported by my Catalyst 3550s, such as Private VLANs.Ā I believe the QoS features differ on 3560 as well.

So, as I was browsing eBay (one of myĀ favourite pastimes! :P), I found an auction for a WS-C3560-8PC-SĀ which had been labelledĀ “untested”. From past experiences, I have found that listings that state that they haven’t been tested are usually faulty devices, but I thought I would take the risk anyway. I was hoping it would be some small issue which I could either work around or repair, such as a bad port, or screwed up IOS image which I could just reload myself (hey! I’ve seen devices sell on eBay for pretty cheap due to non-techy people assuming it was broken because the IOS image was missing!). But I guess my luck was bad, and two days after the end of the auction, IĀ receivedĀ a large green paperweight. šŸ™

After plugging the power in, the LEDs on the front of the Catalyst 3560 go on, but they just stay on in a solid state, where as they should be blinking during the boot process. I plugged the console cable in, only to find that there is no output whatsoever, not even from ROMMON, which is the first step before even loading IOS.

I have pretty little knowledge of electronics, but I did test basic things that I knew how, such asĀ checking if the PSU was giving out the correct voltages, which it was, but that’s pretty much all I know how to check!

From my limited knowledge of electronics, I assume that something must be wrong with the Boot ROM chip since not even ROMMON is able to start. None of the parts on a Catalyst 3560 are fieldĀ replicable, so I don’t think I can test any parts by switching them around either.

I am quite disappointed that this Catalyst 3560 is dead, but I tried my luck, and it turned out bad, no biggie. šŸ™‚

Hopefully I will be able to find a Catalyst 3650 soon!

If anyone has any ideas I can tryĀ in order to fix this device, I would be quite eager to make an attempt! šŸ™‚

Two more Cisco 7204 VXRs Added to My Home Lab!

Cisco 7204 VXRsĀ Last week, I was browsing eBay (as you do!), and noticed two Cisco 7204 VXR routers auctions which were about to end pretty soon, price was Ā£0.99, and there were no bids!Ā So, I figured I would go ahead and bid. To my surprise, I won both!

I managed to win one of them for Ā£20, and the other for Ā£0.99! Ā£20.99 for two 7204 VXRs isn’t bad at all, just a quick search on eBay shows that the NPE-300s,Ā which came with both routers, is generally selling for Ā£30, so I’m quite pleased.

The I/O controllers (C7200-I/O) are a bit old, and use DB-25 connector for the console port and not the normal RJ-45 that most Cisco devices use. The I/O controller don’t have anyĀ EthernetĀ ports either, but I did get some FastEthernet modules with both routers. I will probably upgrade the I/O controllers toĀ C7200-I/O-2FE/E some time this year, but for now, it’ll do. šŸ™‚

I now have three 7204 VXRs in my rack, the first one I bought last year some time.

In the picture:

  • Top 7204 VXR has: NPE-225, 128MB RAM,Ā C7200-I/O, Dual FastEthernet Module and an Enhanced ATM module (ATM PA-A3).
  • Middle 7204 VXR has: NPE-300 with 256MB RAM (if I remember correctly),Ā C7200-I/O, Single EthernetModule, and anĀ Enhanced ATM module (ATM PA-A3).
  • Bottom 7204 VXR has: NPE-300 with 256MB RAM (if I remember correctly),Ā C7200-I/O-2FE/E, and anĀ Enhanced ATM module (ATM PA-A3).

I’m not really sure if the Enhanced ATM modules will be of any use to me, as I don’t think it is possible to use themĀ back-to-back (please correct me if I am wrong!). I do want to get a fewĀ Cisco PA-4T+ 4 Port Serial modules but that’s for later on.

Cisco ASA 5505 RAM Upgrade

Edit: 3rd June 2014 – If you are reading this post, you should check out my follow up post: Cisco ASA 9.2 on Cisco ASA 5505 with Unsupported Memory Configuration Fail.

I have two Cisco ASA 5505s in my home lab which I acquired almost two years ago from eBay. I was pretty lucky, as I paid under Ā£70 for each because the seller wasn’t too sure what they were! Looking on eBay now, they are selling for around Ā£120! šŸ™‚

Pretty much straight away, I wanted to upgrade to the ASA 8.3 code, which required a RAM upgrade, so I upgraded it.

Starting from ASA 8.3, the minimum required RAM needed to run 8.3 code and newer on a 5505 is 512MB. This is also the maximum officially supported amount of RAM.

Buying official Cisco RAM is, as always, quite expensive, but since the ASA 5505 uses standard DDR RAM, it is actually possible to use third-party RAM in the ASA 5505.

When I originally performed this upgrade, I found that on various forums many people had actually upgraded past the official supported amount of RAM, and upgraded their ASA 5505s to 1GB RAM.

Intrigued  by this, and due to needing the extra RAM for the 8.3 code, I decided to upgrade both my ASAs to 1GB as well!

There aren’t any ground breaking advantages to upgrading to 1GB as far as I know. I’m guessing the ASA will be able to hold a lot more entries in the NAT table, but I don’t really push my ASAs to their limits anyway.

I ended up buying two CT12864Z40B sticks from Crucial, which have worked flawlessly for the past year.

Almost 14 months later, I needed to crack open the case of the ASAs again to get to the CompactFlash. I thought I’d make a quick post about the RAM upgrade process while I’m at it.

The upgrade is very easy, anyone could do it, but I was bored, and wanted to write a blog post! šŸ™‚

  1. Place the ASA upside down, and unscrew the three screws at the bottom.
    Cisco ASA 5505 Screws
  2. Remove the cover
    Cisco ASA 5505 Internals
  3. Take out the old RAM, and put in the new RAM.
    Cisco ASA 5505 RAM
  4. You can optionally also upgrade the CompactFlash at this time. I’m using the stock 128MB that came with the ASAs at the moment, but I will probably upgrade sometime soon. šŸ™‚
    Cisco ASA 5505 CompactFlash
  5. Close everything up, and plug-in the power!
    Cisco ASA 5505 Failover Pair

All done! I haven’t got a screenshot of it booting at the moment, but I will probably update this post tomorrow with one.

I plan to upgrade the CompactFlash to 4GB as well so I have more working space when I am using the “packet sniffer” built into the ASA. This is a very easy process as well, but you have to be careful to copy over your licence files as well. I will be making a post about this as well when I have done the upgrade.

My Goals for 2013: CCNP and RHCE?

I’ve been thinking about renewing my RHCE for quite sometime now, and completing my CCNP but I haven’t really got around to it, mainly due to the price of the exam being a little pricey (if I remember correctly!), but also due to not having enough time.

So for this year, I wanted to set a deadline for myself to complete them. With a deadline, it is easier to visualize and plan what to study and when, and allows you see your progress better.

So, my goal is to complete CCNP ideally by the end of May, or by the end of June at the latest. I think it should be possible! There are three parts to the CCNP: ROUTE, SWITCH, and TSHOOT. If I complete one per-month, it should beĀ achievable!

Like most people, I am using the Cisco CCNP Routing and Switching Official Certification LibraryĀ books as my study material, and highly recommend them.

On that note, I have added a “Home Lab” page where you can see pretty pictures of my rack, and my “CCNP Lab”. It’s nothing close to something as awesome as Scott Morris’ Lab, but it’s coming along! šŸ˜‰

I have read that RedHat will be releasing RHEL7 in the second half of this year, so it is a perfect opportunity to renew my RHCE! My goal end date will depend on when RHEL7Ā will beĀ released, and when the test centers are actually testing under RHEL7.

Hopefully this will be earlier into the second half of the year, so I haveĀ plentyĀ of time to take the exam before the end of December!

Both CCNP and RHCE are great certifications, which are very highly regarded by employers and professionals.

A lot of people seem to think they don’t go well together, as RHCE is better for System Administrators and CCNP is more for Network Administrators, but I totally disagree since I feel that the lines between sysadmins and netadmins is very quickly disappearing thanks to virtualization and “cloud” technologies.

Moving Back to London and Virgin Media

Two weeks ago, I moved back to London.

At my place in Cambridge, I had a 100mbit connection from Virgin Media, which I wanted to cancel as my parents already have a connection from Virgin Media so there wasn’t any need for me to move mine along with me!

So I called up VM, and they informed me that as I was still in contract, I would have to pay someĀ ridiculousĀ amount to cancel the contract (I think it may have been Ā£280).

Alternatively, the other option was one which I was not aware of!

Usually, Virgin Media do not allow people to have two connections from them under a single address, BUT, in cases such as mine, they allow it!

So, instead of paying Ā£280 or whatever it was, I decided it’d work out much cheaper if I just move my connection with me. It’d be nice to have anyway! šŸ™‚

Today, the Virgin Media guys arrived at my house. Their first reaction was shock at seeing my server rack, but they were pretty nice guys. I did have to explain what I use all thisĀ equipmentĀ for, and had to explain how terrible the SuperHub actually is! šŸ™‚

They didn’t really have to do much, they just really had to add a splitter and give me two new coax cables going from the splitter to the two modems. They did mention their frustration at VirginMedia about having to do installations for the more technical people when those technical people could really just do it themselves, and I totally agree! šŸ˜€

For some reason, they had to switch out my old SuperHub, and gave me a new one which has a matt-finish instead of the glossy look my old one had. I’m not sure if there is any other difference other than that, not that I care, I enabled modem mode ASAP so I don’t have to deal with this terrible device too much. šŸ™‚

I was a little worried that I might not get the full bandwidth on both connections, but it looks like I am!

Next steps are to figure out how to do load-balancing on my Cisco 2821 ISR.

Virgin Media Cable Wall Outlet Two way splitter VirginMedia SuperHub and other goodies

Apache Traffic Server Basic Configuration on RHEL6/CentOS 6

In this guide, I will explain how to get setup Apache Traffic Server with a very very basic configuration.

I will be using RHEL6/CentOS 6, but actually creating the configuration files for Traffic Server is exactly the same on all distributions.

As a pre-requisite for setting up Traffic Server, you must know a little about the HTTP protocol, and what a reverse proxy’s job actually is.

What is Apache Traffic Server?

I don’t really want to go into too much detail into this as there are many sites which explain this better than I ever could, but in short, Traffic Server is a caching proxy created by Yahoo! and donated to the Apache Foundation.

Installation

Apache Traffic Server is available from the EPEL repository, and this is the version I will be using.

Firstly, you must add the EPEL repositories if you haven’t already:
rpm -ivh http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-7.noarch.rpm
Next, we can just use yum to install Traffic Server:
yum install trafficserver
While we are at it, we might as well set Traffic Server to start at boot:
chkconfig trafficserver on

Configuration

In this tutorial, I will only configure Apache Traffic Server to forward all requests to a single webserver.

For this, we really really only need to edit two files:

  • /etc/trafficserver/records.config
    This is the main configuration file which stores all the “global” configuration options.
  • /etc/trafficserver/remap.config
    This contains mapping rules for which real web server ATS should forward requests to.

Firstly, edit records.conf.

I didn’t really have to change much initially for a basic configuration.

The lines I changed were these:
CONFIG proxy.config.proxy_name STRING xantara.web.g3nius.net
CONFIG proxy.config.url_remap.pristine_host_hdr INT 1

Next we can edit remap.config.

Add the following line to the bottom:
regex_map http://(.*)/ http://webservers.hostname:80/
This should match everything and forward it to your web server.

Start traffic server:
service trafficserver start
And that’s it! It should now just work! šŸ™‚

SSL Certificates for XMPP

Over the last few months, I have been slowly switching all my hostnames and service names from using my personal domain name “hamzahkhan.com” to another domain I have.

This is mainly because I am sharing some of the services I run with other people, and also because… well… I don’t like having my name in hostnames to be honest! šŸ™‚

Today I finally got around to updating my Jabber/XMPP server.

In the process, I had to update the SSL certificate.

Quite some time ago, a friend of mine actually told me that I’ve created the certificate for my XMPP server incorrectly when using a single server to serve multiple domains.

For this, you are actually supposed to have a few extra attributes in the certificate.

To add these records, create a file called “xmpp.cnf” with the following contents:
HOME = .
RANDFILE = $ENV::HOME/.rnd

oid_section = new_oids

[ new_oids ]
xmppAddr = 1.3.6.1.5.5.7.8.5
SRVName = 1.3.6.1.5.5.7.8.7

[ req ]
default_bits = 4096
default_keyfile = privkey.pem
distinguished_name = distinguished_name
req_extensions = v3_extensions
x509_extensions = v3_extensions
prompt = no

[ distinguished_name ]

# This is just your standard stuff!
countryName = GB
stateOrProvinceName = England
localityName = Cambridge
organizationName = G3nius.net
organizationalUnitName = XMPP Services
emailAddress = xmpp@g3nius.net

# Hostname of the XMPP server.
commonName = xmpp.g3nius.net

[ v3_extensions ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature,keyEncipherment
subjectAltName = @subject_alternative_name

[ subject_alternative_name ]

# Do this for each of your domains
DNS.1 = domain1.com
otherName.0 = xmppAddr;FORMAT:UTF8,UTF8:domain1.com
otherName.1 = SRVName;IA5STRING:_xmpp-client.domain1.com
otherName.2 = SRVName;IA5STRING:_xmpp-server.domain1.com

DNS.2 = domain2.com
otherName.3 = xmppAddr;FORMAT:UTF8,UTF8:domain2.com
otherName.4 = SRVName;IA5STRING:_xmpp-client.domain2.com
otherName.5 = SRVName;IA5STRING:_xmpp-server.domain2.com

DNS.3 = domain3.com
otherName.6 = xmppAddr;FORMAT:UTF8,UTF8:domain3.com
otherName.7 = SRVName;IA5STRING:_xmpp-client.domain3.com
otherName.8 = SRVName;IA5STRING:_xmpp-server.domain3.com

Then you just continue the “certificate request” creation as normal specifying the configuration file on the command line:

# Create the private key
openssl genrsa -des3 -out xmpp.g3nius.net.key 4096

# Create the certificate request:
openssl req -config xmpp.cnf -new -key xmpp.g3nius.net.key -out xmpp.g3nius.net.csr

That’s all!

Now you can either use the CSR to request a certificate from CACert.org or anywhere else, or you could self-sign it and point your XMPP server at your shiny new certificate!

RedHat Enterprise Linux 6 Beta

RedHat Enterprise Linux 6 Beta 1 has finally been released as a public beta. It is available as an ISO from the public RedHat FTP site.

A couple of days ago, I decided to play with the beta, and I discovered, as expected, there are a lot of significant differences between RHEL5 and RHEL6.

The the main difference which I found to be very frustrating is that there is no longer any support for Xen dom0.

I had heard about RedHat’s decision to stop supporting Xen, but I did not think that this would mean they would stop shipping it with the distribution.

The loss of dom0 support means that you can no longer use RHEL as a Xen virtualization host, rather only as a guest under other Xen supporting distributions.

XenĀ has beenĀ dropped inĀ favourĀ of Kernel-based Virtual Machine (KVM), which is a Ā virtualization infrastructure included with the Linux kernel. Linux KVM is a hardware-assisted virtualizationĀ infrastructureĀ which requires the CPU to have a special CPU feature called Intel-VT on Intel CPUs and AMD-V on AMD CPUs.

KVM has limited paravirtulization support, but I found in my very simple tests that fully paravirtualizaed guests inside Xen had much better performance.

This latest release of RHEL also means that my RHCE will soon expire. I am hoping to get re-certified as soon as I can. At the same time, I am also considering taking the “Red Hat Certified Virtualization Administrator” course and exam, but I still have some time to think over that. šŸ™‚