A Very Surprising Test Drive…

Posted: October 20, 2014 in Cars

In life there very rarely come times that you know are truly special.  Almost inevitably these are the times when you were expecting nothing, yet ended up overwhelmed by an experience.  I am genuinely shocked, but absolutely thrilled, to report that today I had one of these moments.  Now obviously the subject mentions a test drive and this is filed under “car”, so it’s not a surprise this is an automotive experience we’re talking about, but I think the subject in question will be a surprise to many.

In my last entry I detailed my experiences thus far with the RS5.  Reading between the lines of that entry it might have become apparent that, as amazing a car as the RS5 is, it isn’t a car I can necessarily be passionate about.  At this level of investment, for a real car enthusiast, I think true passion is a mandatory component of a lasting relationship.  Like a marriage, when the passion fades, there is trouble.  If the passion was never there to begin with, then look out!

My criteria remain challenging though.  Seating for four(ish).  Near super car performance or beyond.  Smart looks and high quality fit and finish.  The exclusivity that comes with something truly special.  And, perhaps most importantly, all year, all-weather, usability (meaning AWD and all-season rubber).  Now mind these are my criteria.  I get that many others find dramatically different scenarios perfectly workable.  But not me.  Investing upwards of $90,000 hard earned dollars requires these boxes to be checked for me.  It goes without saying that these are tough criteria, so needless to say any time a new entry in this rarefied category comes about, it catches my attention.

A year or so ago I had tested the Mercedes Benz CLA45AMG.  I never chronicled it because I didn’t find that the drive really pushed any of my buttons.  Something was missing there, but I couldn’t quite figure out what.  Irregardless I crossed it off the list, but kept an eye on it since to me all of the cars at this level are worthy of analysis.  Recently I began noticing some very interesting head-to-head showdowns.  It seemed many were putting the AMG up against a very bizarre vehicle I had no idea existed.  Rolling out of Bavaria, it seems BMW had slapped xdrive onto an M235i.   I know.  Cue the groans and hisses.  As it is I know many of the BMW faithful already deride the brand for “selling out” the M badge, watering it down, attempting to copy Audis success with pseudo performance editions (S line) and so on.  In addition, the 1/2 series have also drawn fire.  The are “chick cars” to some.  “Budget BMWs” to others.  Still, there has always been clear potential in this baby BMW line for many.  First of all, the 3 series keeps growing.  Amazingly enough, the classic E46 3 series really shares proportions with a 1/2, not a modern 3.  The latest 3 series cars (the F series) are actually now on par size wise with the classic E39 5 series. This is the trend, but for those of us who love “small and light”, and always enjoyed the 3 series because it was tossable and felt like an extension of the driver, the trend has been a bit of a letdown.  So right from the start I had some instant interest in the original 1 series if for no other reason than that it was small.  And then BMW did an amazing thing and release the 1M.  This silenced many critics and suddenly BMW Jr was all grown up and legitimate.  It didn’t last, though, and the 1M vanished.  Now we have a big reconfiguration of the line.  Coupes become 2s and 4s, the sedans remain 1s and 3s, but the 1s are gone.  The 1M hasn’t returned as an M2, but we do have this “M” 235.  But is this just marketing?  Is the “M” an appearance kit designed to satisfy brand snobs?  Just what is BMW doing?  And how can anything with an “M” have xdrive and not be a truck?  And why are there “M” series trucks?!  Well I have no plans on tackling that last question, but as for those other ones, I decided it was time to find out first hand!

Marching on down to my local BMW dealer, who just so happened to have a nicely loaded M235 xdrive waiting for me, I prepared in my mind the sarcasm laden entry I’d be producing.  As I smugly parked the RS5 and spied the little white M across the lot, I chuckled to myself with evil glee.  This was going to be great.  The fanboys were going to be livid.  As the sales guy greeted me I put up my best “well I suppose I’ll do you the favor of bothering to look at one of these cars!” air that I find works best as an opening posture, and suggested that I might be willing to test that ridiculous little white car tucked away in aisle 9.  Keys and plate in hand, we headed over.

The initial walk around was my first surprise.  This thing looked better than the 1 series for some reason.  The proportions looked right and the flame sculpting wasn’t completely obnoxious.  I had to admit I liked it.  And it was refreshing to see a small BMW that didn’t remind me of a midsize BMW.  OK so far this wasn’t working out well.  I needed some snark material dammit!  Well in this respect, hopping inside was reassuring.  The interior was, as is the case with all current BMWs in my opinion, a bit hectic.  There is something going on between the legacy look of the center console, the modern look of the thin LCD, and the dramatic trans tunnel that doesn’t quite mesh.  In addition, the 2 series has even crappier materials than the E92 3 series, including some bizarre trim that can’t really be described.  Making it worse is that the other options are brushed aluminum (created by the devil… dents if you breathe on it and looks like crap once dented) or wood (something which has no business in any car bearing an “M”, even if that “M” is fake).  To be fair, the negative first impression was probably also compounded by the fact that I had also just peaked inside a very nicely appointed M4.  The M4 is night and day, with incredible looking seats and the best execution possible of the current BMW interior design language. In addition the M4 uses materials noticeably superior to the outgoing M3.  The saving grace, though, is that this little BMW is a lot cheaper than a 4 series and in the case of the M235 xdrive, the big money you are paying (sticker was $53,200) is clearly going directly to actual performance options)  So I have to say that while the cabin was dangerously close to an economy car, the sticker price makes it forgivable.


What the hell is that weird material? Apparently its “aluminum hexagon interior trim”. This is code for “painted plastic”. It looks sharp in pictures and really cheap close up. Not a win here!

OK, enough ogling.  Time to sit down and drive the thing!  Immediately I was reminded why BMW continues to win accolades (and no, it isn’t because they pay off magazines! stop that!)  The driving position, seat support, and steering wheel feel were all spot on perfect.  It’s ridiculous to say the driver seat fits like a glove, but it fits like a glove.  The controls are all intuitive too and things, despite looking a bit different, haven’t changed that much since “back in the day”.  OK, no snark to be found here.  Let’s fire this thing up.

As with any modern car, starting it means pressing the brake and pushing “start” hoping there is a key fob around somewhere so it will work.  There was and it did and the engine fired up and oh… my… god… there were no noises.  This is a forced induction I6 with an actual torque converter automatic.  After driving a GTR followed by an E92 M3 followed by an RS5, there was surprisingly little occasion.  No complex dual clutch transmission chattering its clutch plates.  No high strung V8 growling to life.  No sport exhaus… Actually.  Scratch that.  There is a nice sport exhaust that did unleash some very nice burble that pleasantly surprised me!  So we were missing some good noises, but also some bad noises, and we retained one of the best noises.  That’s not so bad really!  So far this was definitely “Mish”.  OK, let’s roll…

Throwing it in drive means pushing a button and pulling ack on the weird center stalk thing.  The dashboard has the classic BMW analog gauges, but with some partially digital lower area that displays lots of things I pointedly decided to ignore.  One of them had a picture of a battery and mentioned something about ECO.  Last I checked this thing isn’t a hybrid, so that was something to worry about later.  The stalk has a selector button, so I pushed it.  The expected modes show up in the central digital area on the dash.  Sport+ (I later learned this disabled DSC), Sport (save that one), Comfort (let’s go with this) and “ECO mode” (ah ha!).  I tend to leave these cars in comfort most of the time, so this seemed the best way to get an initial real world impression.  One thing I’ve found changing cars as frequently as I do (I rent cars every month for work as well) is that muscle memory is fascinating.  It’s always important to keep in mind when driving an unfamiliar car that throttle and brake pressures required can differ dramatically.  Applying RS5 level force to the M235 throttle didn’t seem to do much initially.  I mean we were moving, but not “oooooh yeah” moving.  Approaching the end of the dealer lot and heading onto the main road, I adjusted.  Oh MAN were things different!  Giving proper throttle the M235 is surprisingly quick even in comfort mode.  The E46 M3 comparisons, which I had considered the ramblings of a deranged mind, started to make some sense.  My snark laden plans were fading into memory.  This thing might just be interesting!

The friendly sales guy directed me onto the highway and we were en route to some twisties (a very accommodating fellow!) As with most performance cars today, every aspect of the car is dynamically adjustable and tied to the vehicles central control system.  In comfort, the suspension felt a bit soft to me. Almost distractingly so.  This is coming from the stiffly sprung traditional suspension of the RS5 though, so keep that in mind.  In any event I found I got used to it after a mile or so as the car settled down into highway speeds.  As expected of a brand new BMW, it was a very comfortable driving experience.  The cabin was quiet, but the exhaust sounded great.  No stereo fed synthetic sound either, just legitimate exhaust noise as heard from inside.

The sales guy and I had a nice discussion about the past 20 years of BMW design and evolution and, before I knew it, the twisties loomed!  With the time to get serious at hand, I switched the baby beast into Sport mode and waited for the traffic to inch me towards my turn off.  Five… Four… Three… Two…  BOOM!  Ho…LEEE… SMOKES!  At this point I was no longer thinking about snark and was instead evaluating what sauce would go best with crow.  The acceleration and feedback of the M235 in sport mode is exhilarating.  And what’s more is that it sounds great roaring through the curves also!  And for some reason the automatic transmission is blipping the throttle on every shift!  Why? Who knows! But I like it.  At this stage the E46 was firmly in my mind.  Gone was the vaguely soft feeling of the comfort mode EDC setting.  In sport the dampers were firm, with no body roll to be felt, but yet not harsh.  The weighting of the steering, already better than the EPS system on the 4 series by my feel, was now proper M car direct.  I have to say that the back roads experience of the M235 was on par with the NSX and 996 for me.  Blasphemy!  Blasphemy perhaps, but the M235 really does have that E46 magic.  The feeling of being connected to the road, and of the car being an extension of your inputs, is real here.  It is a small car, but it feels even smaller and that is phenomenal in a sports coupe.  And perhaps it is because everything has become so damn heavy, but it even feels a lot lighter than it actually is.  XDrive may be no Quattro, but it also didn’t get in the way.  Keep in mind this is real world driving.  If you’re finding the limits of adhesion and sliding the ass out on a RWD BMW, you should probably be arrested.  Conversely if you trigger understeer on an AWD car, you probably should as well.  So I can’t really comment on the AWD xdrive being less “dynamic” at the limits than the RWD non xdrive version.  I’m sure it is and I’m sure that outside of forum posts and magazine racing, 90% of drivers will never be aware of that.  In my spirited back roads drive the car just felt dead neutral and that is a great thing in my book.  A smile found its way onto my face sailing through those turns that didn’t leave until I was back in the RS5.  It took quite a high speed run in the Audi to erase that missing sense of fun factor that the little BMW provided through sheer agility.  Like the FR-S/BRZ this is a car that is much more than it’s spec sheet.

So is it a “proper M car”?  In my opinion the question is irrelevant.  Whatever this thing is, it’s great.  I’m quite frankly shocked BMW even made this car and having driven it, I can’t quite see where an M2 would fit.  This thing is an absolute bargain and I suspect it may be come something of a classic.  Not to the degree of the limited 1M of course, but it definitely joins the pantheon of “ultimate sleepers”!

Little beasty...

Little beasty…

Well it’s pretty clear I need some sort of intervention. I thought this series had wrapped. I assumed it was put up on the digital shelf to be cracked open for purposes of nostalgia in some foggy future. I honestly had winsome visions of… Oh to hell with it. I knew there was no way my car ADD could possibly rest. So what gives? Let me catch you up…

When we left off the Complaints HQ garage was full to the brim with metal in the form of one E92 M3 DCT, a shiny new Land Rover Ranger Rover Evoque, and the humble yet feisty Mini Countryman All4. I am happy to say that the Mini is still holding strong! Of course it’s worth noting that this is my wife’s car and she’d almost certainly choose it over me, but I digress. As it turns out, with no powerful force to protect them, the Evoque and M3 did not fare nearly as well. So what happened? What exactly was the deficiency?

First let me state that both vehicles are fantastic and should thoroughly satisfy any sane person who is looking for a superlative experience in each of their respective categories. Of course the key words here are “sane” and “respective categories”. Any true performance enthusiast who lives in a region blessed with snowy winters, and has a legitimate need for some level of practicality, can almost certainly identify with this struggle. By all rights 90% of the time all you can truly justify owning is the crossover vehicle that get’s the best MPG. After all you can park them anywhere, they can typically slog through any pre-apocalyptic driving conditions with all-seasons, and they’re less than completely horrible to drive. That other 10% though… Therein that 10% lies the rub. For the true performance enthusiast, even the mightiest crossovers (SRT8, X5M, Cayenne Turbo) can’t truly scratch the itch. Even if they fully talk the talk (go really fast), they can never really walk the walk (look really good and feel agile).

The Evoque is an absolutely wonderful crossover. It actually gives up some practicality in the form of rearward visibility and backseat headroom, in exchange for a stylish look. The interior is lovely and it looks probably as good as a small truck can possibly look. In addition, it’s not too bad to drive and can move as well as any normal car (0-60 in 6.7s or so). It has decent cargo capacity, comfortable room for 4 and wears that oh-so-important Range Rover badge.

The E92 M3, on the other hand, is an incredible performance tool. The DCT shifts orders of magnitude faster than I ever will, the V8 soundtrack is like a symphony, and the car is quicker than probably 95% cars on the road (60 will come in less than 5 seconds in a typical street start scenario). In terms of balance, feedback, precision and driver experience it finds it’s better only in the 911 and NSX of the cars I’ve owned. On top of all of this it looks sharp, is a bit of a sleeper since most write it off as a basic 3 series, and it is reasonably practical. Until November, that is, when the powder falls and the local news gal starts screaming “SNOWPOCALYPSE!” Now look. I acknowledge that I am incredibly lazy and have bad luck. “Swapping tires” has just never worked for me and never will. With my luck inevitably I end up in a freak early snowfall sliding along on summers through October powder, or I wind up glued to the ground in Blizzaks due to an unprecedented February heat wave. As a result I always run all seasons. Yes, this neuters these fine cars. I get it. In any event, a powerful real wheel drive car is never fun in snow. Workable? Maybe. Fun? No.

So I found myself with one car that was extremely fun and semi-practical, which I couldn’t rely on year round. And another car that was quite practical, and semi-fun, which I honestly didn’t want to drive year round. Added to that was the time and effort required to maintain two vehicles. It took less than a year for this to wear thin.

I was determined to find the holy grail! A single vehicle that could be both “practical enough” and driven year round, yet was both “special” and not only scratched, but clawed that performance itch! Now to be sure I’d actually already owned some fine specimens here. In particular, I found myself thinking back to Godzilla and, specifically, why the hell did I sell it?

Putting on the official hat of introspection, most of my objections to the GTR were very personal and somewhat psychological. It’s a peculiar quirk of mine that used performance cars just never quite sit well with me. Sure if they pass a PPI and all is well you get an amazing deal, dodging initial depreciation but still rolling in style. But if it’s not factory new I never quite form that emotional bond with it. This quirk was eating at me with the M3 as well, but with the GTR it was magnified. The GTR is such a special and exotic vehicle that not having bought it new left me feeling a bit of a pretender. Ridiculous to some I know, but I’ve already admitted I need counseling! In addition driving the GTR is like driving a parade float with a naked supermodel on top waving at the crowd.  Some love this.  Some need this.  I hate this. But with the decision made to evict both the Evoque and the M3, the I’d be lying if I said the GTR didn’t come right back on my radar. Not another pre-owned though. No sir. Brand spanking new! After all, the DBA (12+) car might as well be called a GTR 1.5 compared to the CBA car (09-11). The differences are tremendous. In addition, Nissan had introduced the Black Edition so there was a real opportunity here to get back into a GTR and not feel like I was going backwards. A new GTR BE would truly be a whole different beast than my 09 Premium. So is this a “back to GTR” entry? SPOILER ALERT, but no. Well why not? In a nutshell, Nissan has just gone nuts with the MSRP of the mighty Godzilla. Still a bargain perhaps, but well out of my price range with a 2014 MSRP in excess of a hundred grand. Too rich for my blood! An interesting side note is that now (meaning as of this publication) they can be, since there are not only ’14’s, but still brand new untitled ’13’s languishing on lots.  Nissan cannot move these cars and so has ponied up some very deep incentives. As a result there may actually be a “Part II” coming to this entry, but no promises!

Back on track though, with the GTR being priced well out of my range at the time, and a used example no longer an option, the field was getting really narrow. I found myself starting to look at a brand I had long forgotten. Hearkening back to the very dawn of my automotive obsession! Yes, that’s right, Audi. I mean is it any surprise? If the criteria are fast, sporty yet luxurious, all-weather travel who wouldn’t look to Ingolstadt? Well me for one. I have to admit I’ve really just never been an Audi fan. For one reason or another they just don’t occupy the same head space as BMW or Mercedes for me and they’ve also gotten quite expensive. That said, I figured it couldn’t hurt to head on over to the local(ish) big Audi dealer (rather than the actual local small one) and see what was up. It was there that I saw this in the flesh (er… metal):

2014-01-10 12.35.57

Yes, that’s a 2014 RS5

Full disclosure, I had test driven an S4 and an S5 back in 2012 and come away pretty underwhelmed.  I found that the S5 felt surprisingly slow on the street and the steering felt quite artificial.  The S4 was a bit bland.  I filled the salesman in on all of this in true “hostile consumer” form (it was his lucky day!), but he was adamant that this car was nothing like those and encouraged a test drive.  Well.  Who am I to turn down a test drive in an RS right?  Man.  After a quick 15 minute run I had to painfully admit that the guy had a point!  So how is the RS5?

The first thing that struck me was the build quality.  Followed quickly by the materials quality.  And hot on the heels of those the design language.  In all three of these areas Audi is killing BMW.  Full stop.  Now this is just my opinion, but it is by no means a minority one.  And please keep in mind that I don’t really like Audi, yet love BMW.  But the E92 M3 was simply not competitive with this car in these metrics.  The Audi immediately looked and felt more solid and modern, and every surface had a stronger sense of quality.  Yes they both use a lot of plastic, but plastics aren’t all created equal.  And for those curious about how the M4 might fare, one of my last service appointments for the E92 incidentally left me with an F80 loaner. I have to say I feel that car is perhaps actually worse.  The materials quality was a bit better than the E92, but in the F80 the design language is even more of a mess to my eye.  Having since test driven the M4, I maintain that the Audi is a nicer place to be; it’s just a cleaner cabin:

2014-01-10 12.37.04

And most importantly… Door close = solid THUD. THIS is key!

This is all well and good, but what about what matters?  What about performance?  From the first press of the accelerator I knew that this car had nothing to do with the S4 or S5.  It felt fast.  “WTF are those magazines talking about?” fast.  Anyone who has read any review done by anyone other than Motortrend (whose reviews seem to directly mirror my own experience) needs to drive this car for themselves before judging it.  The RS5 is a monster on the street.  Quattro is more brilliant than ever and just grips. The engine is a beauty.  Much like the E92 M3 it is a high revving small block V8 (4.2L in this case making 450HP).  Forget the masses who fixate on torque numbers and lack the ability to rev an engine.  Both the E92 M3 and the RS5 provide an exotic experience in the form of 8000+RPM on a screaming 8 cylinder  For those who are able to actually articulate their ankle, the numbers prove that both fly despite torque figures (295 for the BMW and 315 for the RS5).  Power to weight on these cars is about the same, but the Audi feels quicker.  And this feeling extends far beyond 60.  At the top end, say from 70-110 (in theory… not that I’ve ever tested this! ahem) the RS5 can feel GTR quick (I’m sure it’s not, but it feels it).  Perhaps more importantly, the aural feedback that comes along with all of this performance bests any car I’ve owned short of the Tubi equipped NSX.  That includes the 911 with PSE.  Full disclosure my car has the factory sport exhaust, but I can say with confidence it is the best exotic V8 soundtrack this side of a Maserati.

By the numbers the RS5 pulls to 60 in anywhere from 3.9 – 4.5s depending on who is driving and whether or not they’re using launch control (for reference, MT pulled a 3.9 and, without launch control, I measured 4.5 on my vbox just flooring it at a light), and can do the quarter in 12.4 at around 111MPH trap (as evidenced by many YouTube videos at this point).  By comparison both the M3 and M4 are essentially in the same range.  The difference is that in 80% of normal driving scenarios, the RS5 performance is just far more accessible.  Yes that means it’s all “easier”.  And yes that means a brilliant computer is playing all sorts of games with torque vectoring to ensure you don’t die, but that also means that far more people can use this performance far more of the time.  Do I think the M4 (or E92 M3) are more fun on the track?  In the hands of an experienced RWD driver absolutely.  And the M4 in particular will best the RS5 lap times by a couple of seconds in such a scenario as well.  But for the overwhelming majority of drivers, they’re never going near a track and, if they do, they’re not going near those kinds of numbers.  I’d be willing to bet money that 8 out of 10 drivers will be faster in an RS5 than in a C63 or M4 on a track.  On the street its a wash and it really is more about traction and transmission tuning.  Which brings me to the next point…

I cannot say enough about the Audi dual clutch.  The DCT was great, the Audi S-tronic is beyond great.  It’s the smoothest dual clutch I’ve ever driven (in a different universe than the GTR) and feels almost like an automatic, yet when you engage “dynamic” and whale on it, it is just as neck snappingly quick as the GTR or M3 DCT.  An aggressive blip of the throttle greets every downshift and upshifts are instantaneous.  The sport mode is smarter too.  Whereas on the M3 you have a giant range of settings, and the most extreme trans setting (S5) is honestly a bit too brutal to use for daily driving, the Audi provides only 3 settings (auto, soft and uber) and more intelligence.  The Audi dynamic mode is every bit as aggressive as DCT S5, but is also smarter about how long to hold revs before upshifting and when to downshift based on dynamic conditions.  With DCT I only used S5 when testing and most of the time left the car in S3.  With the Audi I leave the transmission in “sport” about 30% of the time.

To cut to the chase, I’ll summarize my view of the M3 and RS5:

Build Quality/Materials/Interior Design Language: RS5

Exterior Looks: a wash… too subjective and both cars offer understated aggression and build on the looks of the base coupes

Engine: a wash… very similar engines.  Both phenomenal, high reving, V8s that are certainly extinct in this era of forced induction on every car soon to be replaced with electric power plants.

Cabin Tech: RS5 – the RS5 offers in car Google maps, in car 3G and apps and the MMI system is just a lot nicer to use and more intuitive than even the updated BMW iDrive System

Comfort: a wash… both cars are extremely comfortable.  Shockingly so for near sports cars.  The US RS5 lacks dynamic ride control, but I found the factory coil over setup on 19’s (the standard wheels) provides a much nicer, yet also more confident, ride than my M3 did.  My M3 had EDC, but was also rolling on the optional 20’s.  I found no EDC setting was quite right.  Sport was excellent in terms of handling, but too jarring for the street.  Comfort was better, but introduced body roll which I didn’t like.  All in all my opinion on ride quality remains solidly anti “big wheels”.  My rule of thumb is that the rubber to rim ratio should be the greater of all the possible combinations.  If the wheel well can support 20’s, I like 19’s.  If it can support 19’s, I like 18’s.  Very difficult, by my but measurement, to compensate for a lack of rubber with dampening.

Performance: a wash, but I will say that the RS5 makes the performance much more accessible and consistently deliverable for mere mortals

Steering: a wash… both cars are fantastic.  Caveat… My RS5 does not have the electrically assisted dynamic steering.  EPS is the devil and I hate it.  M3 steering is possibly just a hair more direct.

Soundtrack: RS5 – the exhaust note and engine sound of the RS5 has almost no equal among sport coupes.

Practicality: RS5 – interior room is about the same.  Trunk space is about the same.  The RS5 is a tad larger and heavier of a car, but close enough to keep them in the same class.  But the RS5 has Quattro.  And Quattro is a thing of genius.  Quattro eats snow.

Transmission: RS5 – this one is close.  The M3 DCT is great.  It’s smooth and performant.  But the S-Tronic is shockingly even better

Fun Factor: TOUGH one… at the limits, in a highly dynamic situation, the M3 brings it all together.  The M division roots go deep and the M3 “is racecar”.  The RS5, on the other hand, is an Autobahn barnstormer.  That’s what it was built to be.  It’s a land missile that feels ridiculously fast on the street yet is easy.  Despite that, it surprises you often with just how nimble it is and just how far out its limits lie.  While it doesn’t provide the near telepathic feedback of the M3, it’s light years behind any lesser automobile and it sounds better than just about everything.  Really a tough call here.

Did I leave anything out?  I don’t know, maybe.  But this entry is getting too long and you get the idea.  Overall I am extremely pleased with the RS5 even though I’m still only lukewarm on Audi in general.  At $77,000 I think it was too expensive, but that’s how things are going these days and, next to the $100k+ GTR, it actually feels like a bargain.  Yes Godzilla brings insane performance, but as an actual car that you use to drive somewhere other than Cars and Coffee you leave a lot on the table and it is now 911 money.  Compared to the M3 I’d say these are very very close competitors, but the one place the M3 simply cannot compete is all-weather utility and for that alone, the RS5 is the winner!  Now let’s see how it lasts!

For those just joining, in the first 3 entries we introduced NSX overlay networking to a vCenter environment, performed all of the required base configuration, and created our first upper layer network service appliance, the NSX Edge.  To review, the NSX Edge is a lot like the old vShield Edge in that it is a firewall appliance with NAT, firewall, load balancing, VPN termination, but very much unlike the original in that it is a massively capable router as well.  In addition every one of those capability areas has been dramatically expanded since the vShield days.  In short it is a  full features virtual perimeter appliance that can stretch up to layer 7 as well as handle internal  routing between (virtual) subnets (VXLAN vwires or, in NSX parlance, layer 2 domains on the logical switch) as well.

One of the most exciting things about a true overlay network like NSX, and why the Nicira acquisition was such a smart one by VMware, is that the virtual network constructs don’t just stop at the perimeter.  It’s a really powerful thing having this truly virtual, software based router available as a click-to-deploy console.  Some of the limits of the Edge, though, are that it is limited to 10 interfaces (just like vShield) and it also has a lot going on that is all perimeter based.  What if you just want a pure router?  Well the fantastic news here is NSX has you covered.  The logical routing appliance is a subset of the NSX Edge, focused just on routing, that also adds the capability to be a bridge.  This opens a huge range of possibilities. Before we get into those possibilities though, let’s get one up and running!

As always, head to the web client Networking & Security plugin.  Select NSX Edges and click the green “+” to add.  Last time we deployed an Edge Services Gateway, this time we’re doing a “Logical (Distributed) Router”.  Once again we give it both a name (for the VM appliance) and hostname (for the actual OS) and have the option of enabling High Availability:

Screenshot 2014-09-18 15.29.32

Next we enter a password.  Once again, extremely strong password rules are enforced:

Screenshot 2014-09-18 15.29.56

So far the wizard is identical  Select a datastore to which the logical router appliance will be deployed.  In my case once again sending it to the vSAN:

Screenshot 2014-09-18 15.30.39

Next up is the interface configuration.  Finally we diverge a bit from the NSX Edge wizard.  For the logical router we need to configure a dedicated management interface (management plane) in addition to adding actual routing interfaces (control and data planes).  For the management interface I’ve created a port group on the vDS which shares a VLAN with my physical hosts vSS since this is where all of my base VMs reside (vCenter, vCS, etc).  After connecting the management interface to a port group add a valid IP:

Screenshot 2014-09-18 15.37.06


With the management interface configured, we’re back to business as usual.  Here we are adding the routing interfaces exactly the same as we did on the Edge.  Name them and classify them as either internal or uplink, then connect them to an appropriate port group.  It is important to note that interfaces on the logical router require VLAN tagging in order to connect.  That is worth repeating.  If you create an interface on the logical router, and connect it to a port group which is set to VLAN0, it will allow you to do this and deployment will fail.  Not quite sure why this is, or why it isn’t a pre-requisite on the Edge, but it is something to be aware of for the logical router.  Once again, add IPs for any created interfaces and adjust the MTU as needed:

Screenshot 2014-09-18 15.36.09

If HA was selected we configure it once interface configuration is complete:

Screenshot 2014-09-18 15.37.12

One final check before deploying to verify that everything is correct:

Screenshot 2014-09-18 15.37.16

And viola!  As long as resources are available to host the appliance, and VLAN tagging has been set in the connected port groups, the logical router will quickly deploy:

Screenshot 2014-09-19 20.12.47


Our router has been deployed, but we haven’t done any configuration.  Double click on the router name to switch to the logical router configuration pages and be prepared for a treat.  This is a seriously powerful virtual network element!  On the manage tab we are confronted with a whole boatload of options.  The first section, “Settings”, gives us a Configuration overview first.  Here we can change the management interface config, as well as the HA parameters.  We can also setup syslog anddownload logs for tech support troubleshooting.  We can also deploy additional logical routers in the bottom pane:

Screenshot 2014-09-19 20.49.03

Under interfaces we can see the interfaces created during initial deployment.  We can also add more using the same UI.  A staggering 999 interfaces, with 8 of them being uplinks, can be created!  Of course bandwidth of the underlying host should be scaled appropriately for the uplinks:

Screenshot 2014-09-19 20.49.13

The Firewall section provides a really easy to use UI for creating ingress and egress filters.  Very intuitive: name, source, destination, service, permit/deny:

Screenshot 2014-09-19 20.49.34

The Routing section is where things get really interesting and the true power of the logical router is unlocked. It starts off with top level configuration.  Set a default gateway for the router itself, if appropriate, and then enable the dynamic routing options. OSPF and even BGP(!) are supported.  This is fantastic as, with these two protocols, 80% of both internal and external integration cases are covered.  We can also configure logging in this section:

Screenshot 2014-09-19 20.49.53

In the event that the logical router is being deployed into an environment without dynamic routing, static routes can be created.  Once again intuitive, Interface, network, next hop, MTU (powerful – per route MTU, this is fantastic), and of course a description field:

Screenshot 2014-09-19 20.50.08

The static route dialogue is straight to the point:

Screenshot 2014-09-19 20.50.14

The OSPF tab is a bit overwhelming for anyone not familiar with the protocol, but will look like home to anyone who is.  The fundamentals needed to get the logical router working in an OSPF area are here: protocol and forwarding addresses, definition of the OSPF area, and mapping of the area to an interface:

Screenshot 2014-09-19 20.50.25

Adding an area we enter an ID, select a type (normal or NSSA – RFC 1587 “not so stubby area” for redistributing external BGP routes into OSPF) and an authentication method (MD5, password or none) as well as the authentication value (password or MD5 hash):

Screenshot 2014-09-19 20.50.34

Once the Area is setup, we map it to an interface. Option here to ignore the interface MTU as well as advanced options to control protocol intervals, and set both priority and cost:

Screenshot 2014-09-19 20.50.42

OSPF has our internal needs covered, so let’s move on to BGP to cover our external inter-org routing requirements.  Once again, if you know BGP this is familiar territory. Up top we enable the protocol and assign our AS (Autonomous System Number – the identifier by which BGP peers identify each other and associate IP ranges with an endpoint).  We also add our Neighbors – BGP peers with whom we are establishing a BGP routing relationship:

Screenshot 2014-09-19 20.50.53

Peer configuration requires knowing a bit about your neighbor obviously.  The remote organizations AS number is of course the starting point along with assigned IP address, as well as protocol and forwarding IP addresses.  We can also enter timings and weightings and assign mutual authentication password.  Once the foundation has been laid, we can also optionally add BGP filters:

Screenshot 2014-09-19 20.51.04

Adding a filter we set a direction (ingress or egress) and an action (permit/deny) on a specific network expressed in CIDR block format.  We can also use IP prefix conditionals (GE – greater than or equal to, LE – less than or equal to) to apply to a range:

Screenshot 2014-09-19 20.51.13

We’ve got internal routing.  We’ve got external routing.  Let’s link em!  The Route Redistribution tab let’s us do just that:

Screenshot 2014-09-19 20.51.24

First we establish the IP prefixes for route redistribution. Name and CIDR notion network definition:

Screenshot 2014-09-19 20.51.27

Next we create the redistribution criteria.  Select the prefix (network defined above) and then set direction.  The “learner” protocol is where the route is being distributed, the “learning from” entry is where the route is originating.  Origination can be OSPF, BGP, static or directly connected networks.  Destination can be OSPF or BGP:

Screenshot 2014-09-19 20.51.33

Last but not least we have bridging.  Yes, this appliance can be a proper Ethernet bridge as well giving us fantastic layer 2 options for scenarios that need them.  First step on the bridging tab is to add a bridge configuration:

Screenshot 2014-09-19 20.51.40

Very easy configuration: add a name for the new bridge group and the two port groups that are being bridged:

Screenshot 2014-09-19 20.51.46


As you can see there are a huge number of virtual guest environment use cases that can be covered with the rich set of capabilities represented by both the NSX Edge and Logical router.  Next entry we’ll spend some time considering possible architectures that would be difficult before NSX, but become simple once it has been deployed.  Stay tuned!

There is still work to be done on NSX, but I got a number of inquiries asking about how I have the lab server setup from a networking perspective so I thought it would be useful to have a brief intermission and take a look.  Let’s start with a picture:

From the hardware perspective, here is how it breaks down:

  • Core Switch: Netgear GSM7224V2 – this is a fully managed layer 3 switch with 802.1Q VLAN support, 24 1Gb/s ports, 2 SFP+ modules, LAG/LACP and obviously routing
  • Physical Host:  Dell T620 – the beast is setup with 192GB ECC LVDDR3-1333 DRAM, 2 x Intel® Xeon® E5-2650L v2 (1.7GHz/10-core/25MB/8.0GT-s QPI/70W), 2 x 750W PSU, 8 x 2TB Western Digital Red NAS drives, PERC H710 RAID controller with 512MB cache, iDRAC Enterprise ILO board, 2 x 120GB Intel SSD, 4 port Intel i350 1Gb/s NIC, 2 port Broadcomm BCM57810 10Gb/s NIC
  • Firewall:  these days I actually run a dedicated physical firewall in the shape of the (now defunct) McAfee UTM SG720.  It’s no ASA, but it’s actually surprisingly powerful and capable for perimeter defense in a home lab.

Of course hardware porn aside, from a networking perspective, the key statistics above are the 6 1Gb/s ports (no 10Gb/s in the lab unfortunately, so the Broadcomm gets to be bored doing 1Gb/s duty).

In terms of logical configuration, I have allocated the NIC’s to 5 discrete virtual standard switches:

  • vSwitch0: This is the primary VSS and has been allocated two physical ports.  It hosts the following port groups:
    • VM Network: the attach point for any VMs running on the physical host – (VLAN 200)
    • Management Network (vmkernel): primary management network used for management traffic and VM FT – (VLAN 500)
  • vSwitch4: This VSS is dedicated to storage and has one vmkernel attach.  Storage and VMotion traffic traverse this link – (VLAN 200)  Note that it shares the VM network.  My two NAS devices each only have 2 gigabit ports and connect directly to both my client network ( and the lab (  They also need to be accessed by the guest VMs constantly.  Rather than put a routing boundary in the middle, I opted to just flatten out storage access to VLAN200
  • vSwitch1: VSS1 is dedicated to the first nested ESX environment.  This environment contains 3 vESXi guests which live in the same vCenter as the physical host (vCenter 1)
  • vSwitch2: VSS2 is dedicated to the second nested ESX environment.  This environment contains 3 vESXi guests which live in their own vCenter (vCenter 2).  SRM is up and running between the two vCenters
  • vSwitch3: VSS3 serves as a DMZ as well as the provider network (ext network) for vCD and NSX.  It is 192.168.99 (VLAN990) and uplinks to a firewall managed DMZ

In terms of VMware advanced networking (vDS, vCD, vCNS, NSX), I limit this to the nested environments.  It makes configuration changes (including full teardown) super easy even if the entire network traffic flow picture gets (pretty damn) confusing.  Some things to remember about doing this:

  • Enable promiscuous mode on the vSwitch the nested ESX guests attach to
  • Allow forged retransmits on the same vSwitch
  • In the guest properties be sure to select ESX as the actual OS and enable chipset virtualization passthrough support

The reason for this is that normal vSwitch behavior is to assume that a guest is only responsible for itself (meaning traffic destined for the guest OS is actually destined for applications on the guest OS).  In the case where the guest is actually a nested ESX host, the traffic is originating from its guests which have their own vNICs and MAC addresses.  Any traffic inbound to the netsted ESX guest is actually headed for an application in one of its guests.  As such the vSwitch sees lots of what appear to be alien MAC addresses heading for the nested ESX guest that it will want to drop.  These settings prevent that from happening and unlock hypervisor on hypervisor potential.

Over the past two entries we have gone from being NSXless to having a full NSX foundation laid in a fairly painless set of steps.  Next it is time to actually start to use the technology for something interesting.  The true power of overlay network comes from two key areas: agility in the creation and management of layer 2 domains and the collapse of higher layer capabilities into the compute plane.  We’ve seen the former in action with VXLAN and the way NSX leverages the VXLAN foundation to build managed dynamic L2 environments.  The next piece is layer 3 and above.  Being able to actually route, load balance, filter and intelligently direct traffic within the hypervisor enables enormously powerful consumption models.  Past versions of VMware vShield Manager provided a simple “Edge” device which had load balancing, firewall, NAT, static routing, VPN (IPSEC and SSL) and DHCP capabilities.  It was fairly similar to a virtualized version of a high-end home office firewall/router appliance.  There were neat bells and whistles like high availability with very smart failover and the ability to have up to 10 interfaces for guest network usage.  It also came in a host of sizes based on load and throughput requirements so it was resource efficient.  So why change it?  Well the good news is that NSX provides an additive experience.  The traditional vShield type Edge is still available in NSX, but vastly improved.  In addition, NSX provides the ability to deploy a proper virtualized router.  A device which can actually participate in OSPF domains!  That’s great stuff and is a capability of both the Edge appliance as well as the dedicated logical router appliance which is a subset of the Edge functionality plus bridging which we’ll explore in the next entry.  For now let’s get started first by creating a Edge device.

As with all NSX operations, we initiate from the Network & Security plugin.  This time in the left hand menu pane we’re selecting “NSX Edges”.  One interesting footnote; I actually lost my NSX plugin in the web client and nothing seemed able to bring it back.  Skipping right to the resolution, the culprit actually turned out to be a stalled Windows update to .NET.  Once I got Windows fully updated and WU healthy, vCenter magically got itself back into shape (following a final reboot).  The moral of this story, to me at least, is that we really need a containerized version of vCenter running direct on hypervisor.  Anyhow, enough of that.  From NSX Edges, we’re going to click the green plus sign in order to add one.  The New NSX Edge dialogue gives us a few interesting options right off the bat.  First, we can see the traditional Edge Services Gateway (which we’re selecting this round).  Below it, however, we can see this new construct the “logical router”, as discussed above.  We will deploy one of those as well.  Lastly we can see the option to deploy the Edge VM in an HA state.  I’m leaving this deselected for the lab as resource usage is more important than availability.  The last step is to provide both a descriptive name and hostname for the VM, then click Next:

Screenshot 2014-09-17 01.35.54

Next up is to set the appliance password.  Note the password policy is very strong here.  12 characters, upper and lower case mixed, numeric and at least one special character.   A pain for the lab, but a good practice for production anyhow:

Screenshot 2014-09-17 01.36.14

With the password set we move on the the deployment options.  Select a datacenter to deploy the VM into as well as a size.  Size determines the number of vCPUs which will be allocated as well as the RAM.  Obviously the larger the VM, the higher the volume of traffic it can process. Common use cases for the larger sizes would be a high number of IPSEC tunnels or an extremely complex firewall ruleset.  There is also an option to turn off automatic generation of control plane traffic flow service rules.  This is a case where this should only be selected if a specific design and implementation requires control beyond what automatic generation can provide.  Last step is to add resourcing info for the Edge appliance VM:

Screenshot 2014-09-17 01.37.01

Select a cluster, datastore and (optionally) a host.  Note, deploying to vSAN again just to show off!

Screenshot 2014-09-17 01.37.31

The next step is where the real magic begins.  Here we are creating and configuring the network interfaces of the Edge appliance.  If you think about what we’re doing here from the perspective of legacy network engineering, it really is amazing.  Through an easy wizard driven GUI, we’re literally creating and addressing network uplinks.  Extremely cool.  Each interface is classified as either “Internal” or an uplink “external” and should have corresponding connectivity which matches.  I point internal interfaces towards the NSX logical vswitch (VXLAN vwire environment) that the guest VMs will attach to, and external interfaces at a port group that has a physical route path out of the lab network.  In this respect the new edge is very much like the vShield Edge in a vCloud Director scenario, where internal interfaces would be connecting to a tenants organizational network while external interfaces would be connecting to the provider external network.  After selecting the type of interface, provide it a name and then set its vswitch connectivity.  The last step is to provide an IP address for the new interface.  Of course this IP should be valid for the vswitch and port group the interface is being connected to:

Screenshot 2014-09-17 01.38.51

With all options complete, we can now add another interface.  There should be at least one internal and one external if the guest VMs will need to reach outside of the overlay network:

Screenshot 2014-09-18 00.25.43

With both interfaces created and configured, we can move forward to the next step:

Screenshot 2014-09-18 00.25.52

Since this is a gateway device, we should provide it with a default gateway (although this is optional).  Select the appropriate interface and provide the IP of the next hop router on that subnet:

Screenshot 2014-09-18 00.26.04

The last step is an opportunity to create a default firewall policy.  Very useful for setting baseline security so the new appliance comes up configured.  HA parameters can also be set in this dialogue box if the HA option was selected up top:

Screenshot 2014-09-17 01.45.28

With all steps completed it is time to review and submit!

Screenshot 2014-09-17 01.45.32If everything is correctly configured, and there is sufficient host resourcing available to support he creation of the configured Edge appliance VM size, the appliance will deploy and come online:
Screenshot 2014-09-18 00.33.56

We’ve got a working Edge, so let’s see what it can do!  For anyone familiar with the vShield Edge, this will be semi-familiar territory, but there is also a ton of new capability.  Doubleclick on the newly created Edge device object to bring up the configuration page.  The first stop is to head over to the manage tab.  Look at all of those groupings!  There are separate config hierarchies for Firewall, DHCP, NAT, Routing, LBS, VPN, SSL VPN and grouping which makes things very intuitive.  Let’s start with the top level Settings group.  First up is the Configuration page.  Here we can modify the syslog configuration and logging options for the appliance.  We can also check on what services have been enabled at a glance.  There are also sections to modify both the HA configuration and the DNS settings.  Finally, we can deploy a new appliance from this panel as well:

Screenshot 2014-09-19 22.03.36

Moving one level down we arrive at the configuration page for the interfaces.  Here we can see the aforementioned 10 available interface slots, two of which we configured during the deployment steps.  We can modify or delete those, as well as add new ones:

Screenshot 2014-09-19 22.03.46

The final configuration area under Settings is for certificate management.  This appliance reaches up to layer 7 and also supports VPN, so it is likely that it will need to be configured with one or more public certs.  This panel makes complex cert management very easy:

Screenshot 2014-09-19 22.03.58

The next top level configuration grouping is for the Firewall.  Very clean presentation with all rules listed in tabular format.  Click the green “+” to create a new rule (providing the expected source, destination, service and action values) or delete or modify existing ones.  They are processed in order and can be moved.  Keep in mind that the bottom rule will “catch all”, but only the first rule that matches a traffic pattern will be applied (in other words a higher level “permit” will take precedence over a lower level “deny”, but would be rendered superfluous by a higher level deny), so plan rule strategy accordingly:

Screenshot 2014-09-19 22.04.06

The next settings group is for the DHCP server.  I can’t stress enough the utility of this option.  When you consider software defined datacenter strategy, and the automated deployment and configuration of customer environments, having a way to bring guest OS instances onto the network before the first one is deployed is extremely powerful.  Being able to manage (and orchestrate) that capability right in the network edge device is an even bigger bonus.  The first stop is the Pools config block and the options here are very straightforward for anyone familiar with DHCP.  You can enable the service, configure logging and create scopes (IP ranges that the DHCP server will service):

Screenshot 2014-09-19 22.04.13

With the pools defined we can view and configure the Bindings.  Bindings in this context are static assignments.  What this means is that the DHCP server can actually be prepopulated with IP associations by VM ensuring that a specific guest instance will get a specific IP:

Screenshot 2014-09-19 22.04.20

Next up is the NAT configuration.  As an edge device, this section is critical. The rules come in two flavors, SNAT and DNAT.  SNAT are source NAT rules which are for egress.  The translate private internal IP address to the outbound gateway uplink address.  DNAT are destination NAT rules which are for ingress.  They are applied to one of the external gateway IP addresses and translate a specific inbound port to an internal address (changing the port as well if needed).  And of course it goes without saying that in order to NAT traffic and have it flow, you also need corresponding firewall rules that permit it.  The top level UI is very minimalist, click the green “+” to create a rule as usual.

Screenshot 2014-09-19 22.04.27

Here we see the options for a DNAT.  We have the original IP range and protocol (TCP or UDP), as well as the original port range.  Corresponding configuration must also be provided for the translation side of the equation – both IP and port range:

Screenshot 2014-09-19 22.04.44

SNAT is simpler.  Set the interface the rule is being applied to and provide both an original IP range and a translated IP range to start NAT’ing internet traffic out:

Screenshot 2014-09-19 22.04.52

The Routing section is where things get really interesting and the true power of the new NSX flavored Edge is unlocked. It starts off with top level configuration.  Set a default gateway for the router itself, if appropriate, and then enable the dynamic routing options. OSPF and even BGP(!) are supported.  This is fantastic as, with these two protocols, 80% of both internal and external integration cases are covered.  We can also configure logging in this section:

Screenshot 2014-09-19 22.09.34

In the event that the logical router is being deployed into an environment without dynamic routing, static routes can still be created.  Once again intuitive, Interface, network, next hop, MTU (powerful – per route MTU, this is fantastic), and of course a description field:

Screenshot 2014-09-19 22.09.49

The OSPF tab is a bit overwhelming for anyone not familiar with the protocol, but will look like home to anyone who is.  The fundamentals needed to get the logical router working in an OSPF area are here: protocol and forwarding addresses, definition of the OSPF area, and mapping of the area to an interface:

Screenshot 2014-09-19 22.10.04

Adding an area we enter an ID, select a type (normal or NSSA – RFC 1587 “not so stubby area” for redistributing external BGP routes into OSPF) and an authentication method (MD5, password or none) as well as the authentication value (password or MD5 hash):

Screenshot 2014-09-19 22.10.08

Once the Area is setup, we map it to an interface. Option here to ignore the interface MTU as well as advanced options to control protocol intervals, and set both priority and cost:

Screenshot 2014-09-19 22.10.13

OSPF has our internal needs covered, so let’s move on to BGP to cover our external inter-org routing requirements.  Once again, if you know BGP this is familiar territory. Up top we enable the protocol and assign our AS (Autonomous System Number – the identifier by which BGP peers identify each other and associate IP ranges with an endpoint).  We also add our Neighbors – BGP peers with whom we are establishing a BGP routing relationship:

Screenshot 2014-09-19 22.10.22

Peer configuration requires knowing a bit about your neighbor obviously.  The remote organizations AS number is of course the starting point along with assigned IP address, as well as protocol and forwarding IP addresses.  We can also enter timings and weightings and assign mutual authentication password.  Once the foundation has been laid, we can also optionally add BGP filters:

Screenshot 2014-09-19 22.10.26

Adding a filter we set a direction (ingress or egress) and an action (permit/deny) on a specific network expressed in CIDR block format.  We can also use IP prefix conditionals (GE – greater than or equal to, LE – less than or equal to) to apply to a range:

Screenshot 2014-09-19 20.51.13

IS-IS is an internal, link state based, routing protocol.  An alternative to OSPF, the key difference is that while OSPF was built as a pure layer 3 control plane protocol, IS-IS starts with a layer 2 view of its Intermediate Systems.  As such it is a core component of the various IEEE advanced bridging protocols: 802.1ad, q and h.  This is an extremely powerful option to have here.  If you consider integrating with carrier stretched layer 2 topologies (like VPLS), ability to support the 802.1a family protocols (Shortest Path Bridging, Provider Bridging and Provider Backbone Bridging) can spell the difference between being able to actually participate in the extended L2 domain vs having the virtual network environment relegated to its own L3 domain (and consequently new IP space).  It is also a solution for eliminating the need for yet another level of overlay abstraction, the SSL VPN or IPSEC TAP VPN which, while still available as options, create additional overheard.   The base UI for configuring IS-IS allows us to configure a system id and Intermediate System type, create Areas, and map them to an interface:

Screenshot 2014-09-19 22.10.38

Creating IS-IS Areas is easy:

Screenshot 2014-09-19 22.10.46

Interface binding follows the same convention as BGP:

Screenshot 2014-09-19 22.10.52

We’ve got internal routing with OSPF.  We’ve got external routing with BGP.  Let’s link em!  The Route Redistribution tab let’s us do just that:

Screenshot 2014-09-19 22.10.59

First we establish the IP prefixes for route redistribution. Name and CIDR notion network definition:

Screenshot 2014-09-19 22.11.03

Next we create the redistribution criteria.  Select the prefix (network defined above) and then set direction.  The “learner” protocol is where the route is being distributed, the “learning from” entry is where the route is originating.  Origination can be OSPF, BGP, static or directly connected networks.  Destination can be OSPF or BGP:

Screenshot 2014-09-19 22.11.08

Phew.  The routing configuration was intense!  In an upcoming entry I plan to talk through various use cases that were once sealed off which NSX can unlock and the key to many of those is in the power of these routing capabilities.  For now though, let’s move on to the Load Balancer configuration.  Up top are the basic service controls plus options for logging, “Acceleration”. and “Service Insertion”.  These last two require some explanation.  “Acceleration” refers to the load balancing engine that will be activated in the appliance.  Toggling this option switches between the faster Layer 4 engine (which obviously makes decisions based on TCP connection state) and the slower, but far more flexible, Layer 7 engine which enabled the ability to make decisions at the application layer.  Obviously the right choice here is completely dependent on use case.  “Service Insertion” allows the Load Balancer to integrate with third party appliance solutions:

Screenshot 2014-09-19 22.11.16

The next configuration group is “Application Profiles” which is where the L7 and L4 rules engines are configured.  The bottom pane allows certificate configuration.  Absolutely vital when working at the application layer where much traffic will be SSL/TLS:

Screenshot 2014-09-19 22.11.25

Fantastic options here for defining an Application Profile.  Protocol obviously; TCP for L4, HTTP and HTTPS for L7.  HTTP redirect is fully supported and a URL can be entered here.  The ability to determine pathing and redirect via URL is critical for an application focused load balancer.  Persistence and persistence mode can be set and a cookie name provided for cookie based persistence.  In addition to these options, there is a toggle for enabling an “X-Forward-for-HTTP” flag into the forwarded header.  This option is for support of proxy environments.  The field is set to the actual originating IP so the load balancer can make decisions based on true source if desired.  Without this field, in a proxy environment, the IP of the proxy will be seen as the source.  Finally, comprehensive configuration for certificate assignment, auth method and cipher can be set here as well:

Screenshot 2014-09-19 22.11.31

Application Profiles now created, we can move on to the Service Monitoring configuration.  Here we can create monitors based on protocol and set timing intervals to govern load balancer listening behavior:

Screenshot 2014-09-19 22.11.38

An example of creating a Layer 7 service monitor.  HTTP method can be set as well as a specific URL to watch.  In the “Expect” field we enter the literal string that indicates a match in the status line of the HTTP response.  Next we select the “Method” to be used to detect server status.  URL is the URL to be used in the sample request.  Next, if the method is set to POST, comes the data that should be sent to that URL.  For “Receive” we enter the expected response. In the “Expect” field we enter the expected response. If it is not matched the monitor does not try to match the Receive content.  Finally, in the option extension area, we can enter additional monitoring parameters as key/value pairs.  These are predefined (example: warning=10 sets the load balancer to trigger a warning on the service endpoint if a response is not received within 10 seconds).

If you haven’t worked with advanced load balancers before, this may be a bit confusing but if you think through it it’s actually very straightforward.  The point of a load balancer is to provide a single front end to a group of servers in scenarios where the application can “scale out”.  So using a web server as an example, the name and IP address of the “server” are quite likely the logical server represented by the virtual IP of the load balancer.  Behind the load balancer sit any number of actual web servers that handle the traffic.  The load balancer, to do its work, needs to be able to do two things.  First is decide how to distribute traffic, and second is to determine how many servers it is representing.  The first one comes down to load balancing method selection.  It might be a simple round robin which treats the known servers as a list, or it could be as complex as a hash on the originating IP which matches clients to servers based on layer 3 network associations.  Server members and health similarly can be accomplished by a number of methods.  The service monitor capability discussed above represents one of the more advanced ones.  In this case the load balancer will literally have a layer 7 relationship with its member servers and use a URL connect / URL response to determine if the servers are alive:

Screenshot 2014-09-19 22.11.43 Screenshot 2014-09-19 22.11.51

Screenshot 2014-09-19 22.11.55 Screenshot 2014-09-19 22.12.02

Screenshot 2014-09-19 22.12.07 Screenshot 2014-09-19 22.12.13

Screenshot 2014-09-19 22.12.18 Screenshot 2014-09-19 22.12.31

Screenshot 2014-09-19 22.12.37 Screenshot 2014-09-19 22.12.45

Phew!  That’s a ton of options and a really broad range of capabilities!  This is a good place to break for now.  Next up we will create a logical router!



Last entry we got the NSX Manager up and running in vCenter after a quick overview of the technical requirements.  Next up it is time to actually implement the SDN in our environment.  The first step is to login to the vCenter web client and select the Network & Security solution from the Home tab.  If you recall the rundown of NSX components from the last entry, our next task is to install the hypervisor level integration.  To do this we need to prepare the hosts.  This is similar to implementing VXLAN and results in a VIB being installed.  Click on the Install option in the menu pane, then select the Host Preparation tab:

Screenshot 2014-09-16 12.06.06

Under Host Preparation, we can see any clusters in our vCenter configured with a vDS.  In our case there can be only one!

Screenshot 2014-09-16 12.06.14

Clicking the “Install” hyperlink in column 2 will trigger the install after a quick confirmation:

Screenshot 2014-09-16 12.06.25

The Manager will start the download of the VIB to the hosts, and trigger the scripted install.  All of the usual automated workflow orchestration for VIB installation applies.  Lots of things can trip this part up, mostly attributable to host or network misconfiguration.  Our environment is sparking clean so we have nothing to worry about!

Screenshot 2014-09-16 12.06.33

Working away at each host in parallel…

Screenshot 2014-09-16 12.06.51

And just like magic we…. Wait… What the heck is this?!  Hmmm… So much for our clean environment!  Looks like it failed. Luckily there is a handy “Resolve” hyperlink.  Let’s click it.

Screenshot 2014-09-16 12.07.11

A bit more thinking and POOF!  As if by magic we’re good.  So what the heck happened here?  Well in some cases it seems that the install actually requires a host reboot.  The workflow triggered by resolve will perform this reboot so be aware of that before clicking.  It should probably mention this when triggered, but the good news is HA/DRS is there for just such a situation, right?  Well I’m not sure because I can’t be 100% certain that it staggered reboots.  In any event, maintenance mode is probably a great idea when doing massive configuration changes like migrating to SDN!  And in any event it worked so all is well…

Screenshot 2014-09-16 15.18.08


Notice there is a hyperlinked “Configure” next to our cluster?  These are great UI clues in the NSX manager.  Go ahead and click configure to prepare the VXLAN configuration.  There are a few things we need to enter here.  We select our vDS under Switch, enter the VLAN id of the transport VLAN (if applicable), the MTU size of the VXLAN uplink vmkNIC (note – 1600), select a vmkNIC IP addressing scheme – we are going to switch this to IP pool in a second, select a vmkNIC Teaming policy (I chose failover, remember that EtherChannel must have matching switch config on the physical uplink switch) and lastly enter a VTEP id:

Screenshot 2014-09-16 20.57.57

Creating a new IP Pool for VXLAN use is easy.  Simply provide a name and IP subnet info, as well as a range:

Screenshot 2014-09-16 20.58.40

Here we can see the completed VXLAN config dialogue:

Screenshot 2014-09-16 20.59.00

With the configuration applied, we now see additional details populated for the cluster including VTEP id and failover policy:

Screenshot 2014-09-16 20.59.35


VXLAN is up and running in NSX, so let’s go ahead and finish off the config set.  Click on SegmentID to create the SegmentID pool which will be used by NSX to allocate to VXLAN for the creation of vwires (dynamic VXLAN layer 2 domains).  We can also configure multicasting here.  Select a numerical range for the pool starting at 5000.  I selected 5000-5999 and left multicasting off:

Screenshot 2014-09-16 20.45.08

The last step in this config block is to setup the NSX Transport Zone.  Provide a name and select a mode for the Control Plane interaction. I select unicast which works well in a lab setting where scale isn’t a big deal.  In this case the control plane will be entirely managed by the NSX Controller.  Alternatively the control plane activity could be offloaded to the physical network via multicast.  The last option is a hybrid where local traffic replication is offloaded.  The hybrid is probably the best match for production scenarios because of it balance of control efficiency and scalability.  The last step is to add the prepared cluster to the transport zone:

Screenshot 2014-09-16 21.00.02

Here we can see the transport zone successfully added to the NSX configuration:

Screenshot 2014-09-16 21.00.09

Next up is deploying the actual overlay network or “logical switches” in NSX terminology.  Heady stuff!  There is a pretty daunting list of pre-requisites in order for this process to work correctly.  I’ve copied them directly from the implementation guide for reference and I will talk through each one because they require explanation:

  • You must have the Super Administrator or Enterprise Administrator role permission to configure and 
    manage logical switches: This one is a no brainer.  Have right permissions before configuring.  Basically you need super user for this.
  • Network virtualization components must be installed on the clusters that are to be part of the logical switch: make sure that the hosts have been prepared (the above procedure)
  • You have the minimum required software versions: this is standard stuff.  Make sure that the version compatibility matrix is green between vSphere/vCenter/NSX
  • Physical infrastructure MTU is at least 50 bytes more than the MTU of the virtual machine vNIC: this one is trickier.  So the physical infrastructure MTU we can figure out.  In virtualization, we can look at any flavor of vswitch as utilizing the actual physical NIC of the host as uplinks.  So a given vSwitch has N virtual ports connected to virtual NICs connected to virtual machines, but also has ports connected to host vNICs which actually map to real physical links.  The pre-req here is to ensure that the Maximum Transmission Unit size on the physical NIC is 50 bytes larger than the VM vNIC.  In our case the “physical” NIC is really the vNIC on the ESXi guest VM since we are nested.  To check that we go to the host configuration and actually edit the settings of the VMkernel adapter under Networking (not the physical adapter):

Screenshot 2014-09-16 16.34.33


Our MTU is set to 1500.  That doesn’t bode well.  1500 byte MTU is standard so almost certainly the “virtual machine vNIC” MTU is also set to 1500.  Of course now it’s just a matter of figuring out just what the “virtual machine vNIC” refers to!  To understand the answer to this it is important to understand how an overlay network really works.  Consider this diagram:


The easiest way to wrap ones mind around overlay networking is to walk through a “day in the life of a packet”.  Remember that the guest OS has no clue that it’s being virtualized (for the most part, but close enough for this discussion).  It simply formulates Ethernet frames and sends them through the NIC driver.  An Ethernet conversation, of course, starts with an ARP broadcast to find the destination Ethernet address associated with the IP address you’re attempting to connect to.  This ARP query is processed by the vNIC the way a physical NIC would and it is put “on the wire”.  Of course in this case “on the wire” means on the virtual switch hosted by the hypervisor.  If the destination address exists within the ARP table of the vswitch (meaning a VM also attached to the same vswitch and running on the same host), then the ARP query is passed to that host and the conversation never leaves the hypervisor.  If this is not the case however (which means the destination VM lives on another host – common in a vDS environment even if the guests are on the same logical network), then the frame is sent down toward the physical NIC which is acting as the vSwitches uplink.  In an overlay scenario the frame is intercepted by the handler before it gets to the physical NIC driver on the host.  This is why for VXLAN we have to install a VIB.  The handler catches the frame and then handles it.  This means using its own logic to determine where the frame should go and then send it there.  In the case of VXLAN this means over a Virtual Tunnel Endpoint (VTEP) to the correct VXLAN destination over layer 3.  That is where the encapsulation comes into play.  So we are taking an entire 1500 byte Ethernet frame and packing it into another one to send over layer 3.  And this is where the larger MTU comes into play.  Using a 50 byte larger MTU ensures we don’t have to fragment every time an overlay frame is sent.  So what the pre-req is referring to is to set the physical MTU to 50 bytes larger than the VXLAN MTU (the “virtual machine MTU”) All documentation, however, really recommends setting it to 1600.  I feel the documentation here should have been more clear as “virtual machine MTU” is pretty ambiguous, but there it is.  Also worth noting is “logical switch” in NSX parlance is actually referring to “VXLAN”.  So with all of this in mind, we can go ahead and change that physical MTU of the VMKernel NIC that is attached to the vDS to 1600.

  • Managed IP address is set for each vCenter server in the vCenter Server Runtime Settings. See vCenter
    Server and Host Management: This is a straightforward vCenter config option found in the vCenter properties:

Screenshot 2014-09-16 17.07.54

  • DHCP is available on VXLAN transport VLANs if you are using DHCP for IP assignment for VMKNics:  this one catches me all the time as I don’t use a DHCP server on the transport VLAN.  If you don’t you need an IP pool or the configuration will break since vNICs will get an autoconfig address (169.254).  As we will see later, we’ll have an opportunity to associate an IP pool if we don’t want to deploy DHCP in the transport VLAN.
  • A consistent distributed virtual switch type (vendor etc.) and version is being used across a given transport zone. Inconsistent switch types can lead to undefined behavior in your logical switch: this is straightforward – you must use either vDS or OpenvSwitch (for example)
  • 5- tuple hash distribution should be enabled for Link Aggregation Control Protocol (LACP):  this one is the prescribed distribution algorithm that you should use if you are aggregating vSwitch uplinks using LACP.  In our case we are not using LACP, but in cases where it applies this is critical

With the background detail on the pre-requisites in mind, we can move forward with the next step which is Deploying the NSX Controller Node.  Head back over to the Installation section of the Network & Security plugin and select Management.  Here we can click “+” to add our first Controller under NSX Controller Nodes:

Screenshot 2014-09-16 17.18.48


We have a bunch of questions to answer to configure our first controller.  NSX Manager obviously refers to the NSX Manager we are pairing with, created in our first entry.  Datacenter should be set to the vDC we are supporting.  Cluster Resource Pool refers to the HA/DRS cluster we are NSX enabling.  Datastore is the datastore where the controller VM should be created (note in this case we’re installing to a vSAN datastore – more on that later) and host is the host on which it should be instantiated.  Connected to refers to the network to which the controller VM should attach while IP pool is how the node will be addressed.  Finally password sets the admin password for the controller appliance.

Screenshot 2014-09-16 17.23.04

A quick shot of the IP pool configuration.  Easy stuff:

Screenshot 2014-09-16 17.39.15

With everything configured for the Controller VM setup we can go ahead and click OK to create it.  The workflow will trigger and start operating:

Screenshot 2014-09-16 17.23.17


So how did it go?  Well it didn’t.  The workflow completed and the NSX Controller Nodes list stayed empty.  Recent tasks indicated an extremely generic error of “No hosts is compatible with the virtual machine”.  Hmmm.  Not super helpful:

Screenshot 2014-09-16 17.54.10

To get a deeper look, it’s time to SSH into the NSX Manager.  Hurray!  First we need to enable it, so head to the VAMI UI.  From the Summary tab we can easily spot the SSH Service and a handy “Start” button:

Screenshot 2014-09-16 17.53.11


With SSH running we can head to the CLI and check the log with the command:

show manager log follow

It’s a good idea at this stage to re-run the new Controller workflow to trigger the error again.  This is what I captured as the workflow log:

2014-09-16 23:07:13.070 GMT INFO http-nio- ControllerServiceImpl:422 - about to create controller: controller-3 IP =
2014-09-16 23:07:13.079 GMT INFO http-nio- AuditingServiceImpl:141 - [AuditLog] UserName:'vsphere.local\administra tor', ModuleName:'VdnNvpController', Operation:'CREATE', Resource:'null', Time:'Tue Sep 16 23:07:13.077 GMT 2014'
2014-09-16 23:07:13.085 GMT INFO DCNPool-2 VirtualWireInFirewallRuleNotificationHandler:59 - Recieved VDN CREATE notification for conte xt controller-3:Controller
2014-09-16 23:07:13.086 GMT INFO DCNPool-2 VirtualWireDCNHandler:43 - Recieved VDN CREATE notification for context controller-3:Control ler
2014-09-16 23:07:13.207 GMT INFO http-nio- TaskServiceImpl:101 - TF:Created Job with ID jobdata-3535
2014-09-16 23:07:13.221 GMT INFO http-nio- TaskServiceImpl:399 - TF:Scheduling Job jobdata-3535
2014-09-16 23:07:13.393 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXX7898D
2014-09-16 23:07:13.464 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXX6DEF2
2014-09-16 23:07:13.989 GMT INFO pool-9-thread-1 ImmediateScheduler:34 - TF:Schedule Now Job ID jobdata-3535
2014-09-16 23:07:14.002 GMT INFO taskExecutor-7 JobWorker:246 - Updating the status for jobinstance-13626 to EXECUTING
2014-09-16 23:07:14.059 GMT INFO taskScheduler-33 DeployOvfTask:173 - Deploying VM 'NSX_Controller_bc0ed3c4-5182-4448-af0c-dcb46eec3e9f ' using the OVF file.
2014-09-16 23:07:14.083 GMT INFO taskScheduler-33 OvfInstaller:335 - Resource pool id = 'resgroup-84'
2014-09-16 23:07:14.084 GMT INFO taskScheduler-33 OvfInstaller:336 - Datastore id = 'datastore-545'
2014-09-16 23:07:14.084 GMT INFO taskScheduler-33 OvfInstaller:339 - Host id = 'host-543'
2014-09-16 23:07:14.090 GMT INFO taskScheduler-33 OvfInstaller:141 - vApp candidate, Type = 'ResourcePool', Id = 'resgroup-84'
2014-09-16 23:07:14.095 GMT INFO taskScheduler-33 OvfInstaller:141 - vApp candidate, Type = 'ClusterComputeResource', Id = 'domain-c83'
2014-09-16 23:07:14.101 GMT INFO taskScheduler-33 OvfInstaller:141 - vApp candidate, Type = 'Folder', Id = 'group-h23'
2014-09-16 23:07:14.107 GMT INFO taskScheduler-33 OvfInstaller:141 - vApp candidate, Type = 'Datacenter', Id = 'datacenter-21'
2014-09-16 23:07:14.112 GMT INFO taskScheduler-33 OvfInstaller:141 - vApp candidate, Type = 'Folder', Id = 'group-d1'
2014-09-16 23:07:14.114 GMT INFO taskScheduler-33 OvfInstaller:359 - OVF is not being imported under a vApp and a folder has not been s pecified. Trying to associate with the root VM folder of the data center.
2014-09-16 23:07:14.460 GMT INFO taskScheduler-33 OvfInstaller:174 - Datacenter VM folder name = 'vm' id = 'group-v22'
2014-09-16 23:07:14.465 GMT INFO taskScheduler-33 OvfInstaller:105 - Searching for existing VM. Name = 'NSX_Controller_bc0ed3c4-5182-44 48-af0c-dcb46eec3e9f', Search root type = 'VIRTUAL_MACHINE', Search root id = 'resgroup-84'
2014-09-16 23:07:14.742 GMT INFO taskScheduler-33 OvfManagerImpl:120 - Creating OVF import spec.
2014-09-16 23:07:14.812 GMT INFO taskScheduler-33 OvfManagerImpl:122 - Created OVF import spec successfully.
2014-09-16 23:07:14.853 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'api_username'
2014-09-16 23:07:14.854 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'management_ip'
2014-09-16 23:07:14.854 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'keystore'
2014-09-16 23:07:14.855 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'api_private_cert'
2014-09-16 23:07:14.855 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'api_password'
2014-09-16 23:07:14.855 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'gateway_ip'
2014-09-16 23:07:14.856 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'cluster_ip'
2014-09-16 23:07:14.856 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'api_public_cert'
2014-09-16 23:07:14.857 GMT INFO taskScheduler-33 OvfInstaller:498 - Setting value for key 'netmask'
2014-09-16 23:07:14.857 GMT INFO taskScheduler-33 OvfInstaller:420 - Number of CPU cores set in the OVF import spec = '4'
2014-09-16 23:07:14.861 GMT INFO taskScheduler-33 OvfInstaller:425 - Number of CPU cores supported by the host = '1'
2014-09-16 23:07:14.862 GMT INFO taskScheduler-33 OvfInstaller:427 - Changing the number of CPU cores in the OVF import spec to '1'.
2014-09-16 23:07:14.862 GMT INFO taskScheduler-33 ResourcePoolVcOperationsImpl:320 - Importing VM into the resource pool.
2014-09-16 23:07:14.905 GMT INFO taskScheduler-33 ResourcePoolVcOperationsImpl:322 - Waiting for the HttpNfcLease to be ready.
2014-09-16 23:07:14.928 GMT DEBUG VcEventsReaderThread VcEventsReader$VcEventsReaderThread:301 - got prop collector update, but not for us:ManagedObjectReference: type = PropertyFilter, value = session[fa0b277c-c1cb-5f5c-cc78-b5e1e82a1bc4]5243239f-b58a-1539-609d-4d3e7e451 764, serverGuid = AB462F33-E3E0-4E86-BD55-984E0C95FBE1
2014-09-16 23:07:19.175 GMT INFO ViInventoryThread ViInventory:5004 - Virtual Center: Updating Inventory. new:0 modified:1 removed:0
2014-09-16 23:07:19.199 GMT INFO ViInventoryThread EndpointSVMUpdater:206 - Solution 6341068275337691137 is not registered
2014-09-16 23:07:19.239 GMT INFO ViInventoryThread ViInventory:1304 - 84/164 objects published.
2014-09-16 23:07:19.246 GMT INFO ViInventoryThread VimObjectBridge:943 - VimObjectBridge: Ending inventory update
2014-09-16 23:07:19.247 GMT INFO ViInventoryThread VimObjectBridge:222 - Processing 1 updates and 0 deletions for this transaction
2014-09-16 23:07:19.249 GMT INFO ViInventoryThread VimObjectBridge:229 - VimObjectBridge: Time taken to process transaction : 19
2014-09-16 23:07:19.249 GMT INFO ViInventoryThread ViInventory:1512 - Resolved, last version:220 num vc objs:90 num vimos:164
2014-09-16 23:07:19.683 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXX19BEB
2014-09-16 23:07:19.792 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXXC1AC3
2014-09-16 23:07:22.341 GMT INFO ViInventoryThread ViInventory:5004 - Virtual Center: Updating Inventory. new:0 modified:1 removed:0
2014-09-16 23:07:22.364 GMT INFO ViInventoryThread EndpointSVMUpdater:206 - Solution 6341068275337691137 is not registered
2014-09-16 23:07:22.400 GMT INFO ViInventoryThread ViInventory:1304 - 84/164 objects published.
2014-09-16 23:07:22.406 GMT INFO ViInventoryThread VimObjectBridge:943 - VimObjectBridge: Ending inventory update
2014-09-16 23:07:22.407 GMT INFO ViInventoryThread VimObjectBridge:222 - Processing 1 updates and 0 deletions for this transaction
2014-09-16 23:07:22.409 GMT INFO ViInventoryThread VimObjectBridge:229 - VimObjectBridge: Time taken to process transaction : 17
2014-09-16 23:07:22.409 GMT INFO ViInventoryThread ViInventory:1512 - Resolved, last version:221 num vc objs:90 num vimos:164
2014-09-16 23:07:22.722 GMT INFO ViInventoryThread ViInventory:5004 - Virtual Center: Updating Inventory. new:0 modified:3 removed:0
2014-09-16 23:07:22.746 GMT INFO ViInventoryThread EndpointSVMUpdater:206 - Solution 6341068275337691137 is not registered
2014-09-16 23:07:22.766 GMT INFO ViInventoryThread ViInventory:1538 - UNResolved, count:1 reason:Did not find child vimo for additional children in cache. By this time all children should have vimos in the cache
2014-09-16 23:07:23.048 GMT INFO ViInventoryThread ViInventory:5004 - Virtual Center: Updating Inventory. new:1 modified:1 removed:0
2014-09-16 23:07:23.072 GMT INFO ViInventoryThread EndpointSVMUpdater:206 - Solution 6341068275337691137 is not registered
2014-09-16 23:07:23.096 GMT INFO ViInventoryThread ViManagedVirtualMachineObject:244 - vnic change for vm-556: null to
2014-09-16 23:07:23.301 GMT INFO ViInventoryThread ViInventory:1304 - 85/165 objects published.
2014-09-16 23:07:23.317 GMT INFO ViInventoryThread VimObjectBridge:943 - VimObjectBridge: Ending inventory update
2014-09-16 23:07:23.318 GMT INFO ViInventoryThread VimObjectBridge:222 - Processing 4 updates and 0 deletions for this transaction
2014-09-16 23:07:23.320 GMT INFO ViInventoryThread VimObjectBridge:229 - VimObjectBridge: Time taken to process transaction : 219
2014-09-16 23:07:23.321 GMT INFO ViInventoryThread ViInventory:1512 - Resolved, last version:223 num vc objs:91 num vimos:165
2014-09-16 23:07:24.222 GMT WARN VirtualMachineDvfilterMonitor-1 VirtualMachineWorkQueue$WorkQueue:255 - Host not found for Vm vm-556, bypassing.
2014-09-16 23:07:24.223 GMT WARN VirtualMachineDvfilterMonitor-1 VirtualMachineWorkQueue$WorkQueue:279 - no host found for vm-556, remo ving.
2014-09-16 23:07:24.443 GMT INFO DCNPool-4 InventoryUtils:273 - Null hostId for VM vm-556
2014-09-16 23:07:25.944 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXX73216
2014-09-16 23:07:26.014 GMT INFO http-nio- UserSessionManager:43 - New session: XXXXXXXXXXXXXXXXXXXXXXXXXXXB56C3
2014-09-16 23:07:26.807 GMT ERROR taskExecutor-18 ErrorCounter:56 - <AST>:0:0: unexpected end of subtree
2014-09-16 23:07:26.810 GMT WARN taskExecutor-18 AbstractActionEventListener:61 - User Identity Action Event Listener: Error happened w hen dispatch action events.

Cool detailed info, but unfortunately not shedding any additional light.  As shown in bold, the story stays the same.  It doesn’t appear that there were any errors leading up to the terminal condition, and according to log NSX now deals with template configuration mismatches elegantly (rescaled the template from 4 vCPUs to 1 to match the host limit).  Of course logs aren’t always exactly correct, right?  As it turns out, the template was attempting to create a 4 vCPU VM on a 1 CPU host.  Luckily with nested adding CPUs is very easy.  A quick reconfig of the ESXi guest VMs and a reboot and the controller configuration completed without a hitch:

Screenshot 2014-09-17 01.29.34

Huzzah!  Well that’s (more than) enough for this entry!  Next up will take a deeper look at NSX implementation up the stack, edge device capabilities, and talk through some use cases.  Stay tuned!

I’ve covered overlay networks and their importance a few times in these pages over the years, but I have to admit that until this week I was never “walking the walk” at the ComplaintsHQ lab.  To set a baseline, NSX comes in two flavors:

  • The first is NSX multi-hypervisor, the old Nicira Open vSwitch, which can be integrated with both vSphere and other competing hypervisors (KVM, Xen).  The catch is that this really is a vSphere play and not so much a vCenter play.  The integration of Open vSwitch replaces the vDS and so must integrate directly with a hosts vSS.  If you already have a vDS infrastructure in place, this requires some significant rearchitecting.
  • The second is NSX-V, or the native VMware flavor of NSX, which is quickly evolving to be the defacto network architecture for VMware and is core to the SDN strategy.  As an example of this, in upcoming versions of VCNS (vCloud Networking and Security), the NSX virtual firewall/router edge device is replacing the old vShield Edge.  With NSX-V, the NSX SDN capabilities integrate directly with the vDS.

In my OpenStack entry I touched on the plans I had for introducing OpenStack into the lab.  Unfortunately, the realities of NSX integration complicates things and have delayed those plans.  Before we move forward I think it is worthwhile to call these out:

  • A mixed hypervisor (vSphere + other) OpenStack environment will require NSX-MH if you want to take advantage of advanced OpenStack SDN constructs (Neutron)
  • If you do not go that path, you need to fall back to static Nova network models.  These map pretty closely to vCloud Director “port group assignment” org networks.  So you have to configure a bunch of VLANs up front and map them to port groups which are then utilized by the OpenStack controller at the compute deployment layer (Nova).
  • VXLAN requires vDS, NSX-MH can’t integrate with vDS, but I Open vSwitch can integrate with VXLAN.  Confused?  Don’t feel bad.  Overlay networking can get confusing fast.  The net out here is that in a vCenter environment, to take advantage of both Neutron and VXLAN, you need essentially parallel networking setups.  NSX-MH will be speaking VXLAN, but doing it’s own thing and not part of an existing vDS VXLAN.
  • NSX-V (native VMware) and NSX-MH (multi-hypervisor) cannot co-exist in a vCenter

For lots of reasons I don’t want to break down my HA/DRS clusters.  I could have potentially played with OpenStack and NSX-MH exclusively in my entirely nested vCenter 2 environment, but the purpose of that one is really SRM so it would complicate things.  I still may go ahead and create a third nested vCenter environment and play with OpenStack and NSX-MH there, but that will have to wait.  For now I decided to move forward with NSX-V and shelf the OpenStack testing.

So back to the implementation detail… NSX is a fairly complex technology with some dependencies that never quite fit my old white box lab setup.  For example you’ll need to have a vDS which means you’ll need to have a cluster and multiple NICs in each host.  This means you’ll need either a pretty complex white box build, or a really good nested setup.  I never quite had the former as I really was focused on building to a rock bottom budget, but these days I am running the latter so the time was right.

NSX has a few core components to be aware of:

  • NSX Manager: The NSX management plane is built by the NSX manager. The NSX manager provides the single point of
    configuration and the REST API entry-points in a vSphere environment for NSX.
  • NSX Controller: The NSX control plane runs in the NSX controller. In a vSphere optimized environment with VDS the controller enables multicast free VXLAN, control plane programming of elements such as VDR. In a multi- hypervisor environment the controller nodes program the vSwitch forwarding plane. In all cases the controller is purely a part of the control plane and does not have any data plane traffic passing through it. The controller nodes are also deployed in a cluster of odd members in order to enable
    high-availability and scale. Any failure of the controller nodes does not impact any data plane traffic.
  • NSX Edge:  NSX Edge offers L2, L3, perimeter firewall, load-balancing and other services such as SSL VPN, DHCP, etc.
  • Hypervisor Integration: The NSX Data plane consists of the NSX vSwitch. The vSwitch in NSX for vSphere is based on the vSphere Distributed Switch (VDS) (or Open vSwitch for non-ESXi hypervisors) with additional components to enable rich services. The add-on NSX components include kernel modules (VIBs) which run within the hypervisor kernel providing services such as distributed routing, distributed firewall and enable VXLAN bridging capabilities.

As you might imagine from the above, the first step in getting started with implementation is to deploy the NSX Manager.  Luckily, as is frequently the case lately, VMware has packaged this as a click through OVA.  Download the OVA and start the OVF Template deployment wizard from the web client as always:

Screenshot 2014-09-01 18.26.50

The NSX Manager OVF package detail…

Screenshot 2014-09-01 18.26.54

Agree if you’re ready to do this:

Screenshot 2014-09-01 18.27.04

Select a deployment location for the VM:

Screenshot 2014-09-01 18.27.11

Select a storage destination for the VM:

Screenshot 2014-09-01 18.27.17

Connect the NSX Manager to a network (admin network generally since this is a management plane component):

Screenshot 2014-09-01 18.28.11

Provide configuration for the appliance – passwords, hostname and IP info for the appliance:

Screenshot 2014-09-01 18.28.52 Screenshot 2014-09-01 18.32.54

Finish off the configuration and the NSX Manager VM will deploy:

Screenshot 2014-09-01 18.35.44

Very easy https connection to the appliance IP and you will see the VAMI login:

Screenshot 2014-09-01 19.47.34

Simple and clean UI.  You can grab the tech support logs here, view hte configuration summary, manage and update the network configuration, upgrade the appliance and, the two most important at this stage, integrate the appliance with vCenter and back it up:

Screenshot 2014-09-01 19.47.39


vCenter registration is very straightforward.  Enter the vCenter address and login info, as well as the lookup service.  Configured vCenter registration provided below as reference:

Screenshot 2014-09-16 14.17.51

With this part complete the NSX Manager appliance is configured so you should go ahead and back it up just to be safe.  After this we can head into the web client where we will now see the NSX management solution – Networking & Security.  Clicking on that icon will bring us to the next stage of the configuration, but more on that next entry!

Screenshot 2014-09-16 11.16.36