ESXi White Box Adventure – Part VII, Supplemental – the network sidebar


Before getting into the HyperV config process, I decided it was time to take a break from servers and overhaul the network.  My old setup was pretty straightforward.  I have 10Mbps cable modem service coming into the basement where it feeds a Motorola DOCSIS router and a VOIP router.  Off the Moto, I have a D-Link Dir 655.  Off the Dir 655, the house is wired with CAT 6 Ethernet throughout and I use D-Link DGS2208 gigabit switches to distribute the network.  The are switches in the basement wiring closet, the basement play area for the TiVO and game consoles, the family room for the TiVO HD and game consoles, the bedroom for the Roku and the study/office for the main lab setup including the print server, main PC and servers.

For this upgrade, I decided that playing with 802.1Q was a good idea since VLAN segmentation is intrinsic to the isolation aspects of a virtualization solution (isolating storage from management from production).  The DGS2208s are great switches, but they are fully unmanaged “dumb” elements.  In addition, the DIR655 isn’t hackable and is my main wireless access point so I avoid reconfiguring it.

As covered in earlier entries, I decided to introduce a smart switch and went with the Cisco SLM2008 which is a workgroup gigabit switch that supports 802.1Q and is, in my opinion, a really good value.  I decided I would put the servers on the VLAN enabled switch along with the secondary interfaces of both the ReadyNAS Ultra 6 (for iSCSI) and the main PC (for management).  The ReadyNAS is clueless about VLANs but it doesn’t matter since we will be building physically discrete VLAN groups anyhow and not trunking, so the ReadyNAS can stay untagged.

As a last step, I decided that I wanted to have permanent routing between the two network segments (the new server sandbox segment and the rest of the house).  After doing a bit of research I settled on the Netgear WNR3500L which is a low cost ($69 on sale at Best Buy) WAP with 4 port gigabit switch.  It’s main feature is that it has 64MB RAM and 8MB flash and is on the dd-wrt HCL!

For anyone not familiar with dd-wrt, you are in for a treat!  I remember well when this project first started and it is just incredible how far they have come.  In a nutshell, dd-wrt is an open source router firmware project that gives you the ability to flash a variety of supported router/WAPs (and it all started with the old Linksys WRT models that were the first to allow firmware replacement)  into full featured routers.  Here is a collage of config screens from dd-wrt:

Installation was very easy and the procedure is clearly outlined at the dd-wrt site.  Just study the Wiki closely.  First the Netgear had to be prepped with a mini firmware chk file to get it ready to take a foreign full firmware flash.  After that it was just a matter of selecting a supported image for the device.  All very easy from the support pages.  I went with a “big” install as I wanted to experience the full feature set and the Netgear does not support the “mega”.

Once I took a few haphazard runs at the thing which left me frustrated, nonfunctional and swearing, I took a step back and did some genuine network planning (this is my usual approach to any problem – attack, fail, then plan, succeed 🙂 ).  It was funny to be doing Visio work for a home LAN, but here is the config as it ended up:

I have been playing with HyperV a bit, but that will have to wait (again) until the next segment.  This will be the last sidebar though as I have a lot to say about the free HyperV edition.  Some good, some bad, all interesting.  Stay tuned!

EDIT: Addendum… Getting dd-wrt setup as a pure ethernet router isn’t immediately intuitive so I thought it might be worth documenting here.  There is some great documentation on dd-wrt, but it takes a lot to sift through it and it is primarily geared towards wireless.  You need to piece your way to making a dd-wrt a proper wired router.  One particularly great doc I found on there was this internal architecture overview:

I cant stress enough how enormously helpful this diagram is since, inside the GUI, there isnt a lot of clarity around the meaning of predefined objects (eth0, eth1, vlan1, vlan2, br0) We can see that basically, the local switch ports form a VLAN and the WAN port forms a VLAN both off of the logical interface eth0. Routing occurs natively in dd-wrt between VLAN1 and VLAN2 (or the WAN and LAN side). In turn, eth0/VLAN1 is bridged to eth1 (the WLAN) forming a bridge group (br0).

Armed with this understanding of the internal architecture of the dd-wrt firmware, I set forth to configure an ethernet router. The main bits are as follows:

  • Set the WAN connection to static IP
  • I cabled the WAN connection to the intranet segment and addressed it accordingly for reasons I will explain
  • IF you are adding a dd-wrt into an existing network, for the purpose of segmenting that network, and an existing WAP will maintain the gateway to the internet, it is going to be necessary, in most cases, to make the dd-wrt the DHCP server and WAP.  The reason for this is that consumer WLAN routers aren’t actual routers.  They are task focused on routing/NAT between WAN and LAN.  If there are multiple  subnets coming in from the LAN side, they are clueless.  So if your WAP is your wireless entry point and your DHCP server, clients on DHCP and wireless will need local static route entries to route to the secondary intranet subnet(s).  If, on the other hand, the dd-wrt is your primary DHCP and wireless access point (and therefore becomes your default gateway for those clients), it is perfectly capable of routing the traffic whereever it needs to go.  It will send foreign traffic to the internet facing WAP, and route intranet traffic out of its own internal facing interface.
  • Under the “Advanced Routing” section of the Setup tab, set Operating Mode to “Router”, set Dynamic Routing to “LAN & WAN” and create a static route for the default gateway:
      • Destination: 0.0.0.0
      • Mask: 0.0.0.0
      • Gateway: intranet addr of public facing WAP
      • Interface: LAN&WAN
  • On the Networking tab, verify that the WAN port is assigned to the desired VLAN (as defined above)
  • Turn off the firewall.  Unless you want to firewall segment your intranet in which case leave it on,but make sure to set the rules correctly or you’ll spend a lot of time troubleshooting traffic that goes nowhere for what seems like no good reason 🙂

Here are some screencaps of the important bits:

EDIT: After taking a deeper look and reading a bit more, I discovered that via telnet/ssh access you can accomplish a lot more with dd-wrt without tripping up the great looking GUI. With complex network element config, command line really is still best. The flexibility you gain in this case is the ability to easily segment the switch into any number of VLANs (as ports allow). All VLANs are off of the virtual ethernet port (logical port 8), and as a result are routed by the Linux routed daemon.

Armed with this knowledge, I actually turned off the WAN port from the GUI, utilized the add WAN port to switch option, and then in the Port Setup area of the Networking configuration tab, unbridged vlan2 (the default WAN vlan). Next I popped into telnet to verify that all was well:

As you can see, the virtual eth0 port, port 8, must be a member of all VLANs in order to enable routing. If you wanted to create a fully isolated segment, you would just remove that port form the VLAN group. This would block a group of ports into an isolated switch which would then require external routing to get out. Pretty neat!

 

EDIT 2: I had a few folks ask me how I got internet access through to the second internal subnet (192.168.2.0).  This one is a pain actually unless you use dd-wrt for both routers.  With the DIR-655, routing tables are supported, but only on the WAN interface.  A while back I had stumbled across this absolutely brilliant hack:

http://lizzi555.dyndns.org/655/StaticRoute.html

Luckily, it is possible to set the LAN interface on a route, but it is just hidden by the form.  This awesome Firefox 3 add-in actually allows you to reveal the full form with all hidden fields displayed and manually specify the LAN interface.  This works absolutely perfectly for any router where the hardware and firmware can do intranet routing, but the GUI is preventing it from being set.  I keep Firefox 3 around just for this!