How to Install VMware NSX-T (Part 5) – setting up Edge VMs and Edge Cluster on N-VDS

As the journey continues we have now come to the portion where we setup the Edge.  There are a couple different ways you can setup your edge, IE leverage Port Groups from a Distributed Switch or leverage NSX-T N-VDS switches.  Do you deploy two Transport Zones or four? I have elected to deploy the edge on N-VDS to future proof the environment for future SDDC and hybrid cloud initiatives.  Finally, this is the portion where your architectural design really comes to life.

Other Articles in my series
https://digitalthoughtdisruption.com/2019/07/01/how-to-install-nsx-t-2-4-part-1-1st-nsx-manager/
https://digitalthoughtdisruption.com/2019/07/03/how-to-install-nsx-t-2-4-part-2-deploy-compute-manger/
https://digitalthoughtdisruption.com/2019/07/08/how-to-install-nsx-t-2-4-part-3-deploying-2nd-3rd-nsx-manager/
https://digitalthoughtdisruption.com/2019/07/11/how-to-install-nsx-t-2-4-part-4-transport-zone-creation-uplink-profile-tep-ip-pool-configure-nsx-t-for-esxi/
https://digitalthoughtdisruption.com/2019/07/15/how-to-install-nsx-t-2-4-part-5-the-war-that-was-setting-up-edge-vms-and-edge-cluster-on-n-vds/

Special Thanks to Harikrishnan, James, and Jourdan.  Y’all prove that the VMware community is a real community of guys just trying to help one another even if y’all don’t have anything to benefit from it.  Anyone reading this blog and find it helpful, please pass it forward when you can help someone out.

If you wonder how to do it the vDS way I recommend going to Harikrishnan’s blog:
https://vxplanet.com/2019/05/23/deploying-the-nsx-t-edge-vm-cluster-leveraging-vsphere-dvs-portgroups/

Terminology:

• TEP – Tunnel endpoint, interface spun up with other host (think Point to Point).  Each TEP requires an IP address.  Each host transport node is a TEP. 
• fp stands for Fast-Path
• N-VDS – Previously known as a host switch, created and transported across ESXi & NSX Edge Transport Nodes.  Typically, owns a couple or more pNICs.  Name must be unique.
• Transport Zone – Can support either overlay or VLAN traffic.  Allows you to define a group of transport nodes that can communicate with each other across the physical infrastructure.  This communication occurs over TEPs.
• Overlay Transport Zone – used as the internal tunnel between transport nodes.  Carries Geneve traffic.
• VLAN transport zones – used to connect between NSX Edge Uplinks and upstream physical routers to establish N-S connectivity.
• Segment – Previously known as a logical switch.  A segment contains multiple switch ports. Routers, VMs, and so on can connect to a segment though the segment ports.  A segment can belong to only one transport zone.

General Information:

• If you wonder why you need a separate VLAN for Overlay TEP & separate VLAN for Edge TEP in collapsed Edge/Compute is because Geneve encapsulated packet arrives the host does not know if it is for the host itself or for the Edge. That is why you need two VLANs for that.
• We can use same N-VDS for both Overlay and VLAN but we can’t have same N-VDS for two Overlay TZ.  I will have a separate N-VDS for both Overlay & VLAN.
• If using N-VDS on edge have to create edge uplink profile and assign VLAN ID
• It is also VMware preferred practice to not vMotion your Edge VMs, furthermore, they would even like to have DRS disabled on the cluster.
• We need two VLAN Logical segments created on the Edge Uplink Transport Zones, so that the Tier 0 Gateway can attach to the Edge Uplinks.
• In my environment, Uplink1-TZ and Uplink2-TZ are on separate VLANs. It can help with peering the edges on different VLANs on the ToR with BGP.  It allows for HA on the uplinks as well.  It can be done with a single uplink TZ with no issues.
• I’ve sized my Edge VMs as larges.  My preference is large, that way if you have a VM failure then your single VM should still operate with hopefully no noticeable impact to performance.

Bug:

• If deploying NSX-T Edge VMs n an N-VDS you cannot use the simplified UI as it is unable to see the segments you will create in NSX-T. This is a major bug and pain that I hope Vmware fixes in future updates because wasting an entire weekend trying to figure out why the Simplified UI isn’t working is no fun.

Good to know notes:

• When deploying NSX Edge VM OVA do it from the HTML interface. 

Let’s have the fun begin.  Remember, patience is key.
Step 1. – Infrastructure setup

Make sure you have all the Transport Zones you will need to fit your use case.  In my environment I have 4 x TZs.  Edge (Future VMK traffic after migration), Overlay (VM Traffic), Uplink1 (Edge UP1), Uplink2 (Edge UP2)

Next navigate to Networking>Segments>Segments>Add Segment
Provide a name
Under Uplink & Type: Leave None
Transport Zone: select the transport zone you will be associating this segment with.  Remember, I am doing an N-VDS & will need these to be in my Edge-TZ.
VLAN: Select the associated VLAN IE if Uplink-2 is on VLAN 200 then put 200.

Should see something similar in your environment.  Make sure the status is Up.

Step 2. – Deploy Edge VMs

Deploy OVF Template

Select the ova

Give your NSX Edge VM a name that you have already entered into DNS

Select the cluster you want to deploy your Edge VM.  Since I have a collapsed Edge/Mgmt cluster it is going into my Mgmt cluster.

The good old review details page

Select the size of your VM.  I prefer Large, that way if you lose an Edge VM then my environment “should” still work without a performance impact.

Select the datastore you wish you place your NSX Edge VM.  This is one place vSAN makes life easier.

Network 3 is associated with Segment created for uplink 2
Network 2 is associated with Segment created for uplink 1
Network 1 is associated with Segment created for Edge TEP
Network 0 is associated with Segment/PG for your management

Time to enter your Root password

Now enter your Admin & Audit Password

Next enter the name of your NSX Edge VM again!
Default Gateway
IP address for your Edge VM
Netmask for your Edge VM

Enter your DNS servers
Enter your search list
NTP server
Enable SSH
Allow Root SSH Logins

Review
Finish
**Deploying the VM takes about 5-10 minutes so please be patient. 
***Once it is deployed power on the VM and wait for it to fully boot up
****Side note, if the OVA doesn’t deploy and you get a generic error.  Re-deploy again.
*******May take 4 times!!! Just an FYI 🙂

Step 3. – Registering your Edge VM with NSX-T
Log into the Edge VM via putty
Login as Admin

Verify the correct IP information
#get interface eth0

Network connectivity verification:
Ping verification tests
Ping the default gateway
Ping the host the VM is on
Ping the NSX Manager
Ping DNS
Ping vCenter
Get NSX thumbprint

SSH into one of your NSX Managers as Admin

#get certificate cluster thumbprint

Apply thumbprint
Go back to your Edge VM and issue the following command

#join management-plane <VIP IP> username admin thumbprint <Thumbprint from NSX Manager> password <password>

Configure Edge Transport Nodes

Click on Configure NSX on your Edge VM

Add in all the necessary TZ for your edge/use case

Edge Switch Name: I start with the Overylay-TZ
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
IP assignment: Use IP Pool
IP Pool: choose your VLAN/Edge TEP Pool
Virtual NICs: Uplink-1, fp-eth0

Add N-VDS

Edge Switch Name: Choose your NVDS, in my case it is to my first Uplink1 switch
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
Virtual NICs: uplink-1, fp-eth1

If you have another TZ and N-VDS then click add again and repeat
Edge Switch Name: Choose your NVDS, in my case it is to my first Uplink2 switch
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
Virtual NICs: uplink-1, fp-eth2

Step – rinse and repeat

Repeat in order to deploy your second edge VM.

Step 4. – Form Edge Cluster

System>Fabric>Nodes>edge Cluster>Add

Give your Edge Cluster a Name
Check your two Edge VMs and move them over
Add

Should see something similar.

Summary:
This process (If doing N-VDS) is very complicated with a lot of moving parts.  Plus, it was extremely time consuming with many hours of troubleshooting only to discover you cannot deploy Edge VM on N-VDS via Simplified UI. I discovered several bugs and issues that I have worked around.  It is my hope that y’all don’t have to spend as many hours deploying your edges as it took me. Following the process above “should” make this part seamless.  Next article will be on deploying the Tier-0 Gateway.