As the journey continues we have now come to the portion where we setup the Edge. There are a couple different ways you can setup your edge, IE leverage Port Groups from a Distributed Switch or leverage NSX-T N-VDS switches. Do you deploy two Transport Zones or four? I have elected to deploy the edge on N-VDS to future proof the environment for future SDDC and hybrid cloud initiatives. Finally, this is the portion where your architectural design really comes to life.
Other Articles in my series
https://digitalthoughtdisruption.com/2019/07/01/how-to-install-nsx-t-2-4-part-1-1st-nsx-manager/
https://digitalthoughtdisruption.com/2019/07/03/how-to-install-nsx-t-2-4-part-2-deploy-compute-manger/
https://digitalthoughtdisruption.com/2019/07/08/how-to-install-nsx-t-2-4-part-3-deploying-2nd-3rd-nsx-manager/
https://digitalthoughtdisruption.com/2019/07/11/how-to-install-nsx-t-2-4-part-4-transport-zone-creation-uplink-profile-tep-ip-pool-configure-nsx-t-for-esxi/
https://digitalthoughtdisruption.com/2019/07/15/how-to-install-nsx-t-2-4-part-5-the-war-that-was-setting-up-edge-vms-and-edge-cluster-on-n-vds/
Special Thanks to Harikrishnan, James, and Jourdan. Y’all prove that the VMware community is a real community of guys just trying to help one another even if y’all don’t have anything to benefit from it. Anyone reading this blog and find it helpful, please pass it forward when you can help someone out.
If you wonder how to
do it the vDS way I recommend going to Harikrishnan’s blog:
https://vxplanet.com/2019/05/23/deploying-the-nsx-t-edge-vm-cluster-leveraging-vsphere-dvs-portgroups/
Terminology:
- TEP – Tunnel endpoint, interface spun up with other host (think Point to Point). Each TEP requires an IP address. Each host transport node is a TEP.
- fp stands for Fast-Path
- N-VDS – Previously known as a host switch, created and transported across ESXi & NSX Edge Transport Nodes. Typically, owns a couple or more pNICs. Name must be unique.
- Transport Zone – Can support either overlay or VLAN traffic. Allows you to define a group of transport nodes that can communicate with each other across the physical infrastructure. This communication occurs over TEPs.
- Overlay Transport Zone – used as the internal tunnel between transport nodes. Carries Geneve traffic.
- VLAN transport zones – used to connect between NSX Edge Uplinks and upstream physical routers to establish N-S connectivity.
- Segment – Previously known as a logical switch. A segment contains multiple switch ports. Routers, VMs, and so on can connect to a segment though the segment ports. A segment can belong to only one transport zone.
General Information:
- If you wonder why you need a separate VLAN for Overlay TEP & separate VLAN for Edge TEP in collapsed Edge/Compute is because Geneve encapsulated packet arrives the host does not know if it is for the host itself or for the Edge. That is why you need two VLANs for that.
- We can use same N-VDS for both Overlay and VLAN but we can’t have same N-VDS for two Overlay TZ. I will have a separate N-VDS for both Overlay & VLAN.
- If using N-VDS on edge have to create edge uplink profile and assign VLAN ID
- It is also VMware preferred practice to not vMotion your Edge VMs, furthermore, they would even like to have DRS disabled on the cluster.
- We need two VLAN Logical segments created on the Edge Uplink Transport Zones, so that the Tier 0 Gateway can attach to the Edge Uplinks.
- In my environment, Uplink1-TZ and Uplink2-TZ are on separate VLANs. It can help with peering the edges on different VLANs on the ToR with BGP. It allows for HA on the uplinks as well. It can be done with a single uplink TZ with no issues.
- I’ve sized my Edge VMs as larges. My preference is large, that way if you have a VM failure then your single VM should still operate with hopefully no noticeable impact to performance.
Bug:
- If deploying NSX-T Edge VMs n an N-VDS you cannot use the simplified UI as it is unable to see the segments you will create in NSX-T. This is a major bug and pain that I hope Vmware fixes in future updates because wasting an entire weekend trying to figure out why the Simplified UI isn’t working is no fun.
Good to know notes:
- When deploying NSX Edge VM OVA do it from the HTML interface.
Let’s have the fun begin. Remember, patience is key.
Step 1. – Infrastructure setup
Make sure you have all the Transport Zones you will need to fit your use case. In my environment I have 4 x TZs. Edge (Future VMK traffic after migration), Overlay (VM Traffic), Uplink1 (Edge UP1), Uplink2 (Edge UP2)
Next navigate to Networking>Segments>Segments>Add Segment
Provide a name
Under Uplink & Type: Leave None
Transport Zone: select the transport zone you will be associating this segment with. Remember, I am doing an N-VDS & will need these to be in my Edge-TZ.
VLAN: Select the associated VLAN IE if Uplink-2 is on VLAN 200 then put 200.
Should see something similar in your environment. Make sure the status is Up.
Step 2. – Deploy Edge VMs
Deploy OVF Template
Select the ova
Give your NSX Edge VM a name that you have already entered into DNS
Select the cluster you want to deploy your Edge VM. Since I have a collapsed Edge/Mgmt cluster it is going into my Mgmt cluster.
The good old review details page
Select the size of your VM. I prefer Large, that way if you lose an Edge VM then my environment “should” still work without a performance impact.
Select the datastore you wish you place your NSX Edge VM. This is one place vSAN makes life easier.
Network 3 is associated with Segment created for uplink 2
Network 2 is associated with Segment created for uplink 1
Network 1 is associated with Segment created for Edge TEP
Network 0 is associated with Segment/PG for your management
Time to enter your Root password
Now enter your Admin & Audit Password
Next enter the name of your NSX Edge VM again!
Default Gateway
IP address for your Edge VM
Netmask for your Edge VM
Enter your DNS servers
Enter your search list
NTP server
Enable SSH
Allow Root SSH Logins
Review
Finish
**Deploying the VM takes about 5-10 minutes so please be patient.
***Once it is deployed power on the VM and wait for it to fully boot up
****Side note, if the OVA doesn’t deploy and you get a generic error. Re-deploy again.
*******May take 4 times!!! Just an FYI 🙂
Step 3. – Registering your Edge VM with NSX-T
Log into the Edge VM via putty
Login as Admin
Verify the correct IP information
#get interface eth0
Network connectivity verification:
Ping verification tests
Ping the default gateway
Ping the host the VM is on
Ping the NSX Manager
Ping DNS
Ping vCenter
Get NSX thumbprint
SSH into one of your NSX Managers as Admin
#get certificate cluster thumbprint
Apply thumbprint
Go back to your Edge VM and issue the following command
#join management-plane <VIP IP> username admin thumbprint <Thumbprint from NSX Manager> password <password>
Configure Edge Transport Nodes
Click on Configure NSX on your Edge VM
Add in all the necessary TZ for your edge/use case
Edge Switch Name: I start with the Overylay-TZ
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
IP assignment: Use IP Pool
IP Pool: choose your VLAN/Edge TEP Pool
Virtual NICs: Uplink-1, fp-eth0
Add N-VDS
Edge Switch Name: Choose your NVDS, in my case it is to my first Uplink1 switch
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
Virtual NICs: uplink-1, fp-eth1
If you have another TZ and N-VDS then click add again and repeat
Edge Switch Name: Choose your NVDS, in my case it is to my first Uplink2 switch
Uplink profile: I choose default nsx-edge-single-nic-uplink-profile
Virtual NICs: uplink-1, fp-eth2
Step – rinse and repeat
Repeat in order to deploy your second edge VM.
Step 4. – Form Edge Cluster
System>Fabric>Nodes>edge Cluster>Add
Give your Edge Cluster a Name
Check your two Edge VMs and move them over
Add
Should see something similar.
Summary:
This process (If doing N-VDS) is very complicated with a lot of moving parts. Plus, it was extremely time consuming with many hours of troubleshooting only to discover you cannot deploy Edge VM on N-VDS via Simplified UI. I discovered several bugs and issues that I have worked around. It is my hope that y’all don’t have to spend as many hours deploying your edges as it took me. Following the process above “should” make this part seamless. Next article will be on deploying the Tier-0 Gateway.