Archive for : January, 2020

NSX-T Command Line Cheat Sheet

Having recently passed my NSX-T VCP, I thought I’d share the command line tools I used to practice in a lab to get to grips with NSX-T installation, configuration, managing and troubleshooting. 

I know I will be revisiting this site for reference in future.  I hope you all enjoy it!

I will cover a walkthrough of some of these commands in another post if it’s new to you and link it here.

NSX Manager

Setup Troubleshooting

Description

Commands 

Query Management Cluster Status  get cluster status [verbose]
View the cluster configuration and which node is running which cluster component get cluster config
Detach a NSX Manager node from Cluster:  detach node <node-id>
Retrieve NSX Manager Certificate Thumbprint  get certificate api thumbprint 
Join NSX Manager to Cluster  join NSX-Manager-ip-address cluster-id <cluster-id> thumbprint <thumbprint> username <username> password <password>
Disable CLI timeout  set cli-timeout 0 
List the transport nodes registered with NSX Manager  get nodes 
Query the Managers Connection Status get managers
List all ESXi hosts to get the transport Node UUIDS get transport-nodes status
List the transport Node Status get transport-node <uuid> status
List the Transport Node vtep information get transport-node <uuid> vtep
Lists the VIF UUID of a VM connected to Segment on a Transport Node get transport-node <uuid> vifs
Set logging level on NSX Manager set service manager logging-level debug
Configure a remote syslog server set logging-server <hostname-or-ip-address[:port]> proto <protocol> level <level>
Read the Policy Manager log get log-file policy.log <follow>
Read the syslog log  get log-file syslog <follow>
Setup packet capture start capture interface <interface-name> [file <filename>] [count <packet-count>] [expression <expression>]
Enter root privileged Mode st en

 

Logical Switching

List all the Logical Switches get logical-switches
List all the switch ports connected to the Segment get logical-switch <uuid> ports
List information about a Segment get logical-switch <vni-or-uuid>
List the ARP table of a Logical Switch get logical-switch <vni-or-uuid> arp-table
List the MAC table of a Logical Switch get logical-switch <vni-or-uuid> mac-table
List the statistics of a Logical Switch get logical-switch <vni-or-uuid> stats
List the Transport Node table of a Segment get logical-switch <vni-or-uuid> transport-node-table
List the VTEP table of a Segment get logical-switch <vni-or-uuid> vtep
View the Logical Switch Port information get logical-switch-port <uuid>
List the logical Switches statistics get logical-switches stats

 

Logical Routing

View the list of logical routers get logical-router
View the information about a logical router get logical-router <uuid>
View the list of logical router interfaces get logical-router <uuid> interfaces
View the logical router interface information get logical-router <uuid> interface <interface-id>
View the Routers on a logical router get logical-router <uuid> route
List the NSX Edge nodes registered with NSX Manager and their associated controller get transport-node status

 

Firewall

View the Rule count of L2, L3 Firewall Rules get firewall summary
List of firewall entities in the excluded-list get firewall exclude-list
Firewall Section that is not created or deleted completely from the system get firewall orphaned-section
Firewall rules published to CCP get firewall published-entity
Firewall Status get firewall status

 

User Account Administration

Change local user password  Set user <username> [password <password> [old-password <old-password>]]
Password length Set auth-policy minimum-password-length <password-length>
UI and API authentication policies Set auth-policy api lockout-period <lockout-period>
Set auth-policy api lockout-reset-period <lockout-reset-period>
Set auth-policy api max-auth-failures <auth-failures>
Set CLI authentication policy Set auth-policy cli lockout-period lockout-period <lockout-period>

 

ESXi

Setup Troubleshooting

Description

Commands

List the VIBs loaded on ESXi esxcli software vib list | grep -e nsx -e vsip
List all the NSX-T modules currently loaded in the system esxcli system module list | grep nsx
Check the User world agents (UWA):   
nsx-mpa /etc/init.d/nsx-mpa status | start | stop | restart
nsx-proxy /etc/init.d/nsx-proxy status | start | stop | restart
nsx-opsagent /etc/init.d/nsx-opsagent status | start | stop | restart
nsxa /etc/init.d/nsxastatus | start | stop | restart
Check UWAs Connection: 
Port 1235 to Controllers  esxcli network ip connection list | grep 1235
Port 5671 to NSX Manager esxcli network ip connection list | grep 5671
List Physical NICs/vmnic esxcli network nic list
Physical NIC details esxcli network nic get -n vmnic3
List vmk NICs with IP addresses/MAC/MTU and so on (vmk10 is TEP, vmk50 is containers) esxcli network ip interface ipv4 get
Details of each vmk NIC, including vDS information esxcli network ip interface list
Details of netstack IP Stack created on ESXi esxcli network ipinterface list --netstack=vxlan
Ping from a VXLAN TCP/IP Stack vmkping ++netstack=vxlan <host-IP> -s <packet-size>
View routing table of VXLAN-dedicated TCP/IP stack esxcli network ip route ipv4 list -N vxlan
View ARP table of VXLAN dedicated TCP/IP stack esxcli network ip neighbor list -N vxlan
Setup syslog esxcli network firewall ruleset set -r syslog -e true
esxcli system syslog config set --loghost=<hostname-or-ip-address[:port]>
esxcli system syslog reload

 

Logical Switching

View all the logical switches get logical-switches
View the Logical Switch information from ESXi host get logical-switch <logical-switch-id>
View the ARP table of a logical switch get logical-switch <logical-switch-id> arp-table
View the MAC table of a logical switch get logical-switch <logical-switch-id> mac-table
View the Neighbor Discovery (ND) table of a logical switch get logical-switch <logical-switch-id> nd-table
View the VTEP table of a logical Switch get logical-switch <logical-switch-id> vtep-table
View the logical switch port status get logical-switch-port status
View the MAC, ARP, VTEP tables from local or remote host using VNI get logical-switch [local | remote] [mac-cache | arp-cache | vtep-cache] <vni>
Verify the Transport Node Tunnel Status get host-switch <host-switch-name> tunnels
 To view the Switch port ID from root mode net-stats -l
To view the Switches configured on ESXi esxcfg-vswitch -l
Performance monitoring tool on ESXi esxtop
View the VTEP and VNI Configuration net-vdl2 -l
To view the N-VDS Uplink Configuration net-vdr -C -l
View the Logical Routers from ESXi net-vdr -I -l
Verify VXLAN kernel module vdl2 is loaded esxcli system module get -m nsxt-vdl2
Setup packet capture pktcap-uw [-o <filename.pcap>]
View captured packets tcpdump-uw
Capture packets and display live output on screen (-dir 1 is outgoing traffic, -dir 0 is incoming traffic) pktcap-uw --switchport <VM-Switch-Port-Number> [--dir <1 | 2>] | tcpdump-uw
Find VM Switch Port Number esxtop <n>
Packet Capture Target Options

 

PreDVFilter
PostDVFilter

 

Source VM Switch Port
Leaving the VNI Port
Leaving vdrPort

 

Outgoing encapsulated overlay traffic
Incoming encapsulated

 

Arriving at destination VM Switch Port

 

PostDVFilter
PreDVFilter

 

 

 

pktcap-uw --capture PreDVFilter --dvfilter <dvfilter-Name>--capture PostDVFilter --dvfilter <dvFilter-Name>--switchport <VM-Switchport-Number> --dir 0--switchport <VM-Switch-Port-Number> --dir 1 --vni=<Switch-VNI-Number>--switchport <VM-Switch-Port> --dir 0--uplink <vmnic#> --dir 1 --overlay geneve--uplink <vmnic#> --dir 0 --overlay geneve--switchport <VM-Switchport-Number>  --dir 0--capture PostDVFilter --dvfilter <dvFilter-Name>--capture PreDVFilter --dvfilter <dvfilter-Name>

 

Logical Routing

 View the Logical Router Forwarding table get logical-router <UUID> forwarding
View the Logical Router Interface get logical-router <UUID>interface
View the Logical Router Interfaces get logical-router <UUID>interfaces
View the logical Router Neighbour get logical-router <UUID> neighbor 
View the logical Router Neighbours get logical-router <UUID> neighbors

 

Firewall

 View the Firewall Status get firewall status
View the Firewall Rules applied at the VIF with Address Sets get firewall <vif_uuid> addrsets
  get firewall <vif_uuid> profile
List all the VMs dvFilter Names
List all the VMs dvFilter Names associated with a VM and limit the response to 16 lines
summarize-dvfilter
Summarize-dvfilter | grep -A16 <SERVER_NAME>
List the Firewall Rules applied on DvFilter vsipioctl getrules -f <filtername>
View the Firewall Configuration for a given dvFilter name vsipioctl getfwconfig -f <dvfilter-name>
View the DVSPort ID and MAC address associated with a VM nsxdp-cli  -c get ports | egrep -A1 "<VM_NAME>|MAC"
View the number of packets dropped on a host switch nsxdp-cli swsec get stats -dvs <Overlay_N-VDS> --dport <DVSPort-UUID>

 

KVM

Setup Troubleshooting

Description

Commands

List the VIBs loaded on Ubuntu KVM sudo  dpkg --list | grep nsx
List the VIBs loaded on RedHat KVM rpm -qa | grep nsx
Check the User world agents (UWA):   
nsx-mpa /etc/init.d/nsx-mpa status | start | stop | restart
nsx-proxy /etc/init.d/nsx-proxy status | start | stop | restart
nsx-opsagent /etc/init.d/nsx-opsagent status | start | stop | restart
nsxa /etc/init.d/nsxa status | start | stop | restart
Check UWAs Connection:   
Port 1235 to Controllers lsof-i -P -n | grep 1235 [or netstat -an | grep 1235]
Port 5671 to NSX Manager lsof-i -P -n | grep 5671 [or netstat -an | grep 5671]
NIC Details ifconfig [-a]
lspci
Setup syslog Login as root
Create this file
/etc/rsyslog.d/40-vmware-remote-logging.conf
Add this line to the file
'*.*@<syslog_server_ip>:514;RFC5424fmt'
Restart syslog
Systemctl restart rsyslog

 

Logical Switching

Query the NSX Controllers get controllers
View all the logical switches get logical-switches
View the Logical Switch information from ESXi host get logical-switch <logical-switch-id>
View the ARP table of a logical switch get logical-switch <logical-switch-id> arp-table
View the MAC table of a logical switch get logical-switch <logical-switch-id> mac-table
View the VTEP table of a logical Switch get logical-switch <logical-switch-id> vtep-table
View the ports on a logical Switch get logical-switch <logical-switch-id> ports
View the MAC, ARP, VTEP tables from local or remote host using VNI. get logical-switch [local | remote] [mac-cache | arp-cache | vtep-cache] <vni>
Verify the OpenvSwitch Kernel Module lsmod | grep openvswitch (from root mode)
Open vSwitch Configuration File /etc/openvswitch/conf.db
Print the current version of openvswitch ovs-vsctl –V
Prints a brief overview of the switch database configuration ovs-vsctl show
Prints a list of configured bridges ovs-vsctl list-br
Prints a list of ports on a specific bridge ovs-vsctllist-ports <bridge>
OVSDB Log ovsdb-tool show-log

 

Logical Routing

View the Logical Router Forwarding table get logical-router <UUID> forwarding
View the Logical Router Interface get logical-router <UUID> interface
View the Logical Router Interfaces get logical-router <UUID> interfaces
View the logical Router Neighbor get logical-router <UUID> neighbor 
View the logical Router Neighbors get logical-router <UUID> neighbors

 

Firewall

View app firewall virtual interfaces ovs-appctl -t /var/run/openvswitch/nsxa-ctl dfw/vif
View firewall rules with containing addrsets  ovs-appctl -t /var/run/openvswitch/nsxa-ctl dfw/rules <VIF_ID_NUMBER>

 

Edges

Setup Troubleshooting

Description

Commands

Verify SSH service status get service ssh
Start the SSH service start service ssh
Set the SSH service to autostart when the VM is powered on set service ssh start-on-boot
Verify that the SSH service is running and Start on boot is set to True get service ssh
View the Edge configuration get configuration
Display the Edge node UUID get node-uuid
View the Edge interfaces get interfaces
Query the connection to the NSX Managers get managers
View all the host switches information get host-switches
View the tunnel port information get tunnel-ports
Setup packet capture set capture session <session-number> interface <port-uuid> direction <direction>
Remove captured session information del capture session 1
Enter root privileged Mode st en

 

Logical Routing

View the VTEPs get vteps
View the logical routers get logical-routers
View the logical router information get logical-router <uuid>
View the logical router statistics get logical-routers stats
View the logical router interfaces get logical-router <uuid> interfaces
View the logical router neighbour get logical-router <uuid> neighbor
View the logical router interfaces statistics get logical-router interfaces stats
View the logical router bgp neighbour get logical-router
To enter into the VRF construct vrf <VRF-ID-of-Tier-0-SR>
View the bgp neighbor of a Tier-0 SR (tier0_sr)> get bgp neighbor
(tier0>sr)> get bgp neighbor summary
View the interfaces on a Tier-0 SR (tier0_sr)> get interfaces
View the forwarding table (tier0_sr)> get forwarding
View the Routers (tier0_sr)> get route
View the BFG configuration (tier0_sr)> get bfd-config
View BGP IPv4 route info (tier0_sr)> get bgp ipv4
View the Tier-1 or Tier-0 distributed router routing information (tier[0 | 1]_dr)> get forwarding
   

Firewall

 View the list of Firewall Interfaces get firewall interfaces
View the Firewall Ruleset and Rules get firewall <interface_id> ruleset rule

 

Load Balancer

Display load balancers configuration get load-balancers
Display load balancer pool confi get load-balancer <lb-uuid> pools
Display load balancer virtual servers configuration get load-balancer <lb-uuid> virtual-servers
Display specific load balancer virtual servers info get load-balancer <lb-uuid> virtual-server  Virtual_Server_ID
Display load balancer status  get load-balancer <lb-uuid> status
Display load balancer virtual servers stats  get load-balancer <lb-uuid> virtual-servers stats
Display load balancer stats  get load-balancer <lb-uuid> stats

 

DHCP

Retrieve the DHCP server information get dhcp servers
View the configured DHCP pools get dhcp ip-pools
List the leased IP addresses get dhcp leases

 

VPN

Verify L2VPN session is active, identify the peers, and ensure that the tunnel status is up get ipsecvpn session active
Verify that the sessions are up get ipsecvpn session status

Check whether the ipsecvpn session is up between the local and remote peer
get ipsecvpn session summary
Get the l2vpn session, tunnel, and IPSEC session numbers, and check that the status is UI get l2vpn sessions
Get statistical information of the local and remote peers, whether the status is UP, count of packets received, bytes received (RX), packets transmitted (TX), and packets dropped, malformed, or loops get l2vpn session stats
Get the session configuration information get l2vpn session config

Property ‘HA index’ must be configured for the VM to power on

When deploying the NSX-T L2 VPN Client ‘nsx-l2vpn-client’ .ovf you might get this error message if you don’t configure the client HA configuration.  

To get around this issue, go to the vApp settings of the VM and set the value of 0 on the vApp Options page.

Then it should power on for you without any issues.