Nexus Dashboard Networking Deep Dive: Data vs Management Networks & Persistent IPs

Two Networks, Two Purposes

Every Nexus Dashboard node connects to exactly two networks. Getting the design of these right before deployment saves you from having to redeploy the cluster later — changing the data subnet requires a full redeploy.

Management Network (bond1 — mgmt0/mgmt1)

Used for operator and system access:

• Accessing the Nexus Dashboard GUI
• SSH access to the cluster CLI
• DNS and NTP communication
• Firmware upload
• Intersight device connector
• AAA (RADIUS, TACACS, LDAP) traffic
• Multi-cluster connectivity

Data Network (bond0 — fabric0/fabric1)

Used for fabric and cluster communication:

• Inter-node cluster communication (VXLAN-encapsulated)
• Service-to-service communication within the cluster
• Fabric telemetry ingestion from ACI, NX-OS, IOS XR, IOS XE
• Connectivity to APIC and NX-OS controllers
• Persistent IP services (telemetry collectors, POAP/SNMP)

In ND 4.2, the default LAN Device Management Connectivity is set to Data. This means switches and fabrics communicate with ND over the data network. This is the recommended best practice — use management only for UI/API access, DNS, NTP, AAA, and Intersight.

Network Subnet Requirements

These requirements are fixed and cannot be changed after deployment without a full redeploy:

• Management and data networks must be in different subnets
• The data network interface requires a minimum MTU of 1500 (higher MTU supported if needed)
• If external VLAN tags are used on switch ports for data traffic, configure MTU ≥ 1504 bytes
• The management interface rate-limits ICMP to an average of 6 packets/second (burst of 5) — configure monitoring tools accordingly to avoid false positives
• When setting up remote authentication (AAA), the AAA server must not be on the same subnet as the data interface

Sizing recommendation: Use a larger data subnet than strictly required — at minimum a /27. Adding features later may require additional IP addresses, and changing the data subnet means redeploying the cluster.

Internal App and Service Networks

Two additional networks are used internally between ND containers:

These are configured during bootstrap with sensible defaults pre-populated. Key facts:

• Container-to-container traffic is VXLAN-encapsulated using the data interface IPs as source/destination
• App and service network addresses are never exposed outside the ND cluster
• These subnets must not overlap with any external networks you need to reach from ND
• Avoid using 169.254.0.0/16 (the Kubernetes br1 subnet) for these networks

Persistent IP Addresses — What They Are and Why You Need Them

Persistent IPs (also called external service IPs) are IP addresses assigned to ND services that must remain reachable even when the underlying pod or node fails. The word “persistent” means: if the service moves to a different node after a failure, the IP stays the same so switches don’t need reconfiguration.

Services that consume persistent IPs in a LAN deployment:

Minimum persistent IP requirements at bootstrap:

• 1-node cluster: 3 persistent IPs on the data network
• 3-node cluster: 5 persistent IPs on the data network
• Dual-stack (IPv4 + IPv6): Double the above (5 IPv4 + 5 IPv6 for a 3-node cluster)

You must configure the minimum required persistent IPs during the bootstrap wizard. Additional IPs can be added after deployment via Admin > System Settings > General > External Pools.

Layer 2 vs Layer 3 (BGP) Deployment

Layer 2 Mode

All ND nodes share the same management and data subnets. Persistent IPs are advertised within the subnet using gratuitous ARP (IPv4) or neighbor discovery (IPv6). This is the simpler configuration and works for most deployments.

Layer 3 BGP Mode

Nodes are in different Layer 3 networks — each node has a unique data subnet. BGP advertises persistent IPs between the nodes’ networks.

When to use BGP:

• Nodes are deployed across different L3 networks or datacenters
• You need nodes in different subnets for topology reasons
• You’re deploying across campus or WAN links (max RTT between nodes: 50ms)

BGP requirements for L3 mode:

• Must be configured during bootstrap — cannot be added after deployment if nodes are in different subnets
• Persistent IPs must be from a subnet that does not overlap with any node’s data or management subnet
• LAN Device Management Connectivity must be set to Data and cannot be changed

BGP configuration per node during bootstrap:

ASN: <your-asn>           # Can be same for all nodes or unique per node
BGP Peer IP: <peer-ip>    # The upstream router's IP
Peer ASN: <peer-asn>

Round Trip Time Requirements

Exceeding these limits will cause cluster instability or feature degradation.

Key Firewall Ports to Open

If a firewall sits between ND and your network, these ports must be open. Here are the most critical ones for a standard LAN deployment:

Management Network — Inbound to ND:

Data Network — Inbound to ND:

ICMP must be allowed between ND and switches — it’s used during device discovery. Block ICMP and device discovery will fail silently.

URLs Required for Cisco Cloud Connectivity

For Intersight integration and Smart Licensing, the following URLs must be reachable from ND:

For air-gapped or restricted environments, configure a proxy server that permits access to these domains and enter the proxy URL during the bootstrap Configuration page.

VLAN Design: Shared Management, Dedicated Data

This is one of the most practical decisions you can make before deployment — and the one most commonly skipped in lab-turned-production environments. The good news is you don’t need two brand new VLANs.

Management network — sharing with existing infra is fine. The ND management network handles GUI access, SSH, DNS, NTP, AAA, and Intersight — exactly the same things your routers, switches, and other network infrastructure already use. Placing the ND management interfaces on your existing infrastructure management VLAN is perfectly acceptable and common practice.

Data network — dedicate this one. The data network is where you want isolation. This carries inter-node VXLAN clustering traffic, etcd state sync, Kafka telemetry streams, gRPC flows from fabric switches, and persistent IP services. This should be its own dedicated VLAN with nothing else on it.

Here’s why the data network needs its own dedicated VLAN:

Isolate High-Volume Cluster Traffic

ND nodes generate significant inter-node traffic continuously — VXLAN encapsulation on UDP 4789, etcd state synchronization, Kafka telemetry streams, gRPC flows from fabric switches, and more. On a shared VLAN this traffic competes directly with production workloads. On a dedicated VLAN it’s contained, predictable, and easy to police with QoS if needed.

Avoid False Positive Alerts from ICMP Rate Limiting

Nexus Dashboard rate-limits ICMP to an average of 6 packets per second with a burst of 5 — on both the management and data interfaces. This is by design to protect the management plane. On a shared VLAN, your existing monitoring tools will almost certainly exceed this threshold and start generating false positive alerts that the ND nodes are unreachable. A dedicated VLAN lets you configure your monitoring specifically for this limit without affecting other devices.

Reduce Your Attack Surface

Nexus Dashboard has deep visibility into your entire fabric — it talks to APIC, NX-OS controllers, and every monitored switch. If ND shares a VLAN with general workloads and something else on that VLAN is compromised, an attacker has a much shorter path to a system with fabric-wide access. Dedicated VLANs with proper ACLs between them is a straightforward way to reduce that exposure.

Guarantee MTU Without Impacting Other Devices

The ND data network requires a minimum MTU of 1500 bytes, and if you’re using VLAN tags on data interfaces you need 1504 bytes or higher. Enforcing jumbo frames or custom MTU on a shared VLAN affects every device on it — potentially breaking other workloads that weren’t designed for it. A dedicated VLAN lets you set the MTU precisely without side effects.

Make Troubleshooting Dramatically Easier

When something goes wrong with ND — and at some point it will — you want clean packet captures. If ND traffic is mixed in with production on a shared VLAN, captures are noisy and hard to interpret. On a dedicated VLAN you can capture exactly what you need with no filtering gymnastics.

Port Configuration

If you’re connecting ND nodes directly to leaf switches, ports must be configured as trunk to carry the VLAN tags. If you’re connecting through an upstream L3 network (recommended for most deployments), configure the host ports in access mode for each respective VLAN — one for management, one for data.

Recommended VLAN Plan

One new VLAN is all you need. Reuse your existing infrastructure management VLAN for the management network, and create a single new dedicated VLAN for the ND data network. Document both VLAN IDs before starting the OVA deployment — you’ll need them when mapping networks in the vCenter or ESXi deployment wizard.