When Your First HCX Service Mesh Deployment Fails: A Practical Troubleshooting Guide for AVS Migrations

Technologies: Azure VMware Solution (AVS), VMware HCX, ExpressRoute, HCX Service Mesh, Cloud Migration

Introduction

You've successfully deployed:

✅ Azure VMware Solution (AVS) Gen2

✅ HCX Cloud Manager in AVS

✅ HCX Connector on-premises

✅ ExpressRoute connectivity

✅ HCX Site Pairing

Everything looks healthy, and you're ready to start migrating workloads.

You proceed to create your first HCX Service Mesh and failure occurs right away.

This is one of the most common deployment issues during the initial stages of an AVS migration project.

The good news is that if Site Pairing has already succeeded, the problem is typically narrowed down to appliance deployment, network reachability, routing, DNS, firewall rules, compute profile configuration, or interconnect tunnel establishment.

This guide provides a structured troubleshooting methodology that can be followed by any engineer responsible for deploying HCX migrations into Azure VMware Solution.

Understanding What Happens During Service Mesh Deployment

Before troubleshooting, it's important to understand what HCX is attempting to do.

When Service Mesh deployment begins, HCX will:

1. Validate Compute Profiles

2. Validate Network Profiles

3. Deploy HCX Appliances

4. Configure Appliance Networking

5. Register Appliances

6. Establish Interconnect Tunnels

7. Validate Connectivity

8. Bring Service Mesh Online

Failure at any stage may cause Service Mesh creation to fail.

HCX Service Mesh Architecture

Recommended Troubleshooting Workflow

Always troubleshoot in the following order:

Step 1 - Check Service Mesh Failure Details

Step 2 - Validate Compute Profile

Step 3 - Validate Network Profile

Step 4 - Verify Appliance Deployment

Step 5 - Validate DNS

Step 6 - Validate Routing

Step 7 - Verify Required Ports

Step 8 - Check MTU Configuration

Step 9 - Verify Interconnect Connectivity

Step 10 - Review HCX Logs

Step 1: Review the Failure Message

Before collecting logs, review the Service Mesh deployment task.

Navigate to:

HCX Manager

→ Interconnect

→ Service Mesh

→ Tasks

Common messages include:

Error	Likely Cause
Appliance Deployment Failed	Compute Profile
Validation Failed	Network Profile
Tunnel Establishment Failed	Routing / Firewall
Registration Failed	DNS / Certificates
Appliance Unreachable	Gateway / Routing

Step 2: Validate Compute Profile

A misconfigured Compute Profile is one of the most common causes of first-time Service Mesh deployment failures.

Verify & Check:

1. Infrastructure

2. → Compute Profiles

Cluster

1. Connected

2. Healthy

Datastore

1. Adequate Available Capacity

Recommended:

1. 100 GB+ Free Space

Resource Pool

1. Available

2. No Resource Constraints

Port Groups

Verify:

1. Management Network

2. vMotion Network

3. Uplink Network

Step 3: Validate Network Profiles

Network Profiles are frequently responsible for deployment failures.

Verify:

1. Infrastructure

2. → Network Profiles

Check:

Setting	Validation
Gateway	Reachable
DNS Server	Valid
IP Pool	Available
Subnet	Correct
CIDR	Correct

Common Failure

1. IP Pool Exhausted

Symptoms:

1. Appliances never power on

2. Appliances deployed but unreachable

Resolution:

1. Expand available IP range

Step 4: Verify Appliance Deployment

Navigate to vCenter:

1. Recent Tasks

Review:

1. Deploy OVF Template

2. Power On VM

3. Customize VM

4. Network Assignment

Common failures:

Error	Cause
Cannot deploy OVF	Cluster Issue
No space available	Datastore Capacity
Network unavailable	Port Group Missing
Permission denied	HCX Service Account

Step 5: Validate DNS

Even after successful Site Pairing, appliance registration still requires DNS.

SSH into HCX Manager:

1. ssh admin@<HCX-IP>

   
   

2. nslookup <Remote-HCX-FQDN>

   
   

3. nslookup <vCenter-FQDN>

   
   

4. nslookup <NSX-FQDN>

Expected:

1. Forward Lookup Success

2. Reverse Lookup Success

Step 6: Validate Routing

Service Mesh deployment introduces additional appliances and networks.

Verify routes exist for:

1. HCX Uplink Network

2. Migration Network

3. Management Network

Command:

1. ip route

Verify reachability:

1. ping <Remote-HCX-IP>

2. traceroute <Remote-HCX-IP>

Step 7: Verify Required HCX Ports

Site Pairing and Service Mesh use different connectivity requirements.

Service Mesh additionally relies on:

Protocol	Port
HTTPS	443
HTTPS	9443
IKE	UDP 500
NAT-T	UDP 4500

Validate:

1. nc -zv <Remote-IP> 443

   
   

2. nc -zv <Remote-IP> 9443

For firewall validation:

1. UDP 500

2. UDP 4500

must be permitted between interconnect appliances.

Step 8: Verify MTU

This is one of the most overlooked root causes.

Symptoms:

1. Appliances Deploy Successfully

2. Tunnels Fail

3. Service Mesh Remains Down

Recommended:

1. MTU 1500 Minimum

2. MTU 9000 Preferred

Validation:

1. ping -M do -s 1472 <Remote-IP>

For Jumbo Frames:

1. ping -M do -s 8972 <Remote-IP>

Failure:

1. Packet needs fragmentation

Resolution:

1. Correct MTU across entire path

Step 9: Validate Interconnect Connectivity

Once IX appliances deploy, verify tunnel establishment.

Navigate to:

1. HCX

2. → Interconnect

Expected:

1. Tunnel Status = Up

Common issues:

Status	Meaning
Down	Firewall
Pending	Routing
Degraded	Packet Loss
Failed	MTU / Port Block

Step 10: Review HCX Logs

If infrastructure validation appears correct, logs will usually reveal the exact issue.

SSH:

1. ssh admin@<HCX-IP>

2. ssh

Access logs:

1. cd /common/logs/admin

List:

1. ls -ltr

Important files:

1. deployment.log

2. app.log

3. web.log

4. cloud.log

Search for Errors

1. grep -i error *.log

2. grep -i exception *.log

3. grep -i failed *.log

4. grep -i deploy *.log

Common Error Messages

Deployment Failure

• Resource deployment failed

Usually indicates:

• Datastore

• Cluster

• Permissions

UnknownHostException

• UnknownHostException

• DNS Failure

SSLHandshakeException

• SSLHandshakeException

• Certificate Trust Issue

Connection Timeout

• Connection timed out

• Firewall

• Routing

• NSX-T Gateway Firewall

Quick Health Check Commands

Whenever I troubleshoot a failed Service Mesh deployment, these are the first commands I run:

1. hcx service list

   
   

2. nslookup <remote-hcx>

   
   

3. ping <remote-hcx>

   
   

4. traceroute <remote-hcx>

   
   

5. nc -zv <remote-hcx> 443

   
   

6. nc -zv <remote-hcx> 9443

   
   

7. date

   
   

8. ntpq -p

   
   

9. grep -i error /common/logs/admin/*.log

   
   

10. grep -i exception /common/logs/admin/*.log

These checks quickly identify the majority of deployment failures.

Most Common Root Causes

Ranking	Root Cause	Frequency
1	Network Profile Misconfiguration	Very High
2	Firewall Blocking UDP 500 / 4500	Very High
3	DNS Resolution Failure	High
4	Routing Issue Across ExpressRoute	High
5	MTU Mismatch	Medium
6	Datastore Capacity Issue	Medium
7	Appliance Deployment Permission Issue	Medium
8	Certificate Validation Failure	Low

Lessons Learned

In real-world AVS migration projects, HCX Site Pairing success often creates a false sense of confidence that the environment is fully ready for migration. However, Service Mesh deployment introduces additional requirements, including appliance deployment, tunnel establishment, firewall traversal, MTU validation, and interconnect communication.

The majority of first-time Service Mesh failures can be resolved by systematically validating Compute Profiles, Network Profiles, appliance deployment status, routing, DNS, firewall ports, and MTU settings before focusing on HCX-specific logs.

A disciplined troubleshooting approach not only reduces deployment time but also ensures the migration platform is stable before workload migration begins.

Key Takeaway

If Site Pairing succeeds but Service Mesh deployment fails, focus first on:

✅ Network Profiles

✅ Appliance Deployment Status

✅ UDP 500 / 4500 Connectivity

✅ Routing over ExpressRoute

✅ MTU Validation

✅ HCX Deployment Logs

These six checks account for the vast majority of first-time HCX Service Mesh deployment failures in Azure VMware Solution environments.