HCX Site Pairing Failure Between AVS Gen2 and On-Premises: A Practical Troubleshooting Guide
Updated: 2 days ago
Category: HCX | AVS | VCF | Cloud Migration
Audience: VMware Admin, VCF Architects, HCX Engineers, Cloud Consultants
Scenario:
✅ Azure VMware Solution (AVS) Gen2 deployed
✅ HCX deployed in AVS
✅ HCX deployed on-premises
✅ ExpressRoute connectivity established between AVS and on-premises
✅ Both HCX Managers appear healthy
❌ HCX Site Pairing fails
When this situation occurs, the issue is almost always one of the following:
DNS Resolution
Routing
Firewall rules
Certificate trust
Time synchronization (NTP)
HCX Manager service failures
Incorrect HCX URL/FQDN
ExpressRoute route advertisement issues
NSX-T Gateway Firewall in AVS
Solution Architecture
ON-PREMISES DATA CENTER
VMware Cloud Foundation / vSphere
|
On-prem vCenter
|
HCX Connector
|
Management Network
|
Firewall / WAN
|
ExpressRoute / Routing
|
AVS Private Cloud
|
HCX Cloud Manager
|
AVS vCenterTroubleshooting Workflow
1. DNS Validation
2. Connectivity Validation
3. Route Validation
4. Port Validation
5. HTTPS Validation
6. HCX Services
7. NTP Synchronization
8. Certificate Validation
9. AVS Gateway Firewall
10. HCX Log Analysis
Step 1: Validate DNS
Site pairing depends heavily on FQDN-based communication.
Commands
From On-Prem HCX Manager
ssh admin@<remote-hcx-fqdn or IP>
nslookup <remote-hcx-fqdn>
nslookup <remote-hcx-ip>
From AVS HCX Manager
nslookup <OnPrem-HCX-FQDN>
nslookup <OnPrem-HCX-IP>
Expected
Check | Expected Result |
Forward Lookup | Successful |
Reverse Lookup | Successful |
Returned IP | Correct |
Common Failure
Error | Root Cause |
NXDOMAIN | Missing DNS Record |
Wrong IP | Incorrect DNS Entry |
Timeout | Unreachable DNS Server |
Common Findings
Wrong DNS Server:
cat /etc/resolv.conf
Verify:
Correct DNS configured
Forward lookup works
Reverse lookup works
Step 2: Validate Network Connectivity
Verify that traffic traverses ExpressRoute successfully.
Commands
ping <remote-hcx-ip>
traceroute <remote-hcx-ip>
Note - Run above commands from On-Prem HCX and AVS HCX both and validate.
Validation Table
Result | Meaning |
Ping Success | Basic Connectivity Available |
Ping Fails | Routing or Firewall Issue |
Traceroute Stops | Intermediate Routing Issue |
Step 3: Validate Routing
One of the most overlooked causes of site pairing failure. Many customers can ping AVS HCX but the return route is missing.
Command
From On-Prem HCX Manager
ip route
Verify
Network | Reachable |
AVS Management CIDR | Yes |
HCX Management Network | Yes |
Return Path Exists | Yes |
Common Problem
ExpressRoute route advertised only one way
Result:
Ping may work intermittently
Site pairing fails
From AVS HCX Manager
ip route
On-prem HCX subnet reachable
Step 4: Validate Required Ports
HCX site pairing primarily requires secure management communication.
Required Ports
Source | Destination | Port |
HCX Connector | HCX Cloud | TCP 443 |
HCX Connector | HCX Cloud | TCP 9443 |
HCX Cloud | HCX Connector | TCP 443 |
HCX Cloud | HCX Connector | TCP 9443 |
Commands
nc -zv <remote-hcx-ip> 443
nc -zv <remote-hcx-ip> 9443
Test from AVS:
nc -zv <OnPrem-HCX-IP> 443
nc -zv <OnPrem-HCX-IP> 9443
Expected
Connection succeeded
If Failed
Review:
Network Firewall
NSX-T Gateway Firewall
Azure Security Controls
Routing
Step 5: Validate HTTPS Communication
This quickly identifies SSL or firewall issues.
Commands
curl -vk https://<remote-hcx-fqdn>
or
curl -vk https://<remote-hcx-IP>
Test from AVS:
curl -vk https://<ONPREM-HCX-FQDN>
Expected Output
SSL connection established
SSL handshake successful
HTTP response received
Troubleshooting Matrix
Error | Likely Cause |
Connection Timeout | Firewall |
Connection Refused | Service Down |
SSL Handshake Failure | Certificate Issue |
Host Not Found | DNS Issue |
Step 6: Verify HCX Services
SSH into the HCX Manager.
Check Services
hcx service list
systemctl status web-engine
systemctl status app-engine
'or'
systemctl status web-engine app-engine
Expected
RUNNING
Look for:
web-engine
cloud-service
app-engine
Restart Services
hcx service restart
Wait approximately:
10-15 minutes
Step 7: Validate NTP Synchronization
Time drift can break authentication and certificate validation.
Commands
date
ntpq -p
timedatectl status
cat /etc/systemd/timesyncd.conf
Validation
Check | Expected |
NTP Peer | Reachable |
Time Difference | Minimal |
Synchronization | Active |
Step 8: Verify SSL Certificates
Site pairing relies on secure certificate validation.
Validate Certificate
From browser:
Verify:
Certificate valid
CN matches FQDN
No expired cert
Command line:
openssl s_client -connect <remote-hcx-fqdn>:443
Verify
Check | Expected |
Certificate Valid | Yes |
Not Expired | Yes |
Chain Complete | Yes |
Trusted Root | Yes |
Common Error
SSLHandshakeException
This usually indicates:
Expired Certificate
Mismatched CN/SAN
Certificate Trust Issue
Step 9: Verify AVS NSX-T Gateway Firewall
A very common cause of AVS HCX pairing failures.
Validate Rules
AVS Private Cloud
→ Workload Networking
→ Gateway Firewall
Required Access
Source | Destination | Port |
On-Prem HCX | AVS HCX | TCP 443 |
On-Prem HCX | AVS HCX | TCP 9443 |
AVS HCX | On-Prem HCX | TCP 443 |
AVS HCX | On-Prem HCX | TCP 9443 |
Remember:
Successful ping does not guarantee successful HCX communication.
Step 10: Review HCX Logs
If all infrastructure checks pass, logs usually reveal the root cause.
Log Location
ssh admin@hcx
switch to root
cd /common/logs/admin
ls -ltr
Most useful logs:
app.log
web.log
cloud.log
Useful Commands
grep -i error app.log
grep -i exception app.log
grep -i pairing *.log
grep -i fail *.log
Error Interpretation
Log Entry | Likely Cause |
UnknownHostException | DNS |
SSLHandshakeException | Certificate |
Connection Timed Out | Firewall / Routing |
Authentication Failed | NTP / Credentials |
Step 11: Generate HCX Support Bundle
UI:
Administration
→ Troubleshooting
→ Support Bundle
or CLI:
generate-support-bundle
Quick 10-Minute Health Check
Whenever I join a customer troubleshooting bridge, these are the first commands I execute:
nslookup <remote-hcx-fqdn>
ping <remote-hcx-ip>
nc -zv <remote-hcx-ip> 443
nc -zv <remote-hcx-ip> 9443
curl -vk https://<remote-hcx-fqdn>
date
ntpq -p
hcx service list
grep -i exception /common/logs/admin/app.log
These checks identify the root cause for the majority of HCX site pairing failures.
Most Common Root Causes
Ranking | Root Cause | Probability |
1 | DNS Resolution Failure | ⭐⭐⭐⭐⭐ |
2 | NSX-T Gateway Firewall Blocking 443/9443 | ⭐⭐⭐⭐⭐ |
3 | Missing ExpressRoute Route Advertisement | ⭐⭐⭐⭐ |
4 | SSL Certificate Trust Failure | ⭐⭐⭐ |
5 | NTP Synchronization Issue | ⭐⭐⭐ |
Final Thoughts
In real-world AVS migration engagements, HCX Site Pairing failures are rarely caused by the HCX software itself. More often, they stem from underlying infrastructure dependencies such as DNS, routing, firewall policies, certificate validation, or time synchronization.
By following a structured troubleshooting approach and validating each dependency systematically, engineers can usually identify and resolve the issue within minutes rather than hours.
Comments