VCF – Workload Domain Stuck Deleting After Reimaging Hosts

VCF Workload Domain Stuck Deleting Fix

Removing two dead ESXi hosts from the cluster instead of decommissioning them left the workload domain in a broken state — here’s every problem that caused and how each one was fixed.

I tried to shortcut my way to getting two failed hosts back into a workload domain. Instead of following the proper decommission process, I simply tried to remove them from the cluster — which caused a chain of flow-on issues that could have been easily avoided had I followed the correct process. This article covers everything I did that you really shouldn’t.

The scenario

Two ESXi hosts in a VCF workload domain died. SDDC Manager’s own Remove Host from Cluster dialog has a Force Remove Host checkbox for exactly this situation — a host that’s dead and can’t be gracefully evacuated. Using it keeps the whole operation inside SDDC Manager, which handles the NSX transport node and vSAN cleanup as part of that same workflow.

I skipped that and removed the hosts directly in vCenter instead, to save time.

That skips SDDC Manager’s cleanup entirely. The NSX transport node registration, the local service accounts, and SDDC Manager’s own inventory records all stayed in place, now pointing at nothing. Reimaging the hosts to bring them back made it worse: reimaging wipes the SSH host key and local accounts too, so nothing SDDC Manager expected to still be there actually was.

At that point the domain was in a state SDDC Manager had no clean way to reconcile — hosts it half-remembered, credentials that no longer matched, NSX objects nothing was tracking properly anymore. Fixing it in place wasn’t realistic, so the decision was to delete the whole workload domain and rebuild it. That decision is the reason for everything that follows: deleting a domain assumes a consistent, healthy starting state, and every one of the ten problems below is SDDC Manager’s own internal safeguards refusing to proceed because that assumption was no longer true.

Attempting to delete the workload domain afterward failed outright, and every fix uncovered another layer. This is the full chain, in order — including the wrong turns, because they’re the useful part.

TL;DR

• reject HostKey in operationsmanager.log = SSH host key mismatch, not a cert problem → fixHostKeys.py (KB 316028)
• Service account “disconnected” post-reimage = local account is gone → recreate with esxcli system account add + permission set (KB 314643)
• “Resources are not in ACTIVE state” = stuck lifecycle status in the platform DB host table (and later, the domain table) → flip it to ACTIVE manually (KB 379319)
• Retries pile up = clear stale rows in operationsmanager DB’s passwordmanager.password_operations table
• “Incorrect username/password” with a value straight from SDDC Manager can happen even when the account and value look right — pull fresh credentials and compare before assuming the value you have is still current
• SSH login working again doesn’t necessarily mean SDDC Manager’s own login will work — a separate pam_tally2 reset was needed for SSH specifically; whether that alone fixed the deletion workflow’s login is unconfirmed
• “Workload domain is already being removed by another task” on a fresh attempt = the domain object itself, not just the hosts, is stuck in DEACTIVATING
• No route to host against NSX mid-deletion = a prior partial run already deleted the NSX Manager VM; that’s expected, not a new problem — let the timeout run or use Force
• Snapshot SDDC Manager before every DB edit, always
• Removing a dead host straight from the cluster in vCenter isn’t the same as using SDDC Manager’s Remove Host from Cluster dialog — its Force Remove Host checkbox exists for exactly this case and keeps NSX/vSAN cleanup inside SDDC Manager instead of leaving it orphaned

The chain, at a glance

Problem 1Hosts reimaged

Two ESXi hosts died and were reimaged with the same IP and root password. Reimaging regenerates the SSH host key, so SDDC Manager’s stored key no longer matched.

Problem 2SSH key rejected

SDDC Manager refused to SSH in because of the mismatched host key. Fixed by re-registering the new key with fixHostKeys.py.

Problem 3Account missing

The reimage also wiped the local ESXi service account SDDC Manager uses to log in. Recreated it manually with esxcli.

Problem 4Host stuck inactive

SDDC Manager’s own database still had the host marked DEACTIVATING, left over from an earlier failed deletion attempt. Updated the status to ACTIVE directly in Postgres.

Problem 5Old tasks blocking

Earlier failed attempts left stuck task records that blocked new ones from running. Cleared them in the password_operations table.

Problem 6Login still fails

Host login still failed. Set a new password directly on the host and matched it in SDDC Manager.

Problem 7SSH locked out

SSH was also locked out from repeated failed login attempts. Reset the failed-login counter with pam_tally2.

Problem 8Creds mismatched

A fresh pull of SDDC Manager’s stored credentials showed a value that no longer matched the host. Reset both sides to the same value.

Problem 9Domain stuck inactive

The domain itself, not just the hosts, was stuck DEACTIVATING in the database from the same earlier failed attempt. Corrected it the same way.

Problem 10NSX unreachable

A prior partial run had already deleted the NSX Manager VM, so the workflow couldn’t reach it for a health check. Used Force delete to finish.

A single clean removal-instead-of-decommission on a healthy domain will usually stop after step 2 or 3. Two dead hosts, reimaged mid-lifecycle, with an already-failed prior deletion attempt in the mix, manages to hit almost every one of these.

Rabbit hole #1: it’s not a certificate issue

First guess: reimage → new self-signed cert → SDDC Manager doesn’t trust it. Wrong. The real error was an SSH host key mismatch. Reimaging regenerates /etc/ssh/ssh_host_*_key; SDDC Manager still has the old fingerprint pinned in known_hosts and rejects the connection — which can surface as a generic auth failure that looks like a bad password.

Fix — KB 316028, fixHostKeys.py:

ssh vcf@sddc-manager
su -

python fixHostKeys.py --node wld-h1.region2.shank.com
python fixHostKeys.py --node wld-h2.region2.shank.com

Logs: /var/log/vmware/vcf/fixHostKeys.log. FIPS environments: use the fix_known_hosts.sh fallback in the same KB instead.

Lesson: reject HostKey in operationsmanager.log means this, not a cert problem.

Rabbit hole #2: the local service accounts don’t exist anymore

SDDC Manager couldn’t log in to its own local ESXi service accounts (svc-vcf-wld-h1, svc-vcf-wld-h2). A reimage wipes local OS accounts entirely — SDDC Manager’s DB still had records for them; the hosts didn’t.

Fix — KB 314643, recreate manually:

esxcli system account add --id svc-vcf-wld-h1 --password '<password-from-SDDC-Manager>' --password-confirmation '<password-from-SDDC-Manager>'
esxcli system permission set --id svc-vcf-wld-h1 --role Admin

Repeat per host, then Security → Password Management → Remediate in the UI with the same password. Ignore the “already set to this password” warning.

Pulling the stored passwords: KB 327195 (token + curl against /v1/system/credentials/service). Treat the output as live secrets — don’t paste it anywhere outside your own tooling, and rotate afterward.

Rabbit hole #3: the resource is stuck DEACTIVATING

Remediate still failed:

Status of Resource fff9c8c9-abd1-431c-b423-e983d9276def : DEACTIVATING
Prevalidation of password REMEDIATE has failed, error : Resources : [wld-h1.region2.shank.com] are not in ACTIVE state

Not a credential issue — SDDC Manager’s resource tracking had the host stuck in DEACTIVATING, left over from the earlier interrupted deletion attempt.

Fix — KB 379319, correct it directly in Postgres:

psql -h localhost -U postgres -d platform -c "UPDATE host SET status='ACTIVE' WHERE id='<host1-id>';"
psql -h localhost -U postgres -d platform -c "UPDATE host SET status='ACTIVE' WHERE id='<host2-id>';"

systemctl restart operationsmanager

Snapshot SDDC Manager before any direct DB edit. Always.

Rabbit hole #4: five stale tasks blocking retries

Retries kept colliding with stale PREVALIDATION_FAILED rows from earlier attempts. Note the database — operationsmanager, not platform:

psql -h localhost -U postgres -d operationsmanager -c \
"UPDATE passwordmanager.password_operations SET transaction_status='USER_CANCELLED' WHERE transaction_status IN ('FAILED','PREVALIDATION_FAILED');"

Rabbit hole #5: the password itself gets rejected

With that cleared, Remediate reached a real login attempt — and failed again:

com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

esxcli system account add returned no error with either the long original secret or a short replacement, and this build’s advanced-settings tree doesn’t have the legacy /Security/ branch at all — so the usual password-length-policy explanation doesn’t apply here. Cause not confirmed.

The fix that worked: set a short password on both sides.

esxcli system account remove --id svc-vcf-wld-h1
esxcli system account add --id svc-vcf-wld-h1 --password '<new-short-password>' --password-confirmation '<new-short-password>'
esxcli system permission set --id svc-vcf-wld-h1 --role Admin

Use Update Password in the UI, not Remediate — Remediate assumes the stored value already matches.

Rabbit hole #6: SSH login works, vim-API login doesn’t

SDDC Manager’s own login (vim API, not SSH) kept failing with the same InvalidLogin. SSH itself also failed at first — until resetting the failed-login counter:

pam_tally2 --user svc-vcf-wld-h1 --reset

That fixed SSH. Checking the equivalent vim-API lockout setting hit the same missing-branch error as rabbit hole #5. Whether a separate vim-API lockout was also in play: not confirmed — the actual fix turned out to be rabbit hole #7.

Rabbit hole #7: the stored secret had drifted

A fresh pull of SDDC Manager’s stored credentials showed a secret that no longer matched what had just been set on the host, with a newer modification time than expected. Cause not confirmed.

Fix: pull the current DB value, push it onto the host directly, then Remediate (not Update, since the values now genuinely match):

esxcli system account remove --id svc-vcf-wld-h1
esxcli system account add --id svc-vcf-wld-h1 --password '<secret-from-fresh-DB-pull>' --password-confirmation '<secret-from-fresh-DB-pull>'
esxcli system permission set --id svc-vcf-wld-h1 --role Admin

Lesson: re-pull current state before another round rather than trusting a value from a few steps back.

Rabbit hole #8: the domain itself is stuck, not just the hosts

With hosts authenticating cleanly, a fresh Delete Workload Domain attempt failed instantly:

Message: Workload domain reg2-wld is already being removed by another task.

Same bug as rabbit hole #3, one level up — the domain object’s own status, not just the hosts, was stuck DEACTIVATING.

psql -h localhost -U postgres -d platform -c "UPDATE domain SET status='ACTIVE' WHERE id='<domain-id>';"

systemctl restart domainmanager
systemctl restart operationsmanager

You can’t resume a stuck domain-deletion task from the UI, and deleting the task row directly leaves the domain stuck rather than freeing it — the status flag has to be corrected first.

Rabbit hole #9: NSX unreachable — a prior run already deleted it

With the domain status fixed, a fresh attempt got through inventory, credentials, and locking — real progress — then hit:

com.vmware.vapi.client.exception.ConnectionException: No route to host

Not a new bug: a prior partial run had already deleted the NSX Manager VM. The health check was polling something that no longer exists.

Two options: let the 60-second retry/timeout window run its course, or use the Force option on domain deletion (checkbox in the UI, or forceDelete via API) — built for exactly this. Don’t touch NSX-side DB tables to fake a response.

The finish line

All ten rabbit holes later, the workflow ran clean end to end and the domain is actually gone — not just gone from one table while lingering everywhere else.

The path I didn’t take

Broadcom documents a fully manual route: purging a workload domain’s rows out of the database directly when the normal delete workflow won’t complete (KB 327192). Worth knowing it exists — but Broadcom doesn’t publish the commands, flags the risk as HIGH, and points to Support instead. A wrong DELETE across host, domain, and everything referencing them can leave the platform DB worse off than the original problem. If the chain above doesn’t get you there, rebuilding the domain fresh is the more honest lab answer than hand-editing Postgres against guesses. Note that DELETE /v1/hosts?force=true (KB 325780) is not a shortcut here — it only applies to hosts already unassigned from a domain, which these weren’t until the domain itself was gone.

Checklist for next time

• reject HostKey in operationsmanager.log → SSH host key mismatch, not a cert issue → fixHostKeys.py (KB 316028)
• Service account “disconnected” after a host rebuild → local account is gone, recreate via esxcli system account add + permission set (KB 314643)
• “Resources are not in ACTIVE state” → check the relevant table’s status in the platform DB — host or domain (KB 379319)
• Repeated retries piling up → clear stale rows in operationsmanager DB’s passwordmanager.password_operations table
• “Incorrect username or password” with an unexpected value → pull fresh credentials from SDDC Manager and compare against what’s on the host
• SSH login failing → check pam_tally2 --user <account>, reset if locked
• “Already being removed by another task” on a fresh attempt → check domain-level status, not just host-level
• No route to host against NSX mid-deletion → check whether a prior partial run already deleted that component; use Force
• Snapshot SDDC Manager before every direct DB edit. No exceptions, lab or prod.
• Treat any service-account secret you pull out of SDDC Manager as live — don’t paste it outside your own tooling, and rotate it once you’re done.