A virtual machine file lock issue rarely shows up at a convenient time. It usually appears when a VM refuses to power on, a consolidation task is stuck, a backup window has overrun, or vCenter and an ESXi host disagree about the VM’s state.
That is exactly when bad habits get expensive.
The temptation is to find a process, kill it, retry the power-on, and move on. Sometimes that works. Sometimes it turns a lock investigation into a corrupted disk chain, a broken snapshot tree, or an application outage that is harder to explain than the original alert.
Broadcom KB314365 gives the core mechanics for investigating virtual machine file locks on ESXi hosts. This runbook takes that workflow and wraps it in a production-safe operating model: identify the locked file, determine the real lock owner, correlate host ownership, validate datastore health, avoid destructive fixes, and only then decide whether process termination is appropriate.
The goal is not to “break the lock.”
The goal is to understand why the lock exists before taking an action that cannot be rolled back.
Scenario
You are operating a vSphere or VMware Cloud Foundation environment and a VM has one or more of the following symptoms:
- Power-on remains stuck near completion or fails.
- The VM appears as Invalid.
- vCenter and the ESXi Host Client show conflicting VM power states.
- A disk add, snapshot consolidation, clone, or backup-related operation fails.
- The task reports an error such as Unable to access a file since it is locked.
- The VMX file or a VMDK cannot be opened from the host shell.
Broadcom KB 314365 lists these types of symptoms, including power-on tasks stuck around 95%, invalid VM state, conflicting power states, locked-file errors, swap-file errors, VMX access errors, and vmkernel log messages indicating that a swap file could not be opened because the lock was not free.
In production, those symptoms are not enough to justify killing a VMX process. They are only enough to justify starting a lock investigation.
Why File Locks Matter Operationally
ESXi file locking is a protection mechanism, not just an obstacle. A powered-on VM legitimately holds locks on files it is actively using so that multiple hosts or processes do not make unsafe concurrent changes. Broadcom describes this purpose directly: ESXi hosts establish locks on critical VM files and file systems to prevent concurrent changes.
Runtime locks can involve files such as:
VMNAME.vswpDISKNAME-flat.vmdkDISKNAME-00000x-delta.vmdkVMNAME.vmxVMNAME.vmxfvmware.log
Those file types are called out in KB 314365 as examples of VM files locked during runtime.
That matters because a lock does not automatically mean something is wrong. A lock can be legitimate when the VM is powered on, when a snapshot-based backup appliance has hot-added a disk, when a VM is using snapshots, when multi-writer disks are configured, or when a management task is still running.
The unsafe move is assuming every lock is stale.
Symptoms and Risk
The most common operational mistake is treating all lock symptoms as the same issue. A locked VMDK during backup is different from a stale NFS .lck file. A vSAN object lock is different from a VMFS metadata lock. A datastore in APD or PDL is different from a VMX process that did not exit cleanly.
Use this table as the first triage filter.
| Symptom | Likely Investigation Path | Production Risk |
|---|---|---|
| VM will not power on and task shows locked file | Identify locked file and lock owner | Repeated retries may create noise without fixing ownership |
| Snapshot consolidation fails | Check backup proxy, hot-add, and active tasks | Killing the wrong process can damage backup or consolidation state |
| VM is Invalid or inaccessible | Validate datastore access before lock remediation | Storage outage may be the root cause, not a stale VM process |
| Lock owner is another ESXi host | Correlate MAC/IP/host before action | Killing locally will not clear a remote lock |
NFS .lck file exists after VM is powered off | Investigate NFS lock owner and stale lock file | Deleting the wrong .lck while a VM is active can be destructive |
| vSAN object lock exists | Investigate vSAN object UUID lock | File-path assumptions from VMFS may not apply |
Broadcom’s locked virtual disk guidance also notes that locked disks can result from powered-on VM ownership, snapshot-based backup appliances, unsupported disk formats, or existing locks.
Runbook Workflow at a Glance
The important thing to notice in this workflow is that process termination is intentionally late. The runbook moves from evidence collection to ownership correlation to datastore validation before remediation.
Prerequisites and Safety Checks
Before you run commands, establish control of the situation.
You need:
- vCenter access with permission to view tasks, events, hosts, datastores, and VM configuration.
- SSH access to the relevant ESXi hosts.
- The VM name, datastore name, VM folder path, and expected power state.
- The current backup or replication job status for the VM.
- Awareness of whether the datastore is VMFS, NFS, or vSAN.
- A maintenance or change record if remediation may affect a running workload.
- Application owner awareness if a hard VM process kill becomes a possibility.
Do not start with kill.
Do not delete VMDKs.
Do not delete .lck files on VMFS. Broadcom explicitly notes that VMFS volumes do not have .lck files because VMFS locking is handled in VMFS metadata.
Do not use a broad services.sh restart as a casual fix. Broadcom’s ESXi management-agent guidance warns not to use services.sh restart or DCUI Restart Management Agents when vSAN, LACP, NSX, or shared graphics are in use; it recommends restarting single services instead.
That warning matters in VCF environments because vSAN and NSX are common parts of the platform design. A lock incident should not become a management-plane incident.
Stage 1: Capture the Error Before You Change Anything
Start with the failed task and the VM’s current state.
In newer environments, this may be easier than it used to be. Broadcom notes that starting with ESXi 8.0 U2, the file-lock owner can be shown in the vSphere Client by navigating to the VM’s Monitor tab, then Tasks and Events, then Tasks. The locked virtual disk KB similarly states that vSphere 8.0 U2 can show lock details in the failed Power On virtual machine task status, including file path, host, MAC, world name, and lock mode.
Capture:
- Failed task name
- Exact error text
- Locked file path
- Host reported in the task
- MAC address reported in the task
- Lock mode
- World name, if shown
- VM power state in vCenter
- VM power state in the ESXi Host Client
- Recent backup, snapshot, clone, storage, or vMotion activity
If the UI does not expose the lock owner, perform one controlled reproduction only if appropriate. KB 314365 describes powering on the VM so the operation fails and the error can be noted, then connecting to the ESXi host by SSH. In production, do this only when the VM is expected to be powered on and no other recovery workflow is already in progress.
Stage 2: Identify the Datastore Type
The datastore type determines the investigation path.
| Datastore Type | Lock Investigation Pattern | Important Caveat |
|---|---|---|
| VMFS | Use vmfsfilelockinfo against the relevant flat, delta, or sesparse file | Do not look for .lck files on VMFS |
| NFS | Inspect .lck-#### files and decode owner information | Only remove stale .lck files when the VM is powered off |
| vSAN | Resolve the vSAN object UUID and inspect .UUID.lck | vSAN virtual disk objects are not the same as VMFS flat files |
Broadcom KB 314365 states that vmfsfilelockinfo can be run against the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.
Stage 3: Investigate VMFS Locks
For VMFS, identify the active disk file from the VM configuration or from the failure message. If snapshots are present, the locked file may be a delta disk rather than the base descriptor.
From an ESXi host with access to the datastore:
cd /vmfs/volumes/<datastore>/<vm-folder> ls -lah vmfsfilelockinfo -p /vmfs/volumes/<datastore>/<vm-folder>/<locked-file>.vmdk \ -v <vcenter-fqdn-or-ip> \ -u <sso-user>
Example pattern:
vmfsfilelockinfo -p /vmfs/volumes/prod-ds01/App01/App01-000003-delta.vmdk \ -v vcsa01.example.com \ -u administrator@vsphere.local
vmfsfilelockinfo can use vCenter credentials to trace the MAC address back to an ESXi host. KB 314365 shows the command returning the MAC address of the host holding the lock and, when vCenter lookup succeeds, the host owning the lock and the lock mode.
Interpret the lock mode carefully:
| Lock Mode | Meaning |
|---|---|
mode 0 | No lock |
mode 1 | Exclusive lock |
mode 2 | Read-only lock |
mode 3 | Multi-writer lock |
Broadcom lists these lock modes and gives examples such as exclusive locks for powered-on VM files, read-only locks for certain snapshot cases, and multi-writer locks for MSCS or FT-style scenarios.
The practical point is simple: mode 1 is not automatically stale, and mode 3 may be intentional.
Stage 4: Investigate NFS Locks
NFS locking behaves differently from VMFS. Broadcom’s locked virtual disk KB explains that NFS does not provide block-level access for SCSI locks and that NFS locks are implemented by creating .lck-#### files on the NFS server; it also notes that the same command-line tools used for VMFS lock holders cannot be used the same way for NFS.
From the ESXi host where the affected VM is registered:
cd /vmfs/volumes/<nfs-datastore>/<vm-folder> ls -lha
If you find a .lck-#### file, inspect it:
hexdump -C .lck-####
Broadcom documents this approach and notes that the output can provide the hostname of the lock owner.
Only consider moving or removing the lock file after you have confirmed:
- The affected VM is powered off.
- Backup proxies are not using the disk.
- No other VM has the disk mounted.
- No ISO, RDM, or other configuration is referencing the file.
- The application owner understands that power-on or consolidation will be retried after the lock is moved.
KB 314365 gives a conservative NFS remediation pattern: power down the VM, create a bkup directory, move the lck-#### files into it, confirm they moved, and only remove lock files for a powered-off VM.
Use move, not delete, unless your support process says otherwise:
mkdir bkup mv .lck-#### bkup/ ls -lah bkup
If the .lck file is immediately recreated, stop. That means something is still touching the disk.
Stage 5: Investigate vSAN Locks
vSAN requires a different mental model. Broadcom’s vSAN lock KB explains that vSAN uses a specific object type for virtual disks and that virtual disks are not stored with the VM configuration files in the namespace directory in the same way VMFS files are.
Start in the VM namespace:
cd /vmfs/volumes/vsanDatastore/<VM_Namespace>
Find the vSAN object UUID from the VMDK descriptor:
grep RW <VMDiskName>.vmdk
You are looking for an extent similar to:
RW 209715200 VMFS "vsan://########-####-####-####-########31f0"
Broadcom documents this pattern and identifies the UUID in the vsan:// path as the vSAN object representing the virtual disk.
Then look for lock files:
ls -lah .*.lck ls -lah *.lck
Check the object lock:
vmfsfilelockinfo -p .########-####-####-####-########31f0.lck
Broadcom’s vSAN guidance shows vmfsfilelockinfo -p .<uuid>.lck returning the host owning the lock and lock mode, or showing that the file is not locked and is free.
For a broader check in the VM directory, use a quoted loop so VM file names with spaces do not break the command:
for file in *; do
echo "== ${file} =="
vmfsfilelockinfo -p "${file}" 2>/dev/null | grep -iE "locked|mode|owner|free"
done
For hidden vSAN lock files:
for file in .*lck; do
echo "== ${file} =="
vmfsfilelockinfo -p "${file}" 2>/dev/null | grep -iE "locked|mode|owner|free"
done
Broadcom provides similar vSAN loops for checking all VM files and hidden .lck files.
Stage 6: Correlate the Lock Owner to a Host
Once you have a host, MAC address, or world name, map it back to the real ESXi owner before taking action.
If vmfsfilelockinfo returns the host directly, record it.
If you only have the MAC address, map VMkernel adapters across the cluster. This PowerCLI snippet is read-only and is useful when you need to correlate a lock-owner MAC address to a host without clicking through every host in the UI.
# Map ESXi VMkernel MAC addresses to hosts.
# Change the vCenter name before running.
# This does not modify the environment.
Connect-VIServer vcsa01.example.com
Get-VMHost | Sort-Object Name | ForEach-Object {
$vmhost = $_
Get-VMHostNetworkAdapter -VMHost $vmhost -VMKernel |
Select-Object `
@{Name = 'VMHost'; Expression = { $vmhost.Name } },
Name,
Mac,
IP,
PortGroupName
} | Sort-Object Mac | Format-Table -AutoSize
You are looking for the MAC address reported by the failed task or vmfsfilelockinfo.
If the MAC does not appear, verify that:
- You are connected to the correct vCenter.
- The host is still in inventory.
- The VM is not on a datastore shared outside the expected cluster.
- The management VMkernel adapter was not recently changed.
- The host has not been reinstalled or replaced.
- You are not looking at a backup proxy or another VM that has hot-added the disk.
Stage 7: Decide Whether the Lock Is Expected
Before remediation, classify the lock.
| Finding | Interpretation | Next Action |
|---|---|---|
| VM is powered on on the owning host | Likely expected lock | Do not kill; validate VM state and inventory |
| Backup proxy has the disk hot-added | Expected during backup, stale if backup failed | Remove disk from proxy safely; do not delete from disk |
| Snapshot consolidation is running | Task may legitimately hold a lock | Validate task progress before interruption |
| VM is powered off, no backup, no other VM reference | Lock may be stale | Continue to process/task investigation |
| Lock owner is unknown and datastore has APD/PDL signs | Storage issue may be primary | Fix storage path before VM remediation |
| Multi-writer lock is present | May be intentional | Validate cluster/app design before touching it |
For vSAN specifically, Broadcom recommends checking backup proxy servers and removing the affected disk from the proxy if it is still attached, making sure Delete from disk is not selected.
That same principle applies operationally outside vSAN as well: when a backup proxy has hot-added a production disk, the fix is not to delete the disk. The fix is to detach it safely from the proxy and then retry consolidation or power-on.
Stage 8: Find the Process, Cartel ID, or Task Holding the Lock
On the ESXi host that owns the lock, use lsof and esxcli to connect the file lock to a process.
For a known locked file:
lsof | egrep 'Cartel|<locked-file-name>'
Example:
lsof | egrep 'Cartel|App01-000003-delta.vmdk'
KB 314365 shows this pattern and explains that the output can reveal a VMX Cartel ID for the VM holding the file lock.
Then list active VM processes:
esxcli vm process list
Compare the VMX Cartel ID from lsof to the output of esxcli vm process list. KB 314365 documents using esxcli vm process list to map the Cartel ID back to the VM display name and configuration file.
If you suspect a running task rather than a normal VMX owner, inspect VM tasks:
vim-cmd vmsvc/getallvms | grep -i <vm-name> vim-cmd vmsvc/get.tasklist <vmid> vim-cmd vimsvc/task_info <task-id>
Broadcom’s locked virtual disk KB uses this flow to identify running VM tasks such as snapshot removal and inspect task state, progress, start time, and cancelability.
If no obvious process is returned, search for another VM that has the VMDK referenced in its VMX file. Keep this scoped; do not run broad datastore searches during a storage incident unless you understand the load.
cd /vmfs/volumes/<datastore> find . -name "*.vmx" -print0 | \ xargs -0 grep -H "<disk-name-or-vmdk-fragment>"
This is read-only, but it can still be expensive on large datastores.
Stage 9: Validate Datastore Access Before Remediation
A file lock can be a symptom of a deeper storage issue. Before restarting agents, killing processes, or moving lock files, validate that the datastore is healthy from the relevant hosts.
Check for APD or PDL indicators if hosts are disconnected, datastores are inaccessible, migrations are stuck, or VMs are invalid. Broadcom’s APD/PDL KB lists symptoms such as unavailable datastores, all paths marked dead, hosts becoming unresponsive, stuck migrations, and VMs becoming inaccessible.
On the affected host:
cd /var/run/log grep -i "esx.problem.storage.apd.start" vobd.log grep -i "esx.clear.storage.apd.exit" vobd.log grep -i "permanently inaccessible\|perm loss\|APD\|PDL" vmkernel.log
Broadcom documents checking vobd.log for esx.problem.storage.apd.start and esx.clear.storage.apd.exit events when validating whether a host has LUNs or datastores in APD state.
Also check whether the datastore is accessible from all expected hosts:
esxcli storage filesystem list esxcli storage core path list | less
If the datastore is in APD or PDL, do not treat the VM lock as the root problem. Broadcom states that APD must be resolved at the storage array or fabric layer to restore connectivity, and that affected hosts may require reboots to remove residual references after the condition is resolved.
Stage 10: Remediate Using the Least Destructive Option
Remediation should follow the evidence.
If the Lock Belongs to a Running VM
Do not kill the process just because you found it.
Confirm whether the VM is actually running on the owning host. If vCenter is wrong but the ESXi host shows the VM running, update your incident assessment. You may be dealing with inventory inconsistency, host management-agent issues, or a stale vCenter view.
If the Lock Belongs to a Backup Proxy
Coordinate with the backup owner.
Check whether the disk is still hot-added to a proxy. Remove the disk from the proxy without deleting it from the datastore. In Broadcom’s vSAN guidance, the instruction is explicit: if the affected disk is still attached to the proxy, remove it while ensuring Delete from disk is not selected.
If locks persist after host-side cleanup, Broadcom notes that restarting hostd may not be enough when a backup proxy VM is actively retaining locks; the backup service may need to be restarted on the proxy VM, or the proxy VM may need to be rebooted.
If the Lock Appears to Be a Management-Agent Task
Restart only the specific management agents required, and only after checking the platform context.
/etc/init.d/hostd restart /etc/init.d/vpxa restart
Broadcom documents restarting hostd and vpxa individually from ESXi Shell or SSH.
In HA-enabled clusters, be deliberate. Broadcom’s locked virtual disk KB warns to deactivate HA Host Monitoring before restarting management agents to prevent unwanted VM failover.
If the Lock Is an NFS .lck File and the VM Is Powered Off
Move the stale lock file into a backup folder.
cd /vmfs/volumes/<nfs-datastore>/<vm-folder> mkdir bkup mv .lck-#### bkup/ ls -lah bkup
Only do this when the VM is powered off. KB 314365 cautions to only remove .lck files for a powered-off VM, and it scopes .lck file movement to NFS.
If a VMX Process Must Be Killed
This is the last-resort path, not the normal path.
Broadcom’s locked virtual disk KB describes esxcli vm process kill -t force -w <World ID> as a hard shutdown and also notes that, if the guest is responsive, you should try shutting down from inside the guest OS instead.
Use this decision gate before killing a process:
| Question | Required Answer |
|---|---|
| Have you identified the correct host? | Yes |
| Have you identified the correct VM World ID? | Yes |
| Is the VM already unresponsive or confirmed safe to hard power off? | Yes |
| Has the application owner accepted outage/corruption risk? | Yes |
| Have you ruled out backup proxy hot-add ownership? | Yes |
| Have you ruled out active consolidation or storage APD/PDL? | Yes |
| Do you have rollback or recovery steps documented? | As much as possible |
Command pattern:
esxcli vm process list esxcli vm process kill -t force -w <World-ID>
There is no clean rollback for a forced VMX kill. Treat it like pulling power from a physical server.
Validation Steps
After remediation, validate in layers.
Validate Lock State
Re-run the relevant lock command.
For VMFS:
vmfsfilelockinfo -p /vmfs/volumes/<datastore>/<vm-folder>/<locked-file>
For vSAN:
vmfsfilelockinfo -p .<object-uuid>.lck
For NFS:
ls -lha
If an NFS .lck file immediately reappears, something still owns the file.
Validate VM Process State
On the expected owner host:
esxcli vm process list | grep -i <vm-name> -B5 -A10 lsof | grep -i <vm-name>
There should not be duplicate VMX processes across hosts. There should not be a backup proxy holding a production disk unexpectedly.
Validate Datastore Health
Check host visibility and APD/PDL indicators again:
esxcli storage filesystem list grep -i "apd\|pdl\|permanently inaccessible\|perm loss" /var/run/log/vmkernel.log grep -i "esx.problem.storage.apd.start\|esx.clear.storage.apd.exit" /var/run/log/vobd.log
If storage is still unstable, stop VM-level remediation and escalate the storage incident.
Validate VM Power-On or Consolidation
Retry the original operation once:
- Power on the VM, if the VM was down.
- Consolidate snapshots, if the issue was consolidation.
- Re-run the failed backup cleanup only after the proxy state is clean.
- Watch the task in vCenter.
- Tail
vmware.logif needed.
tail -f /vmfs/volumes/<datastore>/<vm-folder>/vmware.log
Validate Guest and Application Health
For production workloads, infrastructure validation is not enough.
Confirm:
- Guest OS booted cleanly.
- VMware Tools status is normal.
- Application services started.
- Filesystem or database checks are complete where appropriate.
- Monitoring has cleared.
- Backup jobs are rescheduled or restarted safely.
- Snapshot/consolidation warnings are gone.
Rollback and Fallback Guidance
Some actions have rollback. Some do not.
| Action | Rollback Option | Notes |
|---|---|---|
Moving NFS .lck file to bkup | Move it back if needed | Only for powered-off VM and only after ownership validation |
Restarting hostd / vpxa | No true rollback; wait or restart again if needed | Avoid broad service restart in vSAN, NSX, LACP, or shared graphics environments |
| Removing disk from backup proxy | Reattach only through approved backup/proxy workflow | Never choose Delete from disk for a production VMDK |
| Killing VMX process | No rollback | Hard shutdown; use only after approval |
| Rebooting ESXi host | No rollback; evacuation or outage required | May be required for residual APD/lock conditions |
| Retrying consolidation | Cancel only if safe and supported | Snapshot chain risk increases if interrupted blindly |
Open a support case when:
- Lock ownership is inconsistent across tools.
- vSAN object lock output does not match inventory reality.
- VMFS lock metadata appears abnormal.
- APD/PDL is present or recently occurred.
- Snapshot chains are unclear.
- Consolidation repeatedly fails after lock cleanup.
- The VM is business-critical and the next step is a forced process kill.
KB 314365 includes opening a Broadcom support request as the next step if the problem persists after completing the investigation and remediation steps.
Command Reference
| Task | Command |
|---|---|
| Check VMFS file owner | vmfsfilelockinfo -p /vmfs/volumes/<ds>/<vm>/<file> -v <vcenter> -u <user> |
| List active VM processes | esxcli vm process list |
| Map file to VMX Cartel ID | lsof | egrep 'Cartel|<locked-file>' |
| Find VM task list | vim-cmd vmsvc/get.tasklist <vmid> |
| Inspect task details | vim-cmd vimsvc/task_info <task-id> |
| Inspect NFS lock file | hexdump -C .lck-#### |
| Check vSAN lock file | vmfsfilelockinfo -p .<object-uuid>.lck |
| Restart host agent only | /etc/init.d/hostd restart |
| Restart vCenter agent only | /etc/init.d/vpxa restart |
| Last-resort VMX kill | esxcli vm process kill -t force -w <World-ID> |
| Check APD events | grep -i "esx.problem.storage.apd.start" /var/run/log/vobd.log |
| Check APD recovery events | grep -i "esx.clear.storage.apd.exit" /var/run/log/vobd.log |
Conclusion
A VM file lock incident is not just a command-line problem. It is an ownership problem.
The safest recovery path is to slow down long enough to answer four questions:
- Which file is locked?
- Which host, process, VM, task, or proxy owns the lock?
- Is that ownership expected?
- Is the datastore healthy enough for remediation?
Once you know those answers, the fix is usually straightforward: stop the conflicting backup workflow, detach a hot-added disk, restart a specific management agent, move a stale NFS lock file, reboot an owning host, or, only as a last resort, terminate a VMX process.
The mistake is doing those steps in reverse.
In a VCF or enterprise vSphere environment, the better runbook is not “kill the process and retry.” It is “prove the owner, validate the datastore, remediate the least destructive layer, then prove the VM and application are healthy.”
That is how you recover the workload without making the corruption story worse.
External Sources
- Broadcom KB 314365 — Investigating Virtual Machine file locks on ESXi Host(s)
- Broadcom KB 316576 — Troubleshooting issues resulting from locked virtual disks
- Broadcom KB 326800 — Investigating virtual disk file locks on vSAN
- Broadcom KB 320280 — Restarting Management Agents in ESXi
- Broadcom KB 318712 — Identifying and resolving APD/PDL datastore access issues
- Broadcom KB 372336 — Multiple VM disk consolidation pending and backup proxy retaining locks
There are times when the normal ESXi patching path is exactly what you should use: SDDC Manager, vSphere Lifecycle Manager, a tested...