First thing first, here is the background information.
1. vSphere has been upgraded from 4.1 to 5.0 update 2 and the problem started. We use Symantec Netbackup 7.5 with NBD transport mode.
2. We have mixed storage of HDS (VMFS) and Netapp (NFS). So far the issue only exhibits on Netapp NFS datastores. Most of the VMs being backed up are running on netapp too. Don't know if it helps but it is better to have more information.
3. We have large amount of VMs (400+) being backed up nightly, most of them are incremental backup. Before backup start, Netbackup calls vCenter to create a snapshot. After backup completes, Netbackup again calls vCenter to remove snapshot.
The problem we have is sometimes random VMs would have delta files (vmname-000001.vmdk) left behind in the VM folder. It is not used by vmx file which means it is completed orphaned so I can remove it manually. And manual removal seems to be the only way to get rid off the orphaned delta files (I am still searching for a script to identify and remove such files. I am grateful If you can share). I tried create a new snapshot then delete all, I also tried "consolidate" option in snapshot manage. Neither of them helped.
The reason why I know it is happening on random VMs is we have a script to search through each datastore for delta files (trust me regular script to search snapshot doesn't work). And every day we have different affected VMs. And funny thing is normally the orphaned delta files are only 16MB or 17MB.
In the screenshot above, the delta file vmname-00002.vmdk is the orphaned file I am talking about. Looking at the vmx file, it is pointing to the base/flat file which is normal. Because the snapshot is considered as "deleted/cleaned". We know this by looking at the vCneter history.
I have created a ticket to VMware support but was told Symantec is needed because we or VMware supoport can't manually reproduce the issue in house. They believe something is wrong while netbackup calls the API. Symantec support is engaged too but they don't seem to have a clue either.
I am wondering if anyone knows where else to look.