after several hours of horrible performance, identified and resolved an issue with VMware esxi 5.1 and our SAN infrastructure.
Solution:
Identify VMware hosts with "SCSI Target" devices having WWN of 00:00:00:00:00:00:00:00 under [Storage View][Maps].
Remove this "SCSI Target" devices by putting host in maintenance mode, rebooting, and going to [Configuration][Storage Adapters] to execute a "Rescan All...".
Events:
Had scheduled UPS maintenance for our 2 datacenters.
Powered down server room for UPS maintenance.
Powered up UPS, Powered up SAN, Powered up Servers.
VMs with disks on one SAN had horrible performance.
linux's sar iowait showed 60-80%, when it should be 0.1%.
VMware hosts performance showed disk latency of 1500-2000, when it should be 5 with occasional spikes of 50.
SAN performance graphs showed 70,000MB/s reads and 50,000MB/s writes for ALL VMware LUNs on one SAN(normal usage around 100MB/s for the entire SAN).
SAN switch performance graphs showed normal usage, highest ports using 200-500Mb/s(not even close to 70,000MB/s).
Other servers(solaris,redhat,suse,etc) showed normal performance to same SAN.
vmotion'ing VMs to the other datacenter often failed, if successful performance still was horrible.
vstorage motion'ing VMs to the other SAN often failed, but if successful performance was back to normal.
Lots of troubleshooting was done over 2 days and multiple shifts of people, so i can't list everything that was attempted.
The above solution was found.
As soon as the last VMware host with the problematic "SCSI Target" was corrected, performance immediately went to normal.
Sharing here in case someone has similar issues.