I have two hosts, a BL480G1 and a DL360G5 using a NexentaStor iSCSI LUN.
The BL480G1 has always had storage issues, ranging from high latency to things that are as critical as this.
When removing snapshots, the snapshots take HOURS (we're talking 8+) to remove, if I take a snapshot and immediately remove it, it takes that long, making backups nearly impossible, during this time the VM will stop responding for up to 15 minutes at a time.
My vmkernel.log:
2012-12-24T05:18:12.416Z cpu4:2177)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x89 (0x41240077bb40, 10724) to dev "naa.600144f0bd9c020000004ff1a959000c" on path "vmhba32:C1:T0:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-12-24T05:18:12.416Z cpu4:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:12.416Z cpu4:2177)ScsiDeviceIO: 2322: Cmd(0x41240077bb40) 0x89, CmdSN 0xa432 from world 10724 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:15.506Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:15.506Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x41240073a380) 0x89, CmdSN 0xa439 from world 10722 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:18.898Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:18.898Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x412400782280) 0x89, CmdSN 0xa43e from world 10689 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2012-12-24T05:18:20.001Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:20.001Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x4124007ae3c0) 0x89, CmdSN 0xa42c from world 3720 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:22.029Z cpu1:2177)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x89 (0x412400781580, 10727) to dev "naa.600144f0bd9c020000004ff1a959000c" on path "vmhba32:C4:T0:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0. Act:EVAL
2012-12-24T05:18:22.029Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:22.029Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x412400781580) 0x89, CmdSN 0xa444 from world 10727 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2012-12-24T05:18:25.082Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:25.082Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x412400791040) 0x89, CmdSN 0xa45c from world 10724 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:28.608Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:28.608Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x41240072d680) 0x89, CmdSN 0xa462 from world 10726 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:31.705Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:31.705Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x4124007b5340) 0x89, CmdSN 0xa466 from world 10724 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2012-12-24T05:18:33.080Z cpu1:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:33.080Z cpu1:2177)ScsiDeviceIO: 2322: Cmd(0x4124007abfc0) 0x89, CmdSN 0xa453 from world 3720 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:34.736Z cpu5:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:34.736Z cpu5:2177)ScsiDeviceIO: 2322: Cmd(0x41240072ae80) 0x89, CmdSN 0xa46a from world 10722 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2012-12-24T05:18:37.778Z cpu4:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:37.778Z cpu4:2177)ScsiDeviceIO: 2322: Cmd(0x41240072b280) 0x89, CmdSN 0xa47d from world 10689 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:40.868Z cpu4:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:40.868Z cpu4:2177)ScsiDeviceIO: 2322: Cmd(0x41240072ed80) 0x89, CmdSN 0xa482 from world 10727 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:43.882Z cpu4:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:43.882Z cpu4:2177)ScsiDeviceIO: 2322: Cmd(0x412400763980) 0x89, CmdSN 0xa487 from world 10726 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2012-12-24T05:18:46.118Z cpu2:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:46.118Z cpu2:2177)ScsiDeviceIO: 2322: Cmd(0x41240073bf80) 0x89, CmdSN 0xa479 from world 3720 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:46.985Z cpu2:2177)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x89 (0x4124007bb6c0, 2110) to dev "naa.600144f0bd9c020000004ff1a959000c" on path "vmhba32:C2:T0:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-12-24T05:18:46.985Z cpu2:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:46.985Z cpu2:2177)ScsiDeviceIO: 2322: Cmd(0x4124007bb6c0) 0x89, CmdSN 0xa48b from world 2110 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:50.104Z cpu2:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:50.104Z cpu2:2177)ScsiDeviceIO: 2322: Cmd(0x412400734a00) 0x89, CmdSN 0xa4a1 from world 10722 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-12-24T05:18:53.120Z cpu2:2177)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x89 (0x412400783680, 10724) to dev "naa.600144f0bd9c020000004ff1a959000c" on path "vmhba32:C2:T0:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-12-24T05:18:53.120Z cpu2:2177)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f0bd9c020000004ff1a959000c" state in doubt; requested fast path state update...
2012-12-24T05:18:53.120Z cpu2:2177)ScsiDeviceIO: 2322: Cmd(0x412400783680) 0x89, CmdSN 0xa4a8 from world 10724 to dev "naa.600144f0bd9c020000004ff1a959000c" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
----
I've tried to replicate this on the DL360G5, snapshots add and remove fine (takes less than a minute, I may lose one or two ping packets during it, the SAN is under a bit of resource contention). The BL480G1 is using a GB2EC managed switch, everything has jumboframes enabled (outside of the GB2EC, which is said to "automatically" support jumboframes), tried both with multipath and without.
ANY ideas?