Disable Delayed Ack for iSCSI on ESXi, not always working

Some iSCSI storage system, requires that “Delayed Ack” is disabled on the ESXi hosts, like DELL Equallogic, HPE StoreVirtual and HPE StoreServ.

If this is not done in the correct order this will not work completely as expected.

The KB from VMware says, just to disable it, and reboot, and this will work if you wan not already been running against a iSCSI target already. in that case some of the connections will continue til use Delayed Ack.

See the KB here: https://kb.vmware.com/kb/1002598

To check for this you can use this command on the ESXi host (the host must be in maintenance mode).

vmkiscsid --dump-db | grep Delayed

What you are looking for is ’node.conn[0].iscsi.DelayedAck’=’0’ This men that “Delayed Ack”, ‘0’ means that “Delayed Ack” is disabled.

Solution:

the solution is:

  • Put the host in “maintenance mode”
  • Remove all iSCSI target from the host in dynamic discover and static target, close the window
  • Rescan storage
  • Readd all dynamic discover target
  • Rescan storage

You may need to configurer Round Robin or other path setting after this.

Result:

Here is some screenshots from a test my Colleague did, he added 1 iSCSI target, from a HPE StoreVirtual.

After adding iSCSI taget, and reboot.

After disable “Delayed Ack” and before reboot.

After a reboot, there are still connections with “Delayed Ack” enabled.

After removing the dynamic discover and static target, rescan, readd the iSCSI taget again, it is fixed.

Thanks to my colleague Christian Grønbæk Møller, for doing the this test and solution.

Update:

I have made a sample PowerCLI script that removes all static and dynamic target from a host, rescans all HBAs and adds the dynamic target again. 

Connect-viserver -name <ESXi host>
#Enter maintenance mode
Get-VMHost | Set-VMHost -State Maintenance

#Get iSCSI targets
$iSCSI_targets = Get-IScsiHbaTarget

# Remove All static
foreach ($iSCSI_target in $iSCSI_targets) {
  if ($iSCSI_target.type -eq "Static") {Remove-IScsiHbaTarget $iSCSI_target -confirm:$false}
}

# Remove All dynamic
foreach ($iSCSI_target in $iSCSI_targets) {
  if ($iSCSI_target.type -eq "Send") {Remove-IScsiHbaTarget $iSCSI_target -confirm:$false}
}

#Rescan HBA
Get-VMHostStorage -RescanAllHba

$hba = Get-VMHost | Get-VMHostHba -Type iScsi | where-object {$_.model -eq "iSCSI Software Adapter"}

#Add dynamic target
foreach ($iSCSI_target in $iSCSI_targets) {
  if ($iSCSI_target.type -eq "Send") {new-IScsiHbaTarget -IScsiHba $hba -Address $iSCSI_target.address}
}

#Rescan HBA
Get-VMHostStorage -RescanAllHba

Get-VMHost | Set-VMHost -State Connected

This script does not reboot or add static target.

If you are using dynamic targets, the static targets will reappear automatically. 

Use the script at your risk, so test it before usage.

Notes:

Host profiles can not be used to fix this.

Notes 2:

If you also have hardware iSCSI adaptors on the hosts, you may also have to disable “DelpayedAck” to get all ’node.conn[0].iscsi.DelayedAck’=’0’.

Notes 3:

I found that in some cases i had to disable software iSCSI, delete the database, on the ESXi host. before starting this, put the host in maintenance mode.

Change to the iSCSI initiator configuration directory

cd /etc/vmware/vmkiscsid

From the ESXi command prompt, use the ‘esxcfg-swiscsi’ to disable the iSCSI initiator on your ESXi host

esxcfg-swiscsi -d

Delete the associated iSCSI initiator configuration file

rm -f vmkiscsid.db

That’s it.

After reboot you will have a ESXi host with a clean iSCSI configuration. Now you can go ahead and reconfigure the iSCSI initiator on your ESXi host and add any targets. remember to change “DelayedAck” before adding targets.

Remember that the information about Hardware iSCSI also is in the database, see “Notes 2”.

Please share this page if you find it usefull:

5 thoughts on “Disable Delayed Ack for iSCSI on ESXi, not always working

  1. For others utilizing this code, be certain not to be attached to your entire vcenter. It’s best to be attached to only your single host in maintenance mode.

  2. When I run the “vmkiscsid –dump-db | grep Delayed” command on my host it returns about 102 lines of results. Most are DelayedAck=0 but about 10 are DelayedAck=1. How do I correlate the lines with a 1 result to my iSCSI setup so I can know what still is trying to use DelayedAck?

    1. You can just browse thue the complete out put of -vmkiscsid –dump-db | less’, and analyse the result.

      If you run this ‘vmkiscsid –dump-db | grep “Delayed\|======”‘ you can see which section you see the “DelayedAck=1” comes from, that will narrow it down.

  3. This plagued me for far too long. THANKS for your post. It has gone a long ways to ameliorating poor performance.

  4. This totally just saved the day. We had serious latency issues after migrating a SAN to a new data center and have been trouble shooting the issue for two weeks with vmware and emc. Host reboots would take 6 hours and storage scans would timeout everything.

    The vmware kb article was total rubbish and didn’t disabled delayed ack properly.

    The “Notes 3” fix is what did the truck for us. Many thank you’s!!

Leave a Reply to _nd3456789 Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.