vCSA Upgrade to 6.7 Failure

Yesterday I was upgrading a vCSA 6.0 to 6.7 Update 2, I ran into a strange failure.

I was using a JSON file for the upgrade, because I better can choose the size and, it’s easier to do multiple tries, and don’t have to go thru the wizard every time.

When i ran the upgrade i failed in the precheck on the source with this strange error:

Cannot detect upgrade runner version on source vc, proceed to cleanup and upload
Uninstalling vmware-upgrade-requirements-* on vcenter.domain.local.
('Failed to uninstall %(rpm)s on %(sourcevc_hostname)s: %(error)s', {'error':
CommandError("Failed to run and wait for command in guest with error 'Command
'/bin/rpm' exited with non-zero status 1'",), 'sourcevc_hostname':
'vcenter.domain.local', 'rpm': 'vmware-upgrade-requirements-*'})
Failed to run and wait for command in guest with error 'Command '['ls',
'/var/tmp/vmware-upgrade-requirements/checker.sh']' exited with non-zero status
2'
Uploading rpm to /var/tmp/upgrade-requirements.rpm, on VM
vcenter.domain.local.

I looked thru the logs but I did not find any use full clue, on the failure.

But after some investigation I found the on the ESXi host where the source vCenter was running the /tmp folder was full:

[root@esxi01:/tmp] vdf -h
......
-----
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        2M       29M   9% --
etc                        28M      300K       27M   1% --
opt                        32M      544K       31M   1% --
var                        48M        1M       46M   2% --
tmp                       256M      256M        0B 100% --
iofilters                  32M        0B       32M   0% --
hostdstats               1803M        9M     1793M   0% --

I found that i was a HPE module (AMS) that was filling the /tmp folder, there is a HPE advisory on this here: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00073323en_us&docLocale=en_US.

So after deleting the /tmp/ ams-bbUsg.txt” file the Upgrade work fine.

Update Juli 31 2019: Just seen the same failure message, but this time, i could see some log files, but not anything use full.

cd /storage/log/vmware/upgrade
find | grep 2019
./CollectRequirements_com.vmware.vmon_2019_07_31_09_41.log
./CollectRequirements_com.vmware.is_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vpxd_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vsan-health_2019_07_31_09_41.log
./CollectRequirements_com.vmware.sps_2019_07_31_09_41.log
./CollectRequirements_com.vmware.license_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vcdb_2019_07_31_09_41.log
./CollectRequirements_com.vmware.rbd_2019_07_31_09_41.log
./CollectRequirements_com.vmware.rhttpproxy_2019_07_31_09_41.log
./CollectRequirements_com.vmware.applmgmt_2019_07_31_09_41.log
./CollectRequirements_com.vmware.netdump_2019_07_31_09_41.log
./CollectRequirements_com.vmware.sso_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vcha_2019_07_31_09_41.log
./CollectRequirements_com.vmware.syslog_2019_07_31_09_41.log
./CollectRequirements_com.vmware.cls_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vcIntegrity_2019_07_31_09_41.log
./CollectRequirements_com.vmware.vmafd_2019_07_31_09_41.log
./CollectRequirements_com.vmware.common_upgrade_2019_07_31_09_41.log
./CollectRequirements_com.vmware.ngc_2019_07_31_09_41.log

The solution was to reboot the source vCenter, and the upgrade worked.

Share This:

Leave a Reply

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.