vCSA /storage/log Running Full
I have been trouble shooting a problem with the /storage/log running full, for a customer.
The problem was that the file /storage/log/vmware/vmware-sps/sps-runtime.log.stderr file was growing very large.
I the log file I found the the vmware-sps (Storage Profile Service) had some problems. The errors looks like this
May 31, 2021 7:18:02 AM org.bouncycastle.jsse.provider.ProvTlsClient notifyAlertRaised INFO: Client raised fatal(2) certificate_unknown(46) alert: Failed to read record org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46) Caused by: java.security.cert.CertPathBuilderException: Certification path could not be validated. Caused by: java.security.cert.CertificateException: Unable to construct a valid chain Caused by: java.security.cert.CertPathValidatorException: Could not validate certificate: certificate expired on 20200917181137GMT+00:00
The last one I did a filter on and could see that the date was different, so that indicated that it was communicating with multiple endpoint, which must be the ESXi server.
So I took a look at the some hosts in the vCenter and found that some of the had expired certificate. I tried to update the certificate from vCenter, but that did not work, and that was because I earlier had used this blog: vCenter 6.7 Update 3, Unable to add Host So I had to set the setting back to the original setting of “vmca” and then I could change the certificates on the hosts.
vpxd.certmgmt.mode = vmca
After all hosts with expired certificates, had them renewed the file stopped growing.
UPDATED June 7, 2021:
There above was only one part of the problem, there was another problem also.
The hosts that was add, when the vCenter was using “thumprint” instead of “vmca” was using a custom certificate that the vmware-sps service could validate, so the also filled up the log file.
To find which hosts certificate was expired or custom, I used the a script from William Lam that can be found thru this blog: Is vCenter Server & ESXi hosts using VMware Certificate Authority (VMCA) or custom CA certificates?
The result of the script looked like this.
To fix the problem I just did a “Renew Certificate” and “Refresh CA Certticates”, on all the hosts with expired and “custom” certificates.