vCSA 6.0 update 1 Error “503 service Unavailable”

Today I had a call from a customer that could not get into there vCenter.

When they opened the web backup https://vCenter.domain.local they got this message:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f32a40624c0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

This i normal during restart of the vCSA or vCenter server. but i normally disappears after 1-3 minuts, We waited for about 20 minuts and no change, we also tried to reboot the vCSA, same problem.

So I started the troubleshooting with checking if the vCenter server var running from ssh to the vCSA "service-control –status vmware-vpxd" and it was stopped.

When trying to start the service "service-control –start vmware-vpxd" i got a message like this.

INFO:root:Service: vmware-vpxd, Action: start
Service: vmware-vpxd, Action: start
2016-07-19T07:14:05.312Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2016-07-19T07:14:05.362Z   Done running command
2016-07-19T07:14:05.362Z   Running command: ['/sbin/service', u'vmware-vpxd', 'status']
2016-07-19T07:14:05.877Z   Done running command
2016-07-19T07:14:05.878Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-vpxd', 'on']
2016-07-19T07:14:05.928Z   Done running command
2016-07-19T07:14:05.928Z   Running command: ['/sbin/service', u'vmware-vpxd', 'start']
2016-07-19T07:14:08.261Z   Done running command
2016-07-19T07:14:08.261Z   Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']
2016-07-19T07:14:08.261Z   RC = 1
Stdout = Entity: line 137: parser error : Specification mandate value for attribute Key
      <maxQueryMetrics Key>-1</maxQueryMetrics Key>
                          ^
Entity: line 137: parser error : expected '>'
      <maxQueryMetrics Key>-1</maxQueryMetrics Key>
                                               ^
Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/py/xmlcfg.py", line 286, in <module>
    sys.exit( XmlCfg().run() )
  File "/usr/lib/vmware-vpx/py/xmlcfg.py", line 260, in run
    cfgView = ConfigView( filename=self.opts.filename, options={ 'appendMode' : self.opts.append}  )
  File "/usr/lib/vmware-vpx/py/xmlcfg.py", line 35, in __init__
    self.doc = ConfigView.createFileStore(filename)
  File "/usr/lib/vmware-vpx/py/xmlcfg.py", line 46, in createFileStore
    return ConfigViewStore(filename)
  File "/usr/lib/vmware-vpx/py/xmlcfg.py", line 22, in __init__
    self.doc = libxml2.readFd(fd.fileno(), None, None, 0)
  File "/usr/lib64/python2.7/site-packages/libxml2.py", line 1369, in readFd
    if ret is None:raise treeError('xmlReadFd() failed')
libxml2.treeError: xmlReadFd() failed
vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
Waiting for the embedded database to start up: success
Executing pre-startup scripts...
vmware-vpxd: Starting vpxd by administrative request.
success
vmware-vpxd: Waiting for vpxd to start listening for requests on
Waiting for vpxd to initialize: Usage: wait_for_vpxd.py [options]

wait_for_vpxd.py: error: -p option requires an argument
failed
failed
vmware-vpxd: vpxd failed to initialize in time.
vpxd is already starting up. Aborting the request.

Stderr = Entity: line 137: parser error : Specification mandate value for attribute Key
      <maxQueryMetrics Key>-1</maxQueryMetrics Key>

I could see that it was not happy with this "<maxQueryMetrics Key>-1</maxQueryMetrics Key>" so i checked the configuration file for the vCenter Demon: "/etc/vmware-vpx/vpxd.cfg". and found the line i that file, so i checked this file at another customer and could not find this i there configuration, so i decided to remove the lines associated with this setting:

    <stats>
      <maxQueryMetrics Key>-1</maxQueryMetrics Key>
    </stats>

Afterwards the vCenter could start without error, and also after a reboot.

Update:

Someone was trying to do the setting from this VMware KB, but got it wrong. https://kb.vmware.com/kb/2107096