Eyeglass Connect Probe Vblock Technical Support Playbook

Eyeglass Connect Probe Vblock Technical Support Playbook





Eyeglass Connect Probe Vblock Capabilities and Components Monitored

Eyeglass Connect Probe Vblock Probes for CA UIM leverage VCE Vision and VMware vCenterTM software to provide a single integration point for VCE Vblock management using the CA UIM Monitor Server and the UIM Unified Management Portal.  CA UIM with Eyeglass Connect Probe Vblock offers visualization of all Vblock ™ components and virtual machines with real-time health status, alarm aggregation and component alarm tagging:


Eyeglass Connect Probe Vblock_Monitor: This probe provides inventory, state and alarm collection from Vblock and vCenter.


Eyeglass Connect Probe Vblock_Render: This webapp provides the Eyeglass Connect Vblock Probe Dashboard Web Page for Vblock and Virtual Machine visualization.


This subject of this document is the Eyeglass Connect Probe Vblock_Monitor probe and the Eyeglass Connect Probe Vblock_Render webapp.


The primary capabilities of the Eyeglass Connect Probe Vblock_Monitor probe are to:

  • discover topology for a VCE Vblock as provided by VCE Vision IO

  • discover Virtual Machines running on the Vblock as provided by vCenter

  • associate the Virtual Machines to the physical blade where they are running on the Vblock

  • discover properties of the VCE Vblock components as provided by VCE Vision IO

  • discover basic properties of the Virtual Machines as provided by vCenter

  • bring into UIM the alarms and component status for the Vblock components as provided by VCE Vision IO

  • bring into UIM the alarms and status for the Virtual Machines running on the Vblock as provided by vCenter

  • discover Vblock Compliance to RCM


The SupernaNET.Monitor probe monitors the following components:


Component

Version

VCE Vblock running Vision IO

Vision IO R2.1

vCenter

vCenter R5.1 and higher



The primary capabilities of the Eyeglass Connect Probe Vblock_Render webapp are to provide following capabilities on the Eyeglass Connect Vblock Probe Dashboard:

  • Converged View of virtual and physical topology for Vblock components and their associated Virtual Machines

  • Root Cause of problems by Domain or Component

  • Health Check by Domain utilizing VCE Vision computed status

  • Intelligent Alarms include subsystem id and Vblock serial number tagging using Nimsoft component ID tags

  • Component Zoom for component properties including location, serial number and version information

  • Vblock RCM Compliance score


Overview of Architecture


Eyeglass Connect Probe Vblock_Monitor is a connector between Vblock and vCenter and Nimsoft.  It collects the inventory and state information from these devices and passes it to the Eyeglass Connect Vblock Probe Dashboard.  It also collects alarm information from these devices and passes it to Nimsoft.


Eyeglass Connect Probe Vblock_Render is a webapp that runs on the UIM wasp web server.  As part of the deployment of the Eyeglass Connect Probe Vblock probes, the Eyeglass Connect Vblock Probe Dashboard web page must be configured on the UIM interface and associated to the Eyeglass Connect Probe Vblock_Render webapp.  The Eyeglass Connect Probe Vblock_Render webapp has a dependency on the webservices-rest webapp having been installed and running on the same wasp probe where the Eyeglass Connect Probe Vblock_Render webapp is running.


The data flow between Eyeglass Connect Probe Vblock_Render, Nimsoft and Eyeglass Connect Probe Vblock_Monitor and the related protocols are illustrated in the figure below.



SupernaNET.Monitor Data Flow_2 (3).png


Description of Main Monitor Probe Tasks:


Task

Description

Vblock Inventory and Calculated (Calc) Status

  • Protocol: REST

  • Scheduled retrieval of information every 30 minutes on the hour/half-hour as per SupernaNET Monitor probe server time

Vblock Compliance

  • Protocol: REST

  • Scheduled retrieval of information every 30 minutes on the hour/half-hour as per SupernaNET Monitor probe server time

Vblock Alarms

  • Protocol: AMQP

  • Permanent AMQP connection established between SupernaNET.Converge Monitor probe and the AMQP on initial discovery of the Vblock

  • Alarms processed as they are generated

vCenter Inventory

  • Protocol: SOAP

  • Scheduled retrieval of information every 30 minutes on the hour/half-hour as per SupernaNET Monitor probe server time

vCenter Alarms

  • Protocol: SOAP

  • Polling for alarms at 1 minute interval



Configuration and Log Files

Eyeglass Connect Probe Vblock_Monitor

Configuration Items


Configuration files are located in the C:\Program Files (x86)\Nimsoft\probes\application\supernanet.converge.monitor\data directory.  Following files have primary function in configuration of running SupernaNET.Converge.Monitor probe:



File

Description

nedata.xml

File containing vblock/vcenter pairs that are being monitored:

  • IP address

  • username/password

Note: Password is encrypted in the file.

system.xml

File containing parameters used by probe on initial startup.  Following default values are expected:

  • <websocketUrl>ws://localhost:2012</websocketUrl>

  •  <apptype>Probe</apptype>

sync.xml

File containing schedule for collecting data from Vision and vCenter.  Following default values are expected

  • <InventoryCollection IsConfigurable="true" Label="Discovery">30M</InventoryCollection>




Log Files


Log files are located in the C:\Program Files (x86)\Nimsoft\probes\application\supernanet.converge.monitor\ogs directory.  Following log files contain the operational logging information for the SupernaNET.Converge.Monitor probe:



File

Description

main.log

File containing logs related to:

  • Starting probe

  • Stopping probe

  • Add/Delete Vblock/vCenter

  • Inventory collection for VCE Vision and vCenter

  • Alarm processing for VCE Vision and vCenter

This log rolls over when it reaches 4 MB and a total of 10 backup log files are stored.


Example log entry for Starting probe:


2013-09-26 15:58:56,544 NDEDaemon:start [272] ***** Starting Probe [ 12@CSS-3 ] *****

2013-09-26 15:58:56,560 NDEDaemon:<init> [46] Initializing Components...

2013-09-26 15:58:56,560 NDEDaemon:<init> [48] Server

2013-09-26 15:58:56,560 NDEDaemon:<init> [51] License Management

2013-09-26 15:58:56,575 NDELicenseManager:initialise [273] Running in Daemon Mode - starting scheduler.

2013-09-26 15:58:56,969 NDELicenseManager:populateLicenses [221] Added Base Application license to instance.

2013-09-26 15:58:56,993 NDELicenseManager:populateLicenses [221] Added NE Discovery license to instance.

2013-09-26 15:58:56,997 NDEDaemon:<init> [55] Key Management

2013-09-26 15:58:57,001 NDEDaemon:<init> [58] Alarm Handler

2013-09-26 15:58:57,069 NDEDaemon:<init> [62] Scheduler

2013-09-26 15:58:57,090 NDEDaemon:<init> [65] Plugin Server

2013-09-26 15:58:57,095 PluginDataIO:getPluginList [19] reading plugindata IO from file

2013-09-26 15:58:57,191 PluginDataIO:getPluginList [28] Successfully unmarshalled plugin file

2013-09-26 15:58:57,195 NDEDaemon:readConfigData [70] Sites

2013-09-26 15:58:57,196 NDEDaemon:readConfigData [74] Syslog

2013-09-26 15:58:57,197 RefreshSyslogData:execute [15] Refreshing Syslog Data...

2013-09-26 15:58:57,198 RefreshSyslogData:execute [33] Syslog Data Refreshed

2013-09-26 15:58:57,198 RefreshSyslogData:execute [36] Syslog Settings Updated

2013-09-26 15:58:57,199 NDEDaemon:startDaemon [191] Starting Components...

2013-09-26 15:58:57,199 NDEDaemon:startDaemon [194] Probe

2013-09-26 15:58:57,199 NDEDaemon:startDaemon [194] Tomcat

2013-09-26 15:58:57,200 NDEDaemon:startDaemon [194] Server

2013-09-26 15:58:57,200 NDEDaemon:startDaemon [194] Plugin Server

2013-09-26 15:58:57,200 NDEDaemon:startDaemon [194] NE Refresh

2013-09-26 15:58:57,202 RefreshNEData:execute [47] Refreshing NE Data... Acquired lock on task.

2013-09-26 15:58:57,220 Fdo:<init> [152] Using web service at - WEB SERVICE DISABLED

2013-09-26 15:58:57,255 RefreshNEData:doDataRefresh [103] Web service is disabled, using NE file data

2013-09-26 15:58:57,256 RefreshNEData:doDataRefresh [114] Searching new NE...

2013-09-26 15:58:57,269 RefreshNEData:doDataRefresh [189] Syslog Schedule Updated

2013-09-26 15:58:57,269 RefreshNEData:execute [51] NE Data Refreshed.. Releasing lock on task.

2013-09-26 15:58:57,270 NDEDaemon:startDaemon [194] Scheduler

2013-09-26 15:58:57,208 NDETomcat6x:startTomcat [119] startTomcat: port8181

2013-09-26 15:58:57,204 NDEServer:run [127] Waiting for a new connection

2013-09-26 15:58:57,204 PluginServer:run [56] Waiting for a new plugin data request

2013-09-26 15:58:57,419 NEContactTask:execute [301] Starting NEContactTask.  Acquired task lock.

2013-09-26 15:58:57,418 CircuitSyncScheduler:start [232] Circuit sync scheduler start

2013-09-26 15:58:57,429 SyncCircuitData:execute [24] Executing circuit sync

2013-09-26 15:58:57,430 SyncCircuitData:execute [39] Web service is disabled.

2013-09-26 15:58:57,398 NDETomcat6x:registerWAR [235] Added context path - /NDEWebService

2013-09-26 15:58:57,227 VblockProbe:<init> [100] Starting probe

2013-09-26 15:58:57,483 VblockProbe:runProbe [119] Initializing Nimsoft Probe...



Example log entry for stopping probe


TBD


Example log entry for adding Vblock/vCenter


2013-09-26 16:24:00,009 NDELicenseManager:networkChanged [1065] NODE ADDED: Licensed NE 'vblock_10.10.10.10' for 'NE Discovery'

2013-09-26 16:24:00,009 NDELicenseManager:networkChanged [1065] NODE ADDED: Licensed NE 'vcenter_10.10.10.20' for 'NE Discovery'

2013-09-26 16:24:00,015 RealTimePMTask:realTimePMTask [26] Starting NE Real Time PM-s action

2013-09-26 16:24:00,017 RetrieveDataTask:retrieveDataNE [22] Starting NE Sync action

2013-09-26 16:24:00,017 NDEUpdateTask:updateStatus [115] Getting Rack Data



Example log entry for deleting Vblock/vCenter


2013-09-26 20:16:00,219 RefreshNEData:disconnectNode [355] vblock_10.10.10.10 was not connected

2013-09-26 20:16:00,228 NDELicenseManager:networkChanged [1080] NODE REMOVED: Unlicensed NE 'vblock_10.10.10.10' for 'NE Discovery'

2013-09-26 20:16:00,237 RefreshNEData:disconnectNode [359] vblock_10.10.10.10 Disconnected

2013-09-26 20:16:00,245 RefreshNEData:disconnectNode [355] vcenter_10.10.10.25 was not connected

2013-09-26 20:16:00,254 NDELicenseManager:networkChanged [1080] NODE REMOVED: Unlicensed NE 'vcenter_10.10.10.25' for 'NE Discovery'

2013-09-26 20:16:00,262 RefreshNEData:disconnectNode [359] vcenter_10.10.10.25 Disconnected



Example log entry for inventory collection start


2013-09-26 19:30:00,002 NDEUpdateTask:run [215] Starting NDEUpdateTask



Example log entry for alarm processing


2013-09-26 19:28:54,966 ProbeAlarmHandler:handleAlarms [46] Sending alarm to Nimsoft from 10.10.10.25

2013-09-26 19:28:54,978 ProbeAlarmHandler:handleAlarms [47] Alarm Parameters: 10.10.10.10 Vision-sim-2 4 THis is a test alarm.

2013-09-26 19:28:54,992 ProbeAlarmHandler:handleAlarms [50] Sent alarm with id: AX90516910-42360




Debugging Facilities


None available.


Eyeglass Connect Probe Vblock_Render

Configuration Items

There are no Eyeglass Connect Probe Vblock_Render specific configuration items.  

The Eyeglass Connect Vblock Probe Dashboard is configured and managed using Nimsoft UMP Manage Pages.


Log Files

There are no Eyeglass Connect Probe Vblock_Render specific log files.


Debugging Facilities


None available.



How to Obtain Licenses


Contact Superna to obtain licenses.  The following information is required:


License Type: Trial or Permanent


For Permanent Licenses:


License Quantity: Number of Vblocks plus vCenter licensed for monitoring


Trial Licenses will be issues with a quantity 2 for monitoring one Vblock/vCenter pair.



Troubleshooting


Vblock Inventory Not Collected

Setup:

  • SupernaNET.Converge.Monitor probe deployed to UIM robot with Vblock/Vision connectivity

  • SupernaNET.Converge.Render probe deployed to UIM wasp probe

  • SupernaNET.Converge licenses deployed

  • Vblock/Vision and Vcenter provisioned on SupernaNET.Converge

Problem:

In the Eyeglass Connect Vblock Probe Dashboard, a Vblock that was added is permanently displayed with a spinner and the associated inventory tree is not populated.   



Troubleshooting:

1.  Refresh the Eyeglass Connect Vblock Probe Dashboard page.  Dashboard may not have been updated after successful completion of inventory.

2.  Determine elapsed time since Vblock was added.  Vblock inventory may take 30 - 90 minutes depending on the state of the system when the Vblock was added:

  • Was inventory collection task in progress when Vblock was added?  The in-progress inventory task has to complete before inventory for a new vblock is added.

  • How many Vblock inventories are being completed simultaneously.

3.  Look in main.log and determine whether there was an inventory failure.  You will see this almost immediately after NDEUpdate task starts with an entry in main.log that looks something like this:

2013-11-04 15:39:14,898 NDEUpdateTask:collectData [211] Failed to get Inventory Data.  
2013-11-04 15:39:14,898 RefreshNEData:doDataRefresh [183] Could not sync inventory for new nodes - Failed to get Shelf Data - Inventory failed

In main.log locate the log entry where the node was added. If you see


Server supports System Library false


there is a problem with connectivity to Vision.  This log most often indicates a DNS problem where it is not possible to resolve the Vision IP address to its FQDN.


Example main.log:


2013-11-04 15:48:00,011 NDELicenseManager:networkChanged [1065] NODE ADDED: Licensed NE 'vblock_1.1.1.1' for 'NE Discovery'
2013-11-04 15:48:00,011 NDELicenseManager:networkChanged [1065] NODE ADDED: Licensed NE 'vcenter_2.2.2.2' for 'NE Discovery'
2013-11-04 15:48:00,014 RealTimePMTask:realTimePMTask [26] Starting NE Real Time PM-s action
2013-11-04 15:48:00,015 RetrieveDataTask:retrieveDataNE [22] Starting NE Sync action
2013-11-04 15:48:00,015 NDEUpdateTask:collectData [158] Running inventory for nodes:  
2013-11-04 15:48:00,015 NDEUpdateTask:collectData [160]           vblock_1.1.1.1 1.1.1.1
2013-11-04 15:48:00,016 NDEUpdateTask:collectData [160]           vcenter_2.2.2.2 2.2.2.2
2013-11-04 15:48:00,016 NDEUpdateTask:updateStatus [123] Getting Compliance Data
2013-11-04 15:48:02,052 VBlockXXX:connect [57] Successful connection to vblockvision.nsit-lab.com
2013-11-04 15:48:02,052 VBlockXXX:connect [58] Server supports System Library false Ver:Not a Vision OS System Library server
2013-11-04 15:48:02,052 VBlockXXX:connect [59] Server supports Compliance  true Ver:2.0.0
2013-11-04 15:48:04,026 ComplianceScore:runComplianceScore [264] Successful connection to vblockvision.nsit-lab.com
2013-11-04 15:48:04,026 ComplianceScore:runComplianceScore [265] Server supports System Library false Ver:Not a Vision OS System Library server
2013-11-04 15:48:04,027 ComplianceScore:runComplianceScore [266] Server supports Compliance  true Ver:2.0.0


Troubleshooting Steps for Vision Connectivity Issue:

1. Confirm that there is an entry in the C:\Windows\System32\drivers\etc\hosts file for the Vision IP and FQDN ON THE SERVER WHERE SUPERNANET.CONVERGE.MONITOR is running. For example, an entry such as:

10.10.10.10 fm-nimsoft.internal.company.net

If it is not in the host file - add it. Next inventory should pick it up.


2. On the server where SupernaNET.Converge.Monitor is running execute following command

nslookup <Vision FQDN>

This should return the IP address of the Vision provisioned in SupernaNET.Converge if the host name and DNS are properly configured.


3. On the server where SupernaNET.Converge.Monitor is running open a browser and enter the URL:

https://10.10.10.10:8443/fm

where 10.10.10.10 is replaced with IP address provisioned in SupernaNET.Converge for Vision


You should be redirected from IP address to the Vision FQDN and then after entering credentials (default is admin/dangerous) see browser return like this:

<slib>

<link href="https://fm-nimsoft.internal.company.net:8443/fm/vblocks" rel="children" type="application/xml"/>

<link href="https://fm-nimsoft.internal.company.net:8443/fm/about" rel="children" type="application/xml"/>

</slib>


Go to each of the links provided. You should pages like this:

https://fm-nimsoft.internal.company.net:8443/fm/vblocks

<vblocks>
<vblock>
<alias>VB-700</alias>
<dbId>843</dbId>
<description>VB-700 (Vblock 700LX)</description>
<moId>VB700-975-318-642</moId>
<name/>
<url>
https://vb701-viofm.vce.company.net:8443/fm/vblocks/843
</url>
<assetId/>
<calculatedStatus>critical</calculatedStatus>
<firmwareRev/>
<fru>false</fru>
<hardwareRev/>
<lastOperStatus>unknown</lastOperStatus>
<lastStatusChange>1970-01-01T00:00:00Z</lastStatusChange>
<localInstanceId/>
<localName/>
<mfgDate>1970-01-01T00:00:00Z</mfgDate>
<modelName>Vblock 700LX</modelName>
<operStatus>critical</operStatus>
<phyClass/>
<serialNum>VB700-975-318-642</serialNum>
<softwareRev/>
<vendor>The VCE Company, Inc.</vendor>
<vendorType>1.3.6.1.4.1.38819.1.5.10.2.700.2</vendorType>
<link href="https://vb701-viofm.vce.superna.net:8443/fm/vblocks/843/vlans" rel="children" type="application/xml"/>
<lastUpdateTime>2013-11-06T18:38:33.847Z</lastUpdateTime>
<location>
<building>350 Campus</building>
<floor>1rd</floor>
<geo>Marlborough, MA</geo>
<room>Eisenhower</

https://fm-nimsoft.internal.company.net:8443/fm/about

<appinfo>
  <buildVersion>2.0.3</buildVersion>
  <dbId>1</dbId>
  <productName>
      VCE Vision(TM) Intelligent Operations System Library
  </productName>
</appinfo>


If these URLs don't work you either still have a DNS problem or a Vision problem.  Contact the Network Administration for the Nimsoft and Vision installations for further troubleshooting.



Unable to Load Licenses Troubleshooting

Assumptions:

  • SupernaNET.Converge.Monitor successfully installed and running

  • SupernaENT.Converge.Render successfully installed

  • license.zip file has been created for the install

Problem:

Try to upload the license zip file from the SupernaNET.Converge Dashboard and nothing happens.


Troubleshooting:

1. Browser

License upload only works with Chrome browser.



Authentication Error when accessing the Eyeglass Connect Vblock Probe Dashboard

Problem:

When you open the Eyeglass Connect Vblock Probe Dashboard, a popup window appears which indicates that there is an authentication error and is requesting credentials.  You enter the credentials that you used to login to Nimsoft and the same authentication error window appears again each time that credentials are entered.  You may also see error popups related to retrieving alarms or hub information.


Troubleshooting:

1. User permissions

The user who is logged in to Nimsoft must have the “Web Service” option selected on his Access Control List.  


To check this setting:

1.  Log in to Nimsoft Infrastructure Manager.

2.  Select Security/Manage Access Control List.

3.  Select your user from the Access Control List.

4.  Ensure that in the Permissions list, the “Web Service” option is selected.






No Vblocks displayed

Problem:

When you open the Eyeglass Connect Vblock Probe Dashboard, the Vblock Inventory view is empty.


Troubleshooting:

1. Logon to the SupernaNET.Converge.Monitor server and check that configuration and data files are populated with the Vision/vCenter pairs that have been provisioned on the system:

 C:\Program Files (x86)\Nimsoft\probes\application\supernanet.converge.monitor\data\nedata.xml



2.  If Vision/vCenter information was found in Step 1, make sure that the webservices rest webapp is running.  

To check the webservices webapp:

1.  Log in to Nimsoft Infrastructure Manager.

2. Locate the wasp probe.

3. Right click on the wasp probe and select the Configure… option.

4. Find the webapp called “webservices-rest”.  It should have a green icon indicating that it is running.  If it is not running it should be started.  If it is not present it needs to be deployed.




3. If Vision/vCenter information is not found in Step 1, contact Superna for further assistance.


No Vblock Alarms in Dashboard or Nimsoft Alarms windows

Problem: No Vblock or vcenter alarms in the Superna Dashboard or Nimsoft Alarms window.


Troubleshooting:

1.  Check in main.log for AMQP entries on probe startup.  For example:



2014-06-05 07:10:00,097 [Reachability-vblock_172.16.70.41] VBlockXXX:startListener [81] Begin AMQPConsumer, now=

2014-06-05 07:10:01,225 [Thread-121] AMQPConsumer:setupAMQPConsumer [117] ST-4323-d3q9xQ15HSYzK4dZU96b-cas01.vce.com




2. Check in main.log for alarm processing entries.  Example of log entry when monitor probe receives and alarm:



2014-06-05 07:22:53,156 [pool-7-thread-2] AMQPConsumer$1:handleDelivery [156] got real event



Example of log entry when monitor probe passes the alarm to Nimsoft:



2014-06-05 07:22:53,290 [Thread-277] ProbeAlarmHandler:AlarmData2NimAlarm [142] ....sending an alarm from FM  .. vblock_ip = 172.16.70.41

2014-06-05 07:22:53,302 [Thread-277] ProbeAlarmHandler:AlarmData2NimAlarm [169] Trying to create a CI obj with: vblock_ip=172.16.70.41

2014-06-05 07:22:53,314 [Thread-277] ProbeAlarmHandler:AlarmData2NimAlarm [171] CREATED a CI obj with: vblock_ip=172.16.70.41

2014-06-05 07:22:53,326 [Thread-277] ProbeAlarmHandler:handleAlarms [53] Sending alarm to Nimsoft from 172.16.70.41

2014-06-05 07:22:53,341 [Thread-277] ProbeAlarmHandler:handleAlarms [54] Alarm Parameters: 172.16.70.41 VB700-975-318-642 1 Compliance scan of benchmark 'VCE Vblock(TM) System 700 Release Certification Matrix' at profile 'VCE Vblock(TM) System 700 Release 3.0.9 Certification Matrix' completed with a score of 33.33.

2014-06-05 07:22:53,353 [Thread-277] ProbeAlarmHandler:handleAlarms [57] Sent alarm with id: JD11056300-00047




If AMQP is running but there are no alarm processing entries check the following:


1) Check that trap destination setup properly for Vblock components to send traps to Vision.


2) Check your Nimsoft configuration.  Settings related to handling of repeat alarms or alarm suppression may be affecting what is displayed in the Dashboard.