Eyeglass Administration Guide


Eyeglass Isilon Edition

Administration Guide

Product Name - Superna Eyeglass

Revision Changes to this Document 1.9.2



.

Table of Contents

Contents

  1. 1 Product Name - Superna Eyeglass
    1. 1.1 Revision Changes to this Document 1.9.2
  2. 2 Table of Contents
  3. 3 Introduction to this Guide
    1. 3.1 Overview
    2. 3.2 Software Release Updates
    3. 3.3 Help Getting Started with Eyeglass
    4. 3.4 Other Help
  4. 4 Eyeglass Ports Requirements and Scalability Limits
    1. 4.1 Eyeglass Ports Requirements
    2. 4.2 Eyeglass Scalability Limits
  5. 5 Eyeglass Deployment Options
    1. 5.1 Warm Standby
    2. 5.2 Active Active
    3. 5.3 3rd Site Witness (Supported)
    4. 5.4 DR Site Witness (Supported)
  6. 6 Eyeglass Backup Options
    1. 6.1 Eyeglass Active Active Deployment
    2. 6.2 Eyeglass Warm Standby Deployment
    3. 6.3 Supported Replication Topologies for 2 copies of data
    4. 6.4 Eyeglass DR LiveOPS Features
  7. 7 Web Widget Build DR Dashboards
    1. 7.1 Procedures to Generate Web Widget code for web sites
      1. 7.1.1 Example HTML page
    2. 7.2 How the web widget looks in a Browser
    3. 7.3 Example DR Dashboard Using Web widget
  8. 8 Eyeglass Jobs
    1. 8.1 Understand how Eyeglass Jobs work
    2. 8.2 Job Types
      1. 8.2.1 Configuration Replication Jobs automatically created by Eyeglass
        1. 8.2.1.1     1) Share, Export, Alias replication  (Type AUTO)       
        2. 8.2.1.2     2) Access Zone Replication (Type ZONES)
        3. 8.2.1.3     3) Quota replication (Type QUOTA)
        4. 8.2.1.4 4) Configuration Replication: Snapshot Schedules  (Type Filesystem)
      2. 8.2.2 Configuration Replication Jobs not auto created by Eyeglass
        1. 8.2.2.1 1) CUSTOM (Type CUSTOM)
        2. 8.2.2.2 2) Configuration Replication: DFS Mode (Type AUTODFS)
        3. 8.2.2.3 3) Disaster Recovery Testing:  (Type AUTOMATIC)
        4. 8.2.2.4 4) Runbook Robot:  (Type RUNBOOKROBOT And AUTOMATIC)
        5. 8.2.2.5 5) Failover Readiness Jobs (Type AUTOMATIC)
  9. 9 Replication
    1. 9.1 Pre-requisite for Configuration Replication
    2. 9.2 Replicating Shares with $ for Security on Target clusters
    3. 9.3 How does Eyeglass determine uniqueness - the Details
      1. 9.3.1 Uniqueness and equality between shares
        1. 9.3.1.1 Uniqueness and equality between exports
        2. 9.3.1.2 Uniqueness and equality between zones
    4. 9.4 What configuration properties are replicated by Eyeglass?
      1. 9.4.1 Configuration properties replicated for Shares
      2. 9.4.2 Configuration properties replicated for Exports
      3. 9.4.3 Configuration properties replicated for Quotas
      4. 9.4.4 Configuration properties replicated for Zones
      5. 9.4.5 Configuration properties replicated for Snapshots
      6. 9.4.6 Configuration properties replicated for Dedupe
      7. 9.4.7 Configuration properties replicated for SyncIQ Policies
  10. 10 Manage Eyeglass Jobs
    1. 10.1 Overview  
    2. 10.2 View Job Definition and Status
    3. 10.3 Edit Configuration
    4. 10.4 Delete
    5. 10.5 Enable/Disable
    6. 10.6 Enable/Disable Microsoft DFS
    7. 10.7 Run Now
    8. 10.8 Running Jobs
    9. 10.9 Troubleshooting Eyeglass Job
    10. 10.10 What are the Eyeglass System Alarms?
    11. 10.11 Renaming SyncIQ Policy & Eyeglass Jobs
  11. 11 Readiness
    1. 11.1 Monitor DR Readiness
    2. 11.2 Policy Readiness and DFS Readiness
  12. 12 Policy/DFS Readiness - OneFS SyncIQ Readiness
    1. 12.1 Policy/DFS Readiness - Eyeglass Configuration Replication Readiness
    2. 12.2 Policy / DFS Readiness - Zone Configuration Replication Readiness
    3. 12.3 Policy/DFS Readiness - Date-Time Validation
    4. 12.4 DR Dashboard Job Details
  13. 13 Zone Readiness
    1. 13.1 Zone Readiness - OneFS SyncIQ Readiness
    2. 13.2 Zone Readiness - Eyeglass Configuration Replication Readiness
    3. 13.3 Zone Readiness - SPN Readiness
    4. 13.4 Zone Readiness - Smartconnect Zone Failover Mapping Readiness
    5. 13.5 Zone Readiness - View Mapping
    6. 13.6 Zone Readiness - Zone Configuration Replication Readiness
    7. 13.7 Zone Readiness - Target Cluster Reachability
    8. 13.8 Zone Readiness - Date-Time Validation
    9. 13.9 Zone Path Validation
    10. 13.10 FQDN Alias Validation
  14. 14 Script Engine
    1. 14.1 Script Engine Overview
    2. 14.2 Typical script use cases
    3. 14.3 Script Engine Admin Procedures
    4. 14.4 Script Engine CURL Tips
    5. 14.5 Script Engine Understanding Remote Execution to Hosts
      1. 14.5.1 Video: How to Remount Exports Automation
      2. 14.5.2 ssh Passwordless Login to Remote Hosts
      3. 14.5.3 Access Zone Example bash script that uses ssh keys to remotely execute a command
      4. 14.5.4 Remote host script example:
      5. 14.5.5 Eyeglass Access Zone failover and failback Script Example
  15. 15 Adding additional Script Language support to the appliance
  16. 16 Script run time variables
  17. 17 Sample Execution Rules and Overall Failover Job status Impact
    1. 17.1 Sample Scripts in the Library
      1. 17.1.1 NodeJS - Example Script
      2. 17.1.2 NodeJS - Example output
      3. 17.1.3 Python - Example Script
      4. 17.1.4 Python - Example output
      5. 17.1.5 Bash - Example Script
      6. 17.1.6 Bash - Example output
      7. 17.1.7 Consolidated Post Failover & Failback Script (Node.JS)
    2. 17.2 Consolidated Post Failover & Failback Script (Node.JS) - example for multiple Zone Access
  18. 18 Superna Eyeglass API Guide
  19. 19 Services
  20. 20 Network Visualization
    1. 20.1 Overview
  21. 21 RPO Reporting and Trending Feature Guide
  22. 22 End User Offline Detection
  23. 23 Eyeglass Reporting
    1. 23.1 Reports are now available from the Reports on Demand Icon.
    2. 23.2 Eyeglass Cluster Config Reports
    3. 23.3 What’s New
    4. 23.4 Cluster Configuration Email Report
    5. 23.5 Cluster Configuration Report On-Demand
    6. 23.6 Cluster Networking For Failover and Cluster Reports
  24. 24 Role Based Access Controls RBAC
  25. 25 Configure Email, Twitter and Slack for  Notifications  of Eyeglass Monitoring Events
  26. 26 DR Design Guides with Eyeglass
  27. 27 Eyeglass Shell Key Features
    1. 27.1 Open the Eyeglass Shell
  28. 28 Eyeglass CLI Commands
    1. 28.1 Ransomware CLI commands
    2. 28.2 igls adv failover mode
    3. 28.3 igls adv failovertimeout
    4. 28.4 igls adv full sync
    5. 28.5 igls adv runbookrobot
    6. 28.6 igls admin health
    7. 28.7 igls admin appid
    8. 28.8 igls admin version
    9. 28.9 igls alarm active
    10. 28.10 igls alarm all
    11. 28.11 igls app restore
    12. 28.12 igls appliance rediscover
    13. 28.13 igls appliance report
    14. 28.14 igls adv initialstate
    15. 28.15 igls adv PolicyFailover
    16. 28.16 Igls adv quotas
    17. 28.17 Igls adv rundedupe
    18. 28.18 igls admin schedules
  29. 29 Eyeglass Reports
    1. 29.1 Enable/Disable
    2. 29.2 Update Schedule
  30. 30 Configuration Replication
    1. 30.1 Enable/Disable
    2. 30.2 Update Schedule
  31. 31 Runbook Robot Schedule Interval
    1. 31.1 Enable/Disable
    2. 31.2 Update Schedule
  32. 32 Zone Readiness
    1. 32.1 Enable/Disable
    2. 32.2 Update Schedule
  33. 33 Runbook Robot Mount Export Enable Disable
  34. 34 Advanced commands
    1. 34.1 igls adv requesttimeout
    2. 34.2 igls adv spndelay
  35. 35 Cluster Storage Monitor CLI commands
  36. 36 Cluster Storage Reports schedule CLI commands
    1. 36.1 Igls admin sched (lists schedules)
  37. 37 RPO Reporting CLI commands
    1. 37.1 igls adv runreports --report_type=rpo
    2. 37.2 igls adv skipscreenshots
  38. 38 CSM Reporting CLI commands
    1. 38.1 igls adv runreports --report_type=csm
  39. 39 Eyeglass Appliance Administration
    1. 39.1 Eyeglass Processes
    2. 39.2 yast
  40. 40 Eyeglass Appliance Time Synchronization Best Practice
  41. 41 Eyeglass Automatic Updates for Recommended Packages
  42. 42 Update Eyeglass Appliance Network Settings
    1. 42.1 To change the Eyeglass appliance IP address:
    2. 42.2 To change the Eyeglass appliance DNS settings:
    3. 42.3 To change the Eyeglass appliance Routing settings:
    4. 42.4 Eyeglass Root Password
  43. 43 Configuring  Eyeglass for WAN with dual NIC environments
  44. 44 Appliance Security updates and Eyeglass updates with HTTP Proxy
  45. 45 Eyeglass - How to install VMware Native tools
    1. 45.1 Checking open-vm-tools installed package:
    2. 45.2 Removing open-vm-tools package:
    3. 45.3 Installing VMware Tools from the Command Line with the Tar Installer
  46. 46 Upgrading the Eyeglass Appliance
  47. 47 Diagnostic Tools for Dark Sites
  48. 48 Eyeglass Remote Logging Service to Syslog Server
    1. 48.1 Overview
    2. 48.2 Setting Up the Remote Logging
    3. 48.3 Example of Syslog Messages received from Eyeglass
  49. 49 APPENDIX 1 Update Isilon sudoer file for Eyeglass Service Account & Isilon Clusters version 7.1.1.0 and 7.1.1.1 for SyncIQ Policy or DFS Mode Failover

Introduction to this Guide


The purpose of this guide is to assist you in configuring and and administering your Eyeglass Isilon Edition.  

Overview

Eyeglass addresses an HA DR automation and orchestration platform for Isilon clusters.  As file based data becomes mission critical to business, a DR application focused on automating and synchronizing configuration data is needed to access data shares.   

A solution that ensures all data (SyncIQ replication) and configuration data is also replicated for a DR event is only one aspect of a well designed high availability solution.   Many factors need to be considered to fail over a file based application.  

Software Release Updates

The initial releases of the Eyeglass product focused on sync of configuration data, cluster and SyncIQ policy automation failover and reporting automation of configuration and DR readiness. Superna is working continuously to improve Eyeglass Isilon Edition with enhancements and feature updates. The following link will provide you with detail on past, current and future enhancements. (What's New )

Help Getting Started with Eyeglass

To get started you should understand the failover options and which one makes most sense for your environment.  Superna offers design and implementation services that can assist with this decision.

  1. Failover Decision Guide http://documentation.superna.net/eyeglass-isilon-edition/eyeglass-start-here-first

  2. The next most important part of DR is planning, planning, planning  and more planning.  To assist with this planning process we provide a checklist that covers all the key steps that we expect customers to execute before attempting a failover with eyeglass (or failover in general).  Our support team will refer you to this document, if you ask for assistance in failover, to understand which of these preparation steps are completed.   This document is designed to ensure your success.  We suggest you review and adapt it to your business processes.

http://documentation.superna.net/eyeglass-isilon-edition/plan/failover-planning-guide-and-checklist

Other Help

This guide is designed to help with the configuration and administration of Eyeglass Isilon Edition, should you have any issues that are not addressed in this guide, Superna offers support in several forms; on line, voicemail, Email, or live on line chat.

  1. The support site provides online ticket submission and case tracking.  Support Site link - support.superna.net 

  2. Leave a voicemail at 1 (855) 336-1580

In order to provide service you must have an account in our system. When calling in leave; customer name, email, description of question or issue, and primary contact for your company. We will  assign the case to primary contact for email followup.

  1. Email eyeglasssupport@superna.net

  2. To download license keys please go to the following  license keys.

  3. You can also raise a case right from in Eyeglass desktop using the help button, search for your issue and if want to raise a case or get a question answered, click the “leave us a message”  with your name, email and appliance ID and a case is opened directly from Eyeglass.

 http://site.superna.net/_/rsrc/1472870726155/support/LeaveUsAMessage.png?height=200&width=167

  1. Or get Support Using Chat M-F 9-5 EDT  (empty box?  we are not online yet)

  2. Eyeglass Live Chat 

  3. You should also review our support agreement here.


Eyeglass Ports Requirements and Scalability Limits

Eyeglass Ports Requirements


Port

Direction

Function

8080

Eyeglass appliance → Isilon cluster

REST API

SSH port 22

Eyeglass appliance → Isilon cluster

SSH access for some CLI commands

NFS

Eyeglass appliance → Isilon Cluster

RunBook Robot to Mount the cluster for DR Automation testing (reads and writes)

443  

browser  → appliance

Secures client to browser access

37356

Service broker UIM Probe → eyeglass

Service broker for UIM probe alarms and communications with eyeglass heartbeats

source port 80 → destination random TCP port on the browser

browser ← appliance

If connection on ip address port 80 is made a http 301,302 redirect is returned on port 80 to switch the browser to https and url https:/x.x.x.x/eyeglass

source random , destination is port 80   

browser  → appliance

redirected to 443 TLS access

2011 websocket

appliance  →  browser  

Websocket for real-time appliance to browser updates (redirected to 2012)

2012 TLS websocket

appliance  → browser

Websocket for real-time appliance to browser updates (redirected to 2012)

SSH 22    

workstation  → appliance  

secure shell access







Eyeglass Scalability Limits


Scaling Limit Area

Tested Scaling Limits 1.5 >

Notes

Number of Managed Clusters 1 appliance

22

Requires 16G RAM

Number of shares replicated (total across all clusters)

15,000

Requires 16G RAM

Number of Exports replicated (total across all clusters)

10, 000

Requires 16G RAM

Number of NFS aliases    replicated (total across all clusters)

10, 000

Requires 16G RAM

Number of Quotas replicated (total across all clusters)

20, 000

Requires 16G RAM

SyncIQ Policies All clusters

100

Requires 16G RAM

Failover job limitations

100 policies selected in a single failover

Requires 16G RAM

Failover job total object count (shares + exports + quotas)

10,000>

Requires 32G RAM

Total objects inventory all clusters

50 000

Requires 16G RAM, some clusters require API requests throttled with large object count, contact support if you have this many (shares,exports and quotas combined) to get recommended setting applied

   
   

Eyeglass Deployment Options

Several options exist that allow customers to deploy Eyeglass in several configurations.

Warm Standby

This procedure is to protect the production Eyeglass appliance.  Using the Eyeglass Automated 7 Day Backup feature  and SyncIQ, the procedure will sync the backup archives to a running Warm Standby Eyeglass appliance in a  2nd cluster.

Notes on this solution:

  1. This process also means only One Eyeglass appliance has clusters added.

  2. This only requires one set of license keys (license keys are appliance specific)

  3. Requires SyncIQ enabled clusters to protect the archives

  4. Production Eyeglass sync’s Export used for DR of Eyeglass configuration data



Eyeglass Isilon Edition - SyncIQ DR Orchestration Appliance Overview v25.png


The following link provides detailed information on how to set up the Warm Standby Backup Procedure  (Warm Standby)

Active Active

This procedure is to protect the production Eyeglass appliance using a 2nd licensed Eyeglass appliance.  This solution offers same time sync of DR data to a second appliance that can take over operations under the following conditions:

  1. Controlled failover - switching the active appliance from one data center to the other

  2. Uncontrolled failover protection - the 2nd appliance can be used for failover operations since it has a current near real time synced status and copy of all policies and configuration data (shares, exports and quotas) needed to complete a failover to the surviving cluster.

Definitions:

  1. Active Sync Appliance - Responsible for syncing configuration data and is the primary appliance for all failover operations

  2. Active Monitoring Appliance - Responsible for inventory of configuration data needed for syncing, monitors syncIQ, collects syncIQ job reports

Notes:

  1. This process means Two Eyeglass appliance have clusters added.

  2. This requires two sets of license keys (license keys are appliance specific)

  3. One appliance has active sync jobs and the other has disabled policies but fully synced view of both clusters






Eyeglass Isilon Edition - SyncIQ DR Orchestration Appliance Overview v28.png


The following link provides detailed information on how to set up the Active Active configuration ( Active Active )   

3rd Site Witness (Supported)

In order to have all DR orchestration and information separate from the DR site, it’s best practise when possible to use a 3rd site to monitor your clusters with Eyeglass that is IP connected to Site A and Site B as shown below.  

DR Site Witness (Supported)

In this configuration the witness function resides at the Cold or DR site, so that configuration and orchestration data is available in a DR event.


Eyeglass Backup Options

Eyeglass Active Active Deployment

Active-Active Eyeglass deployment protects the production Eyeglass appliance by using a 2nd licensed Eyeglass appliance which is real time synced to latest information.  Please refer to How to setup Active Active Appliance Procedures for more information.

Eyeglass Warm Standby Deployment

The Warm Standby Eyeglass deployment protects the production Eyeglass appliance by using a 2nd unlicensed Eyeglass appliance and a backup and restore procedure for Eyeglass configuration.  Please refer to How to setup Warm Standby Backup Procedure for more information.

Supported Replication Topologies for 2 copies of data

Multi protocol NFS and SMB Access Zone failover supports 2 copies of data protection, with Active / Passive configuration.  An Access Zone can not span clusters due to DNS delegation and simplified failover with eyeglass that is fully automated.  

Eyeglass Isilon Edition - SyncIQ DR Orchestration Appliance Overview v36 (1).png

Eyeglass DR LiveOPS Features

Liveops are a set of features designed to allow continuous operations with no impact to production DR or cluster operations.  The first feature, available in this new family of features, is DR Test mode. This allows customers to execute DR testing on 3rd copies of data that also includes configuration data synced from the production cluster into a test access Zone.

For detailed instructions on configuration, operation and requirements of LiveOPS features consult the Live OPS Admin Guide located here.

Web Widget Build DR Dashboards

This new feature uses the Eyeglass API to allow embedding DR dashboard failover tabs into external web pages. The feature  uses an API token to authenticate and provide read only views for centralized dashboards.

The widgets are interactive and dynamically updated based on real-time DR data collected by Eyeglass.  This allows more teams access to key business status data without needing admin login to the Eyeglass console itself.

Procedures to Generate Web Widget code for web sites

  1. Login to Eyeglass

  2. On Eyeglass Menu select eyeglass REST API

  3. Select API Token tab

  4. Generate a token with a name (example; web)

  5. Goto Web Widget tab

Screen Shot 2016-04-15 at 8.59.56 PM.png

  1. Select the DR Dashboard Tab to display in the Widget

  2. Select the token name in the list

  3. Widget width and height

  4. Cut and paste the html code into a web page per the example in step 10  below

NOTE:  Eyeglass uses a self signed certificate which means many browsers will block REST API calls used by the java script in the web widget with end points using un-signed certs.   This requires a signed cert added to eyeglass using an external trusted CA or internal corporate signed cert.

    1. See procedure to install a signed cert to Eyeglass here

    2. You can install the Eyeglass cert into a browser as trusted for testing.  See instructions here for OS X and chrome http://www.robpeck.com/2010/10/google-chrome-mac-os-x-and-self-signed-ssl-certificates/#.VxlSiZMrLBI

  1. Example with IIS a web page:

Example HTML page

This sample page loads two dashboards, one for Access Zone and the other for DFS.  

<html>

<header><title>API Widget</title></header>

<body>



<h3> Access Zone Readiness Failover Dashboard </h3>

<script type="text/javascript" src="https://172.31.1.102/RestClient/utils/widget?widget=zonereadiness&token=igls-1kmrb3h4bh7h9uefenstv3t3voospcv87c5aburs7r27l8al7sq7&height=400&width=700"></script>

<!-- Add the div below to the desired location on your webpage -->

<div class="superna-eyeglass-widget-zonereadiness"></div>


<h3> DFS Mode Failover Dashboard </h3>

<script type="text/javascript" src="https://igls-update1/RestClient/utils/widget?widget=dfsreadiness&token=igls-1kmrb3h4bh7h9uefenstv3t3voospcv87c5aburs7r27l8al7sq7&height=400&width=700"></script>

<!-- Add the div below to the desired location on your webpage -->

<div class="superna-eyeglass-widget-dfsreadiness"></div>


</body>

</html>


How the web widget looks in a Browser

Screen Shot 2016-04-21 at 7.38.53 PM.png

Example DR Dashboard Using Web widget

https://youtu.be/sZqzoTD66i4

Eyeglass Jobs

Understand how Eyeglass Jobs work

Jobs are the basis for all automation in Eyeglass and job types dictate the type of automation that will be performed.  Currently the job types supported are the following:

  • Configuration Replication (shares,exports, permissions, nfs alias)

  • DFS mode enabled config replication policies (shares only)

  • Quota Replication (quotas - all types)

  • Snapshot schedule and Dedupe settings

  • Access zone replication


Failover related job types

  • Failover jobs  (SyncIQ policy or Access Zone)

    • Created by DR Assistant

  • Failover Readiness

  • Runbook robot jobs

  • Disaster Recovery Testing - LiveOPS DR test mode jobs

  • Access zone data migration

    • Created by access zone migration in jobs icon

Further to the above jobs, the following modes can be set on  jobs:

  1. Automatic - These are built by Eyeglass after autodetection of SyncIQ policies which are then used to detect which shares, exports , nfs alias and quotas should be replicated. They are automatically created.

  2. Custom -

    1. These are user created and allow detection of shares, exports, quotas in a path in the file system to be detected and added to a job that will replicate the configuration to a target cluster without needing a SyncIQ policy to exist.

    2. The data and path must exist on the target cluster


Job Types

Configuration Replication Jobs automatically created by Eyeglass

    1) Share, Export, Alias replication  (Type AUTO)       

     Purpose:

  • Identify shares / exports / nfs alias that are related to SyncIQ Policies detected based on SyncIQ policy source path

  • Synchronize these configuration items so that they exist on both clusters

  • associated shares & their configuration

  • associated exports & their configuration

  • associated nfs alias & their configuration


Eyeglass Configuration Replication Job Name convention:  < Isilon Cluster name >_< SyncIQ Policy name >

      

Job Creation:

The Share, Export, Alias replication jobs are auto created by Eyeglass after autodetection of SyncIQ policies which are then used to auto-detect which shares, exports , nfs alias, Access Zones and quotas should be replicated by Eyeglass.

Schedule: All Eyeglass share/export/alias configuration replication Jobs execute on a 5 minute schedule.

Initialstate: User Disabled (does not run)

    2) Access Zone Replication (Type ZONES)

Note: This replication occurs when the associated Zone is NOT the System Zone.

Purpose:

  • Identify Access Zone that is related to SyncIQ Policies detected based on SyncIQ policy source path

  • Synchronize Access Zone so that it exists on both clusters:

  • associated Zone & it's configuration

Note: Deleted Zones on a source cluster are not deleted on the target cluster.

Eyeglass Zone Job Name convention: < Isilon Cluster name >_< SyncIQ Policy name >-< ZONES >

Job Creation:

The Access Zone replication jobs are auto created by Eyeglass based on the SyncIQ Policy of the same name.

Schedule:  Zone replication jobs execute on a 5 minute schedule.

Initialstate: User Disabled (does not run)

    3) Quota replication (Type QUOTA)

Background:

Quota jobs that are created based on SyncIQ autodetection are placed in a pending state.  This state prevents quotas policies that are collected and shown in the Inventory tree from being applied to target clusters paths protected by SyncIQ policies.  This is a best practise due to some scenarios that result in errors when quotas are applied to a target cluster file system.  

The scenarios to apply quota policies are below.

  1. In a failover event, the quota job can be selected and "Run now" option used AFTER the target cluster file system is writeable as a result of SyncIQ failover. This is run automatically under normal conditions. Cluster migrations is another use case where applying to target without a delete on source is desireable.

  2. Custom jobs can replicate quota policies on a schedule for a path selected in the job, the quotas are applied successfully only when the target file system path on the target cluster already exists.  Any new quota created under the selected job path, will be detected and replicated only if the target path also exists

Note: When you run a QUOTA Job associated with an AUTO share/export configuration replication Job, the Job is based on the Eyeglass current inventory view.  If you have made a change in OneFS to a quota, the Eyeglass Inventory Task must have run (runs on a 5 minute schedule) prior to running the QUOTA job in order for the change to be applied on the target.


Warning review quota failover limitations:

This section should be reviewed when planning quota failover solutions.

Review Dell EMC Quota EMC KB - https://support.emc.com/kb/88602

Some combinations of quota domain settings and SyncIQ source and target settings are incompatible. There are several scenarios where this might occur:

  • Multiple quota domains span SyncIQ target subtrees. SyncIQ translates source operations such as file or directory moves (mv) into similar actions on the target cluster. Moving files or directories across quota domains is not supported, and the syncs will fail.

  • Multiple quota domains span SyncIQ source subtrees. If the source cluster will be used for failback, the failback operation could error if multiple quota domains span SyncIQ source subtrees.

  • If the source cluster will not be used for failback, multiple quota domains can exist on the source.

  • Quota domains exist in directories other than the top-level directory of the SyncIQ policy locations. If a SmartQuotas quota domain overlaps with a SyncIQ policy domain, and failback is desired, then the quotas created must exist only on the top-level directory of the SyncIQ policy source and target locations.

  • A QuotaScan job is still running when sync job starts. The quota scan identifies statistics about the files in a quota domain. If the quota scan does not finish identifying all the files that belong in the quota domain before the sync job starts, the sync will fail.

  • Nested subdirectories receiving new files below an applied quota path at the target side. As part of the transfer process, SyncIQ first creates files and directories in a temporary directory in the target path and then later moves (renames) them into the final destination. If the final destination has a quota domain configured, this will run into the quota limitation of not being able to move directories into and out of quota domains.

Purpose:

  • Identify Quotas that are related to SyncIQ Policies detected based on SyncIQ policy source path

  • Synchronize Quotas so that they exist on both clusters:

    • associated quotas & their configuration

      

Eyeglass Quota Job Name convention:  <Isilon Cluster name>_<SyncIQ Policy name>_quotas

Job Creation:

The Quota Failover jobs are auto created by Eyeglass based on the SyncIQ Policy of the same name.

Schedule: Auto-created quota configuration replication Jobs do not run automatically.  They are run on-demand as part of a failover.



Initialstate: User Disabled (does not run)

4) Configuration Replication: Snapshot Schedules  (Type Filesystem)

Purpose:

Sync Snapshot schedules found on SyncIQ paths.  Syncs the schedule as per SyncIQ policy paths defined. Will also read dedup path settings (scan and actual) for SyncIQ Policies that match only , and apply the path (corrected by policy path) to target cluster.  Can be disabled independently of snapshots using igls command.

Job Creation:

Automatically build, user disabled by default must be enabled.

Schedule: The FILESYSTEM Jobs are run automatically on a 5 minute schedule.

Initialstate: The FILESYSTEM Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.

Note: Snapshot schedule config sync does not overwrite existing Snapshot schedules on the target cluster which have a different Snapshot schedule name.

Configuration Replication Jobs not auto created by Eyeglass

1) CUSTOM (Type CUSTOM)

Purpose:

Use when config data is not protected by SyncIQ policies. Scan path to find config data and replicates to target cluster and path defined in the job.  Once created this Job will:

  • Identify shares / exports / nfs alias that are related to the Custom Job path

  • Synchronize these configuration items so that they exist on both clusters

  • associated shares & their configuration

  • associated exports & their configuration

  • associated nfs alias & their configuration

Job Creation:

If you see Job Type “CUSTOM” it means that this is a share/export/nfs alias/quota configuration replication Job that was created manually in the Eyeglass web page.  A CUSTOM job that was not created based on a SyncIQ Policy and a SyncIQ Policy is not required or allowed.

  • Multiple Eyeglass configuration replication jobs where paths overlap is not supported. (i.e. A Custom Job path cannot overlap with another Custom Job path or an “AUTO” Configuration Replication Job)

  • Eyeglass custom job where path is the parent of another job is not supported  

Schedule:

  • The CUSTOM Jobs are run automatically on a 5 minute schedule.

  • QUOTA Jobs associated with a CUSTOM share/export configuration replication Job are run automatically on same replication schedule as the associated CUSTOM share/export replication Job.

Initialstate: User Disabled (does not run)

2) Configuration Replication: DFS Mode (Type AUTODFS)

Purpose:

Please refer to Eyeglass SyncIQ Failover and Failback with Microsoft DFS for more information.

Job Creation:

DFS Mode is enabled manually from the Eyeglass web page. Please refer to Eyeglass SyncIQ Failover and Failback with Microsoft DFS for more information.

Schedule: The AUTODFS Jobs are run automatically on a 5 minute schedule.

Initialstate: The AUTODFS Jobs when enabled will have the same state as the AUTO Job that it came from.  

3) Disaster Recovery Testing:  (Type AUTOMATIC)

Purpose:

3RD Copy LiveOPS feature, sync’s config from prod to DR test access zone if enabled.

Job Creation:

The mode is auto and is automatically built if Eyeglass detects DR test mode policy, user disabled by default must be enabled.

Schedule: The AUTOMATIC Jobs are run automatically on a 5 minute schedule.

Initialstate: The AUTOMATIC Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.  

4) Runbook Robot:  (Type RUNBOOKROBOT And AUTOMATIC)

Purpose:

Continuous DR feature to failover and back daily to DR readiness

Job Creation:

Mode is auto since it's automatically built if Eyeglass  detects robot access zone or policy name, user disabled by default must be enabled. Igls command to set the schedule

Schedule: Th e RUNBOOKROBOT Jobs are run automatically on a 24 hour schedule.

Initialstate: The RUNBOOKROBOT Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.  

5) Failover Readiness Jobs (Type AUTOMATIC)

Purpose:

Analyzes access zones against failover readiness criteria (data, config, SPN, network smartconnect mapping). Updates the DR dashboard with access zone readiness and criteria readiness status.  Collects subnet and pool data to display failover mapping on the Access Zone Readiness panel on the DR Dashboard.   Failover Readiness Jobs are created between replicating cluster pairs, one job for each direction.

Note: involved in failover operations, it is used to update DR assistant on readiness,  update DR dashboard for users to correct errors,  alarms Zone Readiness status to alert administrator.

Job Creation:

Mode is auto and is built automatically by Eyeglass and set to user disabled.  The job  can be enabled to analyse access zone readiness, if access zone failover is not planned it can remain disabled.

Failover Readiness Job Name convention: Isilon Cluster name_Isilon Cluster name

Schedule: Execute on a 15 minutes schedule.

Initialstate: Disabled

Replication

Pre-requisite for Configuration Replication

In order for Eyeglass to be able to execute the configuration replication for the shares/exports/nfs alias/quotas/zones that have been detected for the SyncIQ policies, the following is required:

  • Directories associated with the shares/exports/quotas/zones must exist on the target Isilon cluster as defined in the SyncIQ policy

  • Access Zones associated with the shares/exports/quotas must exist on the target Isilon cluster

Replicating Shares with $ for Security on Target clusters

This section describes how configuration replication can be used to hide shares on DR and unhide shares after failover.    This is to hide the data on the DR cluster automatically.


Use Cases

  1. DFS mode for SMB failover with shares renamed on failover and hidden on DR cluster

  2. Access Zone failover with shares Synced and hidden on DR cluster

Configuration

  1. Enable a policy for DFS mode from jobs icon

  2. Note: This will work for Access Zone failover as well.

    1. Screen Shot 2017-06-14 at 8.55.01 PM.png

  3. Now change the dfs config suffix.

    1. This can be enabled by editing /opt/superna/sca/data/system.xml file.  Change the tag to include $ as shown below <dfssharesuffix>$</dfssharesuffix>  NOTE: on an existing installation the old DFS renamed share will need to be manually deleted.

  4. After failover the share names will flip and hide it on the Source cluster.  


How does Eyeglass determine uniqueness - the Details

Uniqueness and equality between shares

The combination of share name and zone name is the unique value that we use to determine if one share equals another.  This applies to records on the same cluster, as well as comparing shares between two Isilon clusters.

Uniqueness and equality between exports

Uniqueness is determined by the client list and the path to sync between access zones on replicating clusters.  Same path but different client list (all client lists) will be treated as different exports to sync within a given access zone.

Uniqueness and equality between zones

Zone name is used to determine if one zone equals another.  This applies to records on the same cluster, as well as comparing zones between two Isilon clusters.

What configuration properties are replicated by Eyeglass?

Configuration properties replicated for Shares

All configuration properties are replicated with the following exceptions:

  • Local users & their permissions

  • filesystem users & their permissions


! SID for local and filesystem users are replicated but not mapped to users on the target. This means AD at the target cluster should be setup to resolve SID's using the same AD forest.


NOTE: Shares with variable expansion now sync correctly to the target cluster example %u share names.

NOTE: Locally created uses or groups on Isilon are created and generate a cluster unique UID or GID.  This means users with the same name on two clusters have different UID and GID.  This is normal linux machine behaviour.  Sync local users or groups have no value, since userX on cluster A will not be the same userX synced on cluster B, and therefore will not have access to any data secured to userX on cluster A.   The best practice is use LDAP or kerberos for user security for exports or Active Directory for unified security database.

Configuration properties replicated for Exports

For exports, all configuration information is replicated to target with the following exception:

  • export id: Exports on source and target will not have the same export id.

  • Dual path exports must have both paths fall under the same SyncIQ policy.

  • FQDN clients listed must be resolvable by the target cluster. If this is not possible Eyeglass full sync mode must be used to override API validation and force create exports.  See ilgs CLI command documentation section in this guide.

Configuration properties replicated for Quotas

For quotas, all configuration information is replicated to target with the following exceptions:

  • Usage Accounting - include Snapshot Data (best practice to not replicate this setting with SyncIQ)

  • Usage Accounting - include Data-Protection Overhead (best practice to not replicate this setting with SyncIQ)

  • Quota definitions that use user or group quota from local or filesystem are unique across clusters even if the same name is used.  This means local/filesystem users and well known accounts should not be used with Quota Replication feature.   Active Directory should be used with user and group quota and custom or automatic quota replication jobs.

Configuration properties replicated for Zones

The following configuration properties are replicated for Zones other than the System zone:

  • Access Zone Name

  • Zone base directory

  • Authentication Providers (name only)

  • User Mapping rules (Isilon 7.1.x.x only)

! Authentication providers / Local providers must exist on the target to resolve the SID' used in share permissions and the filesystem ACL's.

Configuration properties replicated for Snapshots

  • Snapshot schedule properties (only for paths that match SyncIQ policies cluster wide) with the following exceptions:

    • next_run  (auto-populated by OneFS)

    • next_snapshot  (auto-populated by OneFS)

Configuration properties replicated for Dedupe

  • Dedupe path for job processing (only for paths that match SyncIQ policies cluster wide)

  • Dedupe assessment paths (only for paths that match SyncIQ policies cluster wide)

Configuration properties replicated for SyncIQ Policies

  • On Failover only the SyncIQ schedule is re-applied to newly created mirror policies

  • File pattern filters set on SyncIQ policies are NOT synced on failover, these patterns can result in unprotected data during failover and failback.   Failover and failback work flows require customer testing.  All file filter use cases  are untested without support or documentation.

Manage Eyeglass Jobs

Overview  

The Jobs Feature is used to keep track of Eyeglass tasks.  Two Job views are provided :

  • Job Definitions: Use the Job Definition view to see details for and manage Eyeglass configuration replications Jobs

  • Running Jobs: Use the Running Jobs view to see the statue of active tasks.

View Job Definition and Status

Job Definitions Window :

Job Name : It displays name as was defined during job creation.

This column displays a + symbol along with it which when expanded provides more details about the job such as :

Enabled/Disabled : The Job can be enabled or disabled from the Jobs window itself. When it is disabled , this field displays “USERDISABLED” State , If it is enabled, it displays “ENABLED” . For Auto Type Jobs, If the Policy is disabled from OneFS, it displays “POLICY DISABLED”.

Job Type : It display two Job types : “AUTO” and “MANUAL”.

  • Auto is for Jobs automatically detected by Eyeglass when a policy is created in OneFS and the Job is scheduled from there.

  • Manual is for Jobs that were manually created from the Jobs window.

Source : It displays Isilon cluster name configured as Source for a replication Job.

Source Path:  The source path is used to provide a point in the file system for Eyeglass to autodetect shares,exports, quotas to discover and include in the job.  (a job can be edited, using Edit Configuration(s) to remove a detected configuration that you don’t want included in the job)

Destination: It displays Isilon cluster name configured as Destination for replication Job.

Last Success: This field displays date when the job was last successfully run. If the Job was never run successfully, this field is empty.

Last Run Date:  It displays the date when the job was last run regardless of its last state was failed or success.

State: It displays the current state of Job.

  • “OK” if the Last Run date and Last success date matches.

  • “ERROR” if the Last Run date and Last success date doesn’t match.

  • "SYNC IQ RUNNING" if the associated SyncIQ Job is in running state

Edit Configuration

To view the list of associated shares/exports/nfs alias/quotas for an Eyeglass Job, select Jobs.



   


To see the shares/exports/nfs alias associated with your Eyeglass configuration replication Job, select the checkbox for your Job and then select the Select a bulk action button.  The Edit Configuration(s) option will take you to the Jobs - Edit Configuration(s) View window where you can expand the tree to see which shares and exports and nfs alias are associated with your job.




Follow the same steps for a Quota configuration replication Job to see which Quotas are associated with the Job.


Follow the same steps for a Zone configuration replication Job to see which Zones are associated with the Job.

Delete

Able to delete manually created jobs and the jobs that are not currently running.

Enable/Disable

Able to enable/disable all type of jobs except Eyeglass Failover:Runbook Robot (AUTOMATIC) (TYPE = RUNBOOK).

Enable/Disable Microsoft DFS

Able to enable/disable DFS mode for Configuration Replication: Share, Export, Alias replication type of jobs. Please refer to Failover Feature Guide for Microsoft DFS Mode Failover and Failback.

Run Now

Able to run Eyeglass Configuration Replication Jobs on demand.

  1. Open the Jobs window.

  2. Select an Eyeglass Configuration Replication Job.

  3. Select the button Select a bulk action.

  1. Elect Run Now.  This will initiate the Eyeglass Configuration Replication task which will run all Eyeglass Configuration Replication Jobs.

  1. You can check the progress of the Configuration Replication Job in the Running Jobs view of the Jobs window.

Running Jobs

This category displays the currently running jobs and its details such as :

Job Name : displays the name of Job

State : displays the state of Job : Finished or Running

Started : displays time when it was started

Finished : displays time when it was finished.

To view a Job in progress select Running Jobs from the Jobs window.  In the bottom Job Details pane you will see the steps associated with the Job and the status for each step.




Troubleshooting Eyeglass Job

When a problem occurs, for a Job in progress the Running Jobs view will indicate at which step a problem occurred with a link to the information available for the error.


For a completed Job, the Eyeglass system alarm related to the failed Job may contain the extra information you need to troubleshoot the problem






What are the Eyeglass System Alarms?

Refer to Eyeglass Isilon Alarm codes for the list of system alarms.

Renaming SyncIQ Policy & Eyeglass Jobs

When a SyncIQ Policy is renamed, Eyeglass considers it to be a new SyncIQ Policy and therefore the following happens:

  1. Eyeglass Job(s) related to the original SyncIQ Policy Name are deleted.

  2. Eyeglass Job(s) for the new SyncIQ Policy Name are created.

As a new Job, the Eyeglass Configuration Replication Jobs will have a state depending on the Eyeglass appliance default “Initialstate” configuration (see section igls adv initialstate in this Admin Guide).  By default the Jobs will be disabled.  Any other customizations to the Job such as DFS Mode must also be re-applied.  For a SyncIQ Job that has been renamed, follow these steps in Eyeglass:

  1. Login to the Eyeglass web page.

  2. Open the Jobs window.

  3. For any Jobs that were in DFS mode previously, re-set to DFS Mode.

  4. For any Jobs where sync of a share or export had been customized to be skipped, from the Jobs page, use Edit Configuration to deselect those objects from the Job.

In addition to the Job setup, RPO Reporting for the original SyncIQ Policy name will not be linked to RPO reporting for the new SyncIQ Policy name.

Readiness

Monitor DR Readiness

To view and manage your DR readiness, use the DR Dashboard window:

Policy Readiness: Provides a view of your readiness for SyncIQ Policy Failover.

Zone Readiness: Provides a view of your Access Zone Failover Readiness.

DFS Readiness: Provides a view of your readiness for a SyncIQ Policy DFS Mode Failover.




Policy Readiness and DFS Readiness

The DR Failover Status on the Policy or DFS Readiness tabs provides you with a quick and easy way to assess your Disaster Recovery status for a SyncIQ Policy Failover (non-DFS mode) or SyncIQ Policy Failover (DFS mode).  

Screen Shot 2016-11-23 at 3.19.45 PM.png


Each row contains the following summary information:


Column

Description

Notes

Name

Name of the Eyeglass configuration Replication Job

Eyeglass configuration replication Job created automatically for each SyncIQ Policy detected with exactly the same name + prefixed with Isilon Cluster name.

Quota Jobs are suffixed with “quotas”.

SyncIQ Policy

Name of the SyncIQ Policy associated with the Eyeglass Job


Source

The Isilon cluster that is the source configured in the SyncIQ Policy

Eyeglass Job will have same source

Destination

The Isilon cluster that is the target configured in the SyncIQ Policy

Eyeglass Job will have same target

DR Failover Status

A status calculated by Eyeglass based on the failover validation criteria for the Policy or DFS Failover mode

This column displays the overall status.  Select the link to see individual status for each validation criteria.

New option shows “Failed Over” for any policy that is read-only status in SyncIQ.



Click on the DR Failover Status link to see the details for each of the failover validation criteria for a selected Job.






The DR Failover Status will be one of the following:


Status

Failover Impact

OK

Able to Failover

WARNING

Warning state does NOT block failover.


IMPORTANT: While failover is not blocked, the issue(s) causing this Warning may cause the failover to fail.  Recommendation is to understand and resolve these issues prior to failover.

ERROR

Error state BLOCKS failover.


DISABLED

Disabled state DOES block failover.


FAILED OVER

Failed Over state DOES block failover.



The DR Failover Status is based on Status for each of the following areas for Policy and DFS Failover:

OneFS SyncIQ Readiness

Have the SyncIQ policies in the Access Zone been successfully run and are they in a supported configuration?

Eyeglass Configuration Replication Readiness

Have the Eyeglass Configuration Replication jobs in the Access Zone been successfully run to sync configuration data for all policies that are members of the Access Zone?

Zone Configuration Replication Readiness

Has the Eyeglass Zone Configuration Replication job been successfully run to create target cluster access zones that don't already exist for configuration sync complete?

Date-Time Validation

Are the date-time differences between nodes and between eyeglass and the clusters within an acceptable range that will not affect SyncIQ operations?


Additional information for Policy / DFS Readiness criteria is provided in the following sections.


Policy/DFS Readiness - OneFS SyncIQ Readiness

The OneFS SyncIQ Readiness criteria is used to validate that the SyncIQ policy has been successfully run and that it is in a supported configuration.   For each SyncIQ Policy the following checks are done:  





SyncIQ Policy:

Notes:

Policy Source Nodes Restriction

Validate Isilon best practices that recommend that SyncIQ Policies utilize the Restrict Source Nodes option to control which nodes replicate between clusters.

SyncIQ Policy Status

DR Status is “OK” when all of the conditions below are met:

  • Your SyncIQ Policy is enabled

  • Your SyncIQ Policy last state was finished or needs attention

DR Status is “Warning” when the condition below is met:

  • SyncIQ Policy has a last state that was not successful

  • SyncIQ Policy has a last state that was paused or cancelled

  • SyncIQ Policy does not not have a last state (has never been run)

  • SyncIQ Policy has Excluded Directories and/or Included Directories configured


IMPORTANT: SyncIQ Policy in Warning state MAY NOT be able to be run by Eyeglass assisted failover depending on it’s current status.  If not run, you will incur data loss during failover.  

  • Example 1: SyncIQ Policy has an error state.  If it cannot be run from the OneFS, it will also not be able to run from Eyeglass.  

  • Example 2: SyncIQ Policy is paused.  Eyeglass failover cannot RESUME a paused SyncIQ Policy - this must be resumed from OneFS

You must investigate these errors and understand their impact to your failover solution.


DR Status is “Disabled” when either of the conditions below are met:

  • Eyeglass configuration replication Job is user disabled

  • OR the SyncIQ policy in OneFs is disabled


Corrupt Failover Snapshots

Validate that Target Cluster does not have an existing SIQ-<policyID>-restore-new or SIQ-<policyID>-restore-latest snapshot from previous failovers/synciq jobs for the Policy

Policy Local Target Validation :

  • Duplicate SyncIQ Local Targets

Validate that there is only 1 Local Target per SyncIQ policy.

Policy Local Target Validation:

  • Target Writes Disabled

Validate that the target folder of SyncIQ policy has writes disabled.

Policy Enabling

Validate that the SyncIQ Policy is enabled in OneFS.


Policy/DFS Readiness - Eyeglass Configuration Replication Readiness

The Eyeglass Configuration Replication Readiness criteria is used to validate that the Eyeglass Configuration Replication job related to the SyncIQ Policy has been successfully run to sync the related configuration data.  For the Eyeglass Configuration Replication Job the following check is done:




<Eyeglass Configuration Replication Job Name>

Validate that the Eyeglass Configuration Replication Job status is not in error state.


DR Status is “OK” when:

  • Your Eyeglass configuration replication Job is enabled

  • Your Eyeglass configuration replication Job Last Run and Last Success timestamp are identical

  • Your Eyeglass configuration replication Job Audit Status is OK


DR Status is “Warning” when either of conditions below are met:

  • Eyeglass configuration replication Job is new and has not been run yet

  • Eyeglass configuration replication Job has been successfully executed but not audited

  • Eyeglass configuration replication Job Last Run was not successful

  • Eyeglass configuration replication Job Audit was not successful


DR Status is “Disabled” when either of the conditions below are met:

  • Eyeglass configuration replication Job is user disabled

  • OR the SyncIQ policy in OneFs is disabled


Policy / DFS Readiness - Zone Configuration Replication Readiness

The Zone Configuration Replication Readiness criteria is used to validate that the Zone Configuration Replication job, related to the SyncIQ Policy, has been successfully run in order to create a new target cluster Access Zone  for configuration sync completion.  


Policy/DFS Readiness - Date-Time Validation

The Date-Time Validation is used to validate that the time difference between the cluster nodes, and between clusters and Eyeglass, are within an acceptable range that will not affect SyncIQ operations.  SyncIQ commands, like re-sync prep, can fail if the time between cluster nodes is greater than the time between the Eyeglass vm and the cluster with regards to latency of issuing the API call (a scenario where a node returns a timestamp for a step status message is earlier than the beginning of the Re-sync prep request).  API calls can return different completion times between clusters.  Differences here can cause resync prep failover commands to fail, if the difference between Eyeglass and the source cluster is greater than the time it takes for a resync prep command to complete.

NOTE: This condition is rare but hard to detect manually and Eyeglass can now detect this condition.  It's best practise to use NTP on clusters and Eyeglass appliance.  This allows failover logs and the new feature in release 1.8 or later to collect cluster SyncIQ reports for each step and append to the failover log. This will make debugging multi step multi cluster failover simpler.  This process will require time to be synced.


For each Cluster the following checks are done:  





Date-Time Validation

Notes:

Nodes Validation

Validation that the maximum time difference between the nodes of a cluster is less than the time required for the cluster node time request made by Eyeglass to complete.

Eyeglass & Clusters Validation

NOTE: Only applicable if Nodes Validation is OK. Validation that the earliest node time for a cluster and the Eyeglass appliance time are less than the time required for the cluster node time request made by Eyeglass to complete plus a default additional skew factor (default 1s).

Executed if Nodes Validation is OK.


DR Dashboard Job Details

Each Policy or DFS Job can be expanded in the DR Dashboard Policy Readiness or DFS Readiness view to see Job Details:





DR Dashboard Configuration Replication Job Details

Column

Description

Notes

SyncIQ Policy

All information in the SyncIQ Policy details section comes from the Isilon Cluster itself

If this section is empty then the Job is a custom Eyeglass job not associated with a SyncIQ policy

Job Name

Name of the SyncIQ Policy

Same name on the Isilon cluster

Last Started

Date/time when the last SyncIQ Policy job was started

Information retrieved from SyncIQ Policy details on the Isilon Cluster

Last Success

Date/time when SyncIQ Policy last run successfully on the Isilon

Information retrieved from SyncIQ Policy details on the Isilon Cluster

Last Job State

Status for Last SyncIQ Policy Job

Information retrieved from SyncIQ Policy details on the Isilon Cluster used to determine Overall DR Status

Enabled

Indicates whether the Isilon SyncIQ policy is enabled


Eyeglass Configuration Replication

All information in the Eyeglass Configuration Replication details section comes from Eyeglass


Job Name

Name of the Eyeglass Configuration Replication Job

Eyeglass Configuration Replication Job created automatically for each SyncIQ Policy detected with exactly the same name and prefixed with Isilon Cluster name.

Quota Jobs also suffixed with “quotas”.

Last Run

Date/time when the last Eyeglass Configuration Replication Job was started

Information used to determine Overall DR Status

Last Success

Date/time when Eyeglass Configuration Replication Job was last run successfully

Information used to determine Overall DR Status

Audit Status

Indicates status of the Eyeglass Configuration Replication Job Audit  

After the Eyeglass Configuration Replication Job has completed, Eyeglass performs an audit to compare source and destination configuration and ensure that replicated configurations are identical

Enabled

Indicates whether the Eyeglass Configuration Replication Job is enabled


Last Successful Readiness Check

Date/time when Eyeglass last successfully ran the Readiness Check Job



Zone Readiness

The Zone Readiness DR Failover Status provides you with a quick and easy way to assess your DR Status for an Access Zone Failover.   The Zone Readiness check is performed for both directions of a replicating Isilon cluster pair to ensure that you have your status not only for failover but also for failback.  



The Zone Readiness Status will be one of the following:

Status

Description

OK

All Required and Recommended conditions that are validated by Eyeglass software have been met.

WARNING

One or more Recommended conditions that are validated by Eyeglass software have not been met.

Warning state does NOT block failover.

Review the Access Zone Failover Guide Recommendations to determine impact for recommendations that have not been met.

ERROR

One or more of the Required conditions that are validated by Eyeglass software have not been met.

Error state DOES block failover.

Review the Access Zone Failover Guide Requirements for failover to determine resolution for these error conditions.

FAILED OVER

This Access Zone on this cluster has been failed over.

You will be blocked from initiating failover for this Access Zone on this Cluster.


IMPORTANT:  Not all conditions are validated by Eyeglass software.  Please refer to the Eyeglass Access Zone Failover Guide for complete list of requirements and recommendations.


Notes:

  1. For the case where the Target cluster pool that has the Eyeglass hint mapping for failover does not have a Smartconnect Zone defined:

    1. On Failover the Access zone will be in Warning state due to SPN inconsistencies

    2. On Failback it will not have the FAILED OVER status displayed

  2. For the case where there is no Eyeglass Configuration Replication Job enabled in an Access Zone there will be no entry in the Zone Readiness table for that Access Zone.


The DR Failover Status is based on Status for each of the following areas for Access Zone Failover:

OneFS SyncIQ Readiness

Have the SyncIQ policies in the Access Zone been successfully run and are they in a supported configuration.

Eyeglass Configuration Replication Readiness

Have the Eyeglass Configuration Replication jobs in the Access Zone been successfully run to sync configuration data for all policies that are members of the Access Zone.

SPN Readiness

Is Active Directory delegation completed for cluster machine accounts  to detect missing  SPNs and remediate existing and newly created Smartconnect Zones as short and long SPN’s created for cluster Active Directory machine accounts.

Smartconnect Zone Failover Mapping Readiness

Validation that confirms if all IP pools in the access zone have an Eyeglass  hint (smart connect alias using igls syntax).  Each Smartconnect Zone name associated to the IP pools (and any smartconnect aliases) must be mapped to a target cluster IP pool prior to any failover.  This ensures all smart connect names used to access sour cluster data will failover to a target cluster IP pool.  It is best practice and requirement to create IP pools in matched pairs on source and destination cluster.

Smartconnect/IP Pool Readiness

Smartconnect/IP Pool Failover Readiness provides the status of whether the IP pool is ready for the failover or, has been already failed over.   It also verifies each IP pool as a smartconnect name applied.  This validation will be used for IP pool based failover in addition to access zone failover where all pools must have a smartconnect name defined.

Zone Configuration Replication Readiness

Has the Eyeglass Zone Configuration Replication job been successfully run to create target cluster access zones that don't already exist for configuration sync complete.

Target Cluster Reachability

Is Eyeglass able to connect to the Failover Target Cluster using API

Date-Time Validation

Are the date-time differences between nodes and between eyeglass and the clusters, within an acceptable range that will not affect SyncIQ operations.

Zone Path Validation

Zone Path Validation provides the status of whether access zones have colliding paths. Status of OK indicates that the access zone paths have no conflicts. Status of ERROR indicates that this access zone collides with another access zone's path.


FQDN Alias Validation

If cluster was added to Eyeglass with FQDN smartconnect name for management, this smartconnect zone must have an igls-ignore hint applied to avoid a failover impacting Eyeglass access. Error means no igls-hint was found on the IP pool for smartconnect zone used for cluster management. ok means igls-ignore hint was found.



IMPORTANT

By default the Failover Readiness job which populates this information is disabled.  Instructions to enable this Job can be found in the Eyeglass Isilon Edition Administration Guide.  

Note: If there are no Eyeglass Configuration Replication Jobs enabled there is no Failover Readiness Job.

Preparation and planning instructions for Zone Readiness can be found in the Access Zone Failover Guide:

Requirements for Eyeglass Assisted Access Zone Failover

Unsupported Data Replication Topology

Recommended for Eyeglass Assisted Access Zone Failover

Preparing your Clusters for Eyeglass Assisted Access Zone Failover


Additional information for Zone Readiness criteria is provided in the following sections.

Zone Readiness - OneFS SyncIQ Readiness

The OneFS SyncIQ Readiness criteria is used to validate that the SyncIQ policies in the Access Zone has been successfully run and that they are in a supported configuration.   You will find one entry per SyncIQ Policy in the Access Zone.  For each SyncIQ Policy the following checks are performed:  















SyncIQ Policy Check

Notes:

Policy Hot/Hot Validation

For Hot-Hot (Active-Active data) replication topology, validate that there is a dedicated Access Zone for each replication direction.

Policy Zone Path Check

  • Policy Source Path Check

  • Policy Target Path Check

Validate that the SyncIQ Policy(s) source root and target directories are at or below the Access Zone Base Directory.

Policy Source Nodes Restriction

Validate Isilon best practices that recommend that SyncIQ Policies utilize the Restrict Source Nodes option to control which nodes replicate between clusters.

Policy Hostname Validation

Validate the the SyncIQ Policy target host hostname is associated with a subnet pool that is NOT going to be failed over

Corrupt Failover Snapshots

Validate that Target Cluster does not have an existing SIQ-<policyID>-restore-new or SIQ-<policyID>-restore-latest snapshot from previous failovers/synciq jobs for the Policy

System Zone Config Restriction

Validate that all shares, exports and alias have been created in the Access Zone that is being failed over.  It is not supported to have shares, exports and alias with a path that is outside (higher in the file system) than the Access Zone base path.

Policy Enabling

Validate that the SyncIQ Policy is enabled in OneFS.

Policy Status

Validate that the SyncIQ Policy is not in error state in OneFS.

Policy Local Target Validation :

  • Duplicate SyncIQ Local Targets

Validate that there is only 1 Local Target per SyncIQ policy.

  • Policy Local Target Validation:

  • Target Writes Disabled

Validate that the target folder of SyncIQ policy has writes disabled.


Zone Readiness - Eyeglass Configuration Replication Readiness

The Eyeglass Configuration Replication Readiness criteria is used to validate that the Eyeglass Configuration Replication jobs in the Access Zone have been successfully run, to sync configuration data for all policies members of the Access Zone.  For each Eyeglass Configuration Replication Job in the Access Zone, the following check is performed:



<Eyeglass Configuration Replication Job Name>

Validate that the Eyeglass Configuration Replication Job status is not in error state.


Note: With both enabled and disabled Eyeglass Configuration Jobs in the Access Zone, the Eyeglass Configuration Replication Readiness validation will only display status for the Enabled jobs.

Zone Readiness - SPN Readiness

The SPN Readiness criteria is used to:

  1. Validate that the Active Directory delegation is completed for cluster machine accounts,  

  2. Detect missing  SPNs

  3. Remediate existing and newly created Smartconnect Zones as short and long SPN’s created for each  cluster Active Directory machine account.  

This check is done for each domain for which each cluster is a member.


Note: For the case where the Isilon Cluster is not joined to Active Directory, the SPN Readiness will show the following:

  • For OneFS 7.2 the SPN Readiness check is displayed with message “Cannot determine SPNs”

  • For OneFS 8 the SPN Readiness check is not displayed in the Zone Readiness window

Zone Readiness - Smartconnect Zone Failover Mapping Readiness

The Smartconnect Zone Failover Mapping Readiness criteria is used to validate that the Smartconnect Zone alias hints have been created between source and target cluster subnet IP pools.  This check is done for each subnet:pool in the Access Zone.

For details on configuring the Smartconnect Zone Failover Mapping Hints, please refer to the Eyeglass Access Zone Failover Guide.




Use the Zone Readiness View Mapping feature to display pools in the Access Zone and how they have been mapped using the Smartconnect Zone Alias hints.


Zone Readiness - View Mapping

Use the Zone Readiness View Mapping link to display the subnet:pool mappings with configured hints for the Access Zone.


Zone Readiness - Zone Configuration Replication Readiness

The Zone Configuration Replication Readiness criteria is used to validate that the Zone Configuration Replication jobs in the Access Zone have been successfully run to create target cluster Access Zones that don't already exist for configuration sync complete.  



Zone Readiness - Target Cluster Reachability

The Target Cluster Reachability criteria is used to validate that Eyeglass is able to connect to the Failover Target Cluster using Onefs API.


 

Zone Readiness - Date-Time Validation

The Date-Time Validation is used to validate that the time difference between the cluster nodes, and between clusters and Eyeglass are within an acceptable range that will not affect SyncIQ operations.  SyncIQ commands like re-sync prep can fail if the time between cluster nodes is greater than the time between the eyeglass vm and the cluster with regards to latency of issuing the API call (a scenario where a node returns a timestamp for a step status message is earlier than the beginning of the Re-sync prep request).  API calls can return different completion times between clusters.  Differences here can cause resync prep failover commands to fail, if the difference between Eyeglass and the source cluster is greater than the time it takes for a resync prep command to complete.

NOTE: This condition is rare but hard to detect manually and Eyeglass can now detect this condition.  It's best practise to use NTP on clusters and Eyeglass appliance.  This allows failover logs and the new feature in release 1.8 or later to collect cluster SyncIQ reports for each step and append to the failover log. This will make debugging multi step multi cluster failover simpler.  This process will require time to be synced.

For each Cluster the following checks are done:  


Date-Time Validation

Notes:

Nodes Validation

Validation that the maximum time difference between the nodes of a cluster is less than the time required for the cluster node time request made by Eyeglass to complete.

Eyeglass & Clusters Validation

Validation that the earliest node time for a cluster and the Eyeglass appliance time are less than the time required for the cluster node time request made by Eyeglass to complete plus a default additional skew factor (default 1s).

Executed if Nodes Validation is OK.


Zone Path Validation

Zone Path Validation provides the status of access zones. Status of OK indicates that the access zone paths have no conflicts. Status of ERROR indicates that this access zone collides with another access zone's path.

Screen Shot 2016-06-12 at 8.29.02 PM.png

FQDN Alias Validation

If cluster was added to Eyeglass with FQDN smartconnect name for management, this smartconnect zone must have an igls-ignore hint applied to avoid a failover impacting Eyeglass access. Error indicates that no igls-hint was found on the IP pool for the smartconnect zone used for cluster management. OK indicates that  igls-ignore hint was found.

Script Engine

Script Engine Overview

The new script engine feature provides an icon on the desktop that provides the following functions:

  1. Save scripts

  2. Create new scripts

  3. Activate and deactivate scripts for failover

  4. Test scripts in simulated failover scenario

  5. Debug and edit scripts

  6. Scripts support in bash, Nodejs, python languages

  7. Run time variables provide access to failover meta data, to complete simplified scripts that leverage variable replacement to handle many different failover scenarios

  8. Pre-failover - shutdown applications, unmount

  9. Unified scripts - handles either failover or failback with a single script that can handle logic for either operation

  10. Post-failover scripts - runs when target cluster is writeable unmount, remount or mount only logic and application start up

Typical script use cases

Many failover scenarios depend on extra steps performed on devices, software, and infrastructure external to the NAS cluster.  This tasks can now be automated with output captured and appended to Eyeglass failover logic logs.

  1. DNS updates post failover for Smartconnect zone CNAME editing

  2. NFS host mount and remount automation

  3. DNS server cache flushing

  4. Application bring up and down logic to start applications post failover

  5. Send alerts or emails

  6. Run API commands on 3rd party equipment.  (i.e. load balancer, switch, router or firewall)

  7. Shutdown an application

  8. IP Load balancing solution and storage layer failover for web tier and storage tier dependencies

The screenshot below shows the editor, admin console for script editing activation and testing.

Screen Shot 2016-04-15 at 7.14.47 PM.png

Screen Shot 2016-04-15 at 7.14.55 PM.png

Script Engine Admin Procedures

Script Library - store many scripts in the library but only activate some using the enable/disable menu to enable or disable a script for one or more failover mode:

  • Add new scripts or delete existing scripts from the library

  • Select a script to edit

    • Numbered lines are for easier script editing and debugging

    • Select a script click Test to see how it performs

      • You must select a failover mode and a SyncIQ policy (it won’t be failed over). This is done to pass variables that might be used for this failover into the script test to allow easier script that can handle multiple policies

    • Decide if the script should be pre script unified script or post failover script and place in the correct folder to ensure it executes at the right location in the failover logic

    • Enable or disable scripts for each failover mode as required

NOTE: All scripts enabled run for all failovers using that mode, so logic needs to handle each policy, access zone option or ensure logic does not run when not required.

Screen Shot 2016-04-15 at 7.14.55 PM.png

Screen Shot 2016-04-15 at 7.14.47 PM.png

Screen Shot 2015-11-06 at 7.35.38 AM.png

Screen Shot 2015-11-06 at 7.35.50 AM.png

Script Engine CURL Tips

If you are using curl as the method to automate with the api, be aware of the following

  1. Curl -k will be required since the API is using a self signed certificate This is not added to CURL command with CURL builder interface and should be added to your curl command

  2. To avoid 411 response for content length or body error from API server add -d ""   to the CURL command Generated by the CURL builder interface of the API explorer

Screen Shot 2016-04-14 at 4.53.18 PM.png

Script Engine Understanding Remote Execution to Hosts


Video: How to Remount Exports Automation

How to use Script Engine to remount Linux Exports post Failover

ssh Passwordless Login to Remote Hosts

A common use case is running a script locally on Eyeglass to take an action on a remote Linux host, or running a command on a remote host to complete failover.   When using ssh from a supported language, you can use bash and .ssh keys to avoid passwords with follow these steps.

Note: bash scripts run as the sca user, when they execute, this user also owns Eyeglass files and processes.

  1. Ssh to eyeglass appliance

  2. Sudo -s (to switch to root)

  3. Enter admin password

  4. Cd /opt/superna  (this is the home directory for the sca user used by eyeglass processes)

  5. Create directory /opt/superna/.ssh/id_rsa

  6. Type ‘ssh-keygen -t rsa’  do not set a password and accept all default prompts but enter a path of /opt/superna/.ssh/id_rsa

  7. Now set ownership on files for remote execution all scripts run as the sca user

    1. Su sca

    2. Ssh user@remotehost (creates known_hosts file for target host, answer yes to accept ssh ID)

    3. Exit (you are now root again)

    4. Cd /opt/superna/.ssh

    5. Chown sca *

    6. Chgrp users *

  8. ssh User@remotehost mkdir -p .ssh   (User is the user that has access to the script that must execute,  remotehost is dns or host name of remote linuxhost ) - this will create .ssh if it does not already exist

  9. cat /opt/superna/.ssh/id_rsa.pub | ssh User@remotehost 'cat >> .ssh/authorized_keys' (this places pub ssh keys into the remote users .ssh authorized keys file to allow passwordless login from a script)

  10. Enter password for User on remotehost

  11. Test SCA remote ssh

    1. Su sca

    2. Ssh user@remotehost (if no pwd requested the setup is complete)  

  12. done.

Access Zone Example bash script that uses ssh keys to remotely execute a command

The Access Zone Based failover preserves smartconnect zone names across failovers, which only requires an unmount and remount of the same FQDN smartconnect zone name.   This means the failover logic can be used for failover or failback since it's’ the same operation.

This sample solution uses a script that is unique on each host with the same name. Example; remount.sh placed in the user home directory used with the ssh remote execution rsa pub key.  (see steps above to set up Eyeglass for ssh passwordless login to remote hosts).

Remote host script example:

Script name remount.sh placed in the home directory of the user account setup for ssh login automation from the Eyeglass appliance

#remount script

echo "remounting filesystem post failover"

umount -fl /mnt/data

mount -a

mount | grep "/mnt/data"

Eyeglass Access Zone failover and failback Script Example

#!/bin/bash

#  

#  Script: remount.sh

# Purpose: unmount and remount from /etc/fstab persistent mounts post failover script, depends on remote script on remote host to execute

# Location: eyeglass post failover scripting paste into script engine and enable for Access zone based failovers

#

echo starting unmount remount remotely called script on remote hosts

echo source-cluster: $SOURCE

echo zone data: $ZONE_DATA

me=$(whoami)

echo name of user that runs the script : $me

# The variables set during failover include many variables and attributes of the access zone selected for failover that can be used to grep and select when # to apply failover logic.  This can be used to group which hosts to automate failover based on the access zone selected for failover.  The example below  

# can be expanded to be used with per syncIQ policy names using the same grep solution and variables shown in the eyeglass echo example scripts.

#  This string "source":{"name":"xxxx"   replace the xxxx with the name of the zone you want to failover, hint you can use test feature in script engine to

# run the sample scripts with your clusters and access zones to see which string to grep for test see bolded section used below to select and access zone

# zone data: #{"source":{"name":"data","subnets":[{"name":"subnet0","smartConnectServiceIp":"172.31.1.201","pools":[{"name":"subnet0:dfsdata","ranges":"172.31#.1.113-172.31.1.113","smartConnectZoneName":"dfsdata-dr.ad1.test","smartConnectAliases":["igls-ignore"]},{"name":"subnet0:userdata","ranges":"17#2.31.1.111-172.31.1.111","smartConnectZoneName":"userdata.ad1.test","smartConnectAliases":["igls-user-prod"]}]}]},"target":{"name":"data","subnet#s":[{"name":"subnet0","smartConnectServiceIp":"172.31.1.200","pools":[{"name":"subnet0:dfsdata","ranges":"172.31.1.112-172.31.1.112","smartConn#ectZoneName":"dfsdata.ad1.test","smartConnectAliases":["igls-ignore"]},{"name":"subnet0:userdata","ranges":"172.31.1.110-172.31.1.110","smartCon#nectZoneName":"igls-original-userdata.ad1.test","smartConnectAliases":["igls-user-prod"]}]}]},"poolMap":[{"sourcePool":{"name":"subnet0:userdata","#ranges":"172.31.1.111-172.31.1.111","smartConnectZoneName":"userdata.ad1.test","smartConnectAliases":["igls-user-prod"]},"targetPool":{"name":"s#ubnet0:userdata","ranges":"172.31.1.110-172.31.1.110","smartConnectZoneName":"igls-original-userdata.ad1.test","smartConnectAliases":["igls-user-#prod"]}}]}

 

if (echo "$ZONE_DATA" | grep -q '"source":{"name":"data"'); then

   echo found zonename

# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)

   rc=$(ssh root@linux ./remount.sh)

# remote script runs and returns output, can be output below to be captured in the failover log

   echo result of host script was: $rc

else

   echo did not find zonename to process

fi

Adding additional Script Language support to the appliance

  1. Nodejs can be added by: (https://nodejs.org/en/docs/)

    1. ssh as root to the appliance

    2. then run zypper install npm

    3. answer yes (requires internet access)

  2. bash - pre-installed in the OS (2.7.8)

  3. Python - pre-installed in the OS (2.7.8)

Script run time variables

The following variables can be used to pass in values to a script to handle various policies or scenario’s using substitution of the values

  1. source -  Represents the name of the source cluster  of the SyncIQ policy

  2. target - Represents the name of the target cluster of the SyncIQ policy

  3. policy - Used to return metadata about the policy itself see example output below (NodeJS - Example Output)

  4. failover_type - syncIQ, dfs or access zone

  5. zone_data - zone data about the access zone that can be used example smartconnect zone list and zone alias for DNS updates

Sample Execution Rules and Overall Failover Job status Impact

  1. They run after all Eyeglass automation

  2. One or more scripts can be enabled per failover type and  both will execute in series during failover

  3. Return code provided by the script should return 0 to  indicate  the script had no errors and completed successfully

  4. Return code > 0 indicates an error

  5. Return codes can be set to any value and number and meaning in the script, Eyeglass takes no actions based on specific return codes

  6. Return codes are logged in the failover log for post failover review and debugging

  7. Script output is captured in the failover log.  It is best practice to use the echo command  to output script execution so that it's included in the Failover log

  8. If running two or more scripts each script should have discrete function to complete AND should not have any dependency on other scripts.  No ability to have IF script return code of X then 2nd script do Y exists

  9. Put host side script automation into its own script

  10. Put DNS automation logic into it’s own script

  11. Put application specific logic into it’s own script

  12. If any scripting logic needs dependant logic then a single script should be used for all functions

  13. Return code > 0 will failover the overall job status

Sample Scripts in the Library

NodeJS - Example Script

#!/usr/bin/env node

console.log("these are the environment variables");

console.log("source", process.env.SOURCE);

console.log("target", process.env.TARGET);

console.log("type", process.env.FAILOVER_TYPE);

console.log("zone data", process.env.ZONE_DATA);

console.log("policy data", process.env.POLICY_DATA);

NodeJS - Example output

hese are the environment variables

source {"pass":"password!","port":8080,"ip":"172.31.1.105","name":"Cluster2-7201","guid":"005056ba72edf6450c552312a728d3a22a23","user":"admin"}

target {"pass":"password!","port":8080,"ip":"172.31.1.104","name":"Cluster-1-7201","guid":"005056ba34580f410c55fd077989478a3821","user":"admin"}

type SYNCIQ

zone data

policy data [{"name":"dfs9_mirror","targetIp":"172.31.1.104","targetHostname":"172.31.1.104","sourcePath":"/ifs/data/policy1","targetPath":"/ifs/data/policy1","enabled":true,"shares":[],"exports":[],"zones":[],"lastJobStatus":"running","lastSuccess":"null","lastStarted":"1446812101","schedule":"every 1 days every 5 minutes between 12:00 AM and 11:59 PM","sourceExludePaths":[],"sourceIncludePaths":[]}]

Process completed with return code: 0

Python - Example Script

#!/usr/bin/env python


import os

print "these are the environment variables"

print os.environ['SOURCE']

print os.environ['TARGET']

print os.environ['FAILOVER_TYPE']

print os.environ['ZONE_DATA']

print os.environ['POLICY_DATA']


Python - Example output

these are the environment variables

{"pass":"password!","port":8080,"ip":"172.31.1.105","name":"Cluster2-7201","guid":"005056ba72edf6450c552312a728d3a22a23","user":"admin"}

{"pass":"password!","port":8080,"ip":"172.31.1.104","name":"Cluster-1-7201","guid":"005056ba34580f410c55fd077989478a3821","user":"admin"}

SYNCIQ

[{"name":"dfs9_mirror","targetIp":"172.31.1.104","targetHostname":"172.31.1.104","sourcePath":"/ifs/data/policy1","targetPath":"/ifs/data/policy1","enabled":true,"shares":[],"exports":[],"zones":[],"lastJobStatus":"finished","lastSuccess":"1446812701","lastStarted":"1446812701","schedule":"every 1 days every 5 minutes between 12:00 AM and 11:59 PM","sourceExludePaths":[],"sourceIncludePaths":[]}]

Process completed with return code: 0


Bash - Example Script

#!/bin/bash


echo these are the environment variables

echo source: $SOURCE

echo target: $TARGET

echo failover type: $FAILOVER_TYPE

echo zone data: $ZONE_DATA

echo policy data: $POLICY_DATA


Bash - Example output

hese are the environment variables

source: {"pass":"password!","port":8080,"ip":"172.31.1.105","name":"Cluster2-7201","guid":"005056ba72edf6450c552312a728d3a22a23","user":"admin"}

target: {"pass":"password!","port":8080,"ip":"172.31.1.104","name":"Cluster-1-7201","guid":"005056ba34580f410c55fd077989478a3821","user":"admin"}

failover type: SYNCIQ

zone data:

policy data: [{"name":"dfs9_mirror","targetIp":"172.31.1.104","targetHostname":"172.31.1.104","sourcePath":"/ifs/data/policy1","targetPath":"/ifs/data/policy1","enabled":true,"shares":[],"exports":[],"zones":[],"lastJobStatus":"running","lastSuccess":"null","lastStarted":"1446812701","schedule":"every 1 days every 5 minutes between 12:00 AM and 11:59 PM","sourceExludePaths":[],"sourceIncludePaths":[]}]

Process completed with return code: 0


Consolidated Post Failover & Failback Script (Node.JS)

#!/usr/bin/env node


var exec = require('child_process').exec;

var child;

var mycmd = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failover.sh"';

var mycmd2 = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failback.sh"';


// refresh name resolution


if (process.env.SOURCE.indexOf('cluster20') !== -1)

{

console.log("Failover");

child = exec(mycmd, function (error, stdout, stderr)

{

console.log('result output: ' + stdout);

console.log('result errors: ' + stderr);

}

);

}

else

{

   console.log("Failback");

   child = exec(mycmd2, function (error, stdout, stderr)

{

console.log('result output: ' + stdout);

console.log('result errors: ' + stderr);

}

);

}


var node_ssh = require('node-ssh');

var ssh = new node_ssh();


var cmd1 = "ls -l /proc/*/cwd | grep /mnt/z01-nfs01 | awk '{print $9}' | grep -o '[0-9]*' | xargs kill -s 9",

   cmd2 = 'umount -fl /mnt/z01-nfs01',

   cmd3 = 'mount -t nfs -o vers=3 cluster20-z01.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01',

   host = '172.16.81.161',

   user = 'root',

   pass = 'GoSuperna!';

console.log('Executing command: ' + cmd1);

console.log('Executing command: ' + cmd2);

console.log('Executing command: ' + cmd3);


console.log(' On host: ' + host);

ssh.connect({

  host: host,

  username: user,

  password: pass

}).then(function() {

   ssh.execCommand( cmd1, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

}).then(function() {

   ssh.execCommand( cmd2, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);    

   }).then(function() {

   ssh.execCommand( cmd3, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

       console.log('ssh operation complete');

       ssh.end();

});

});

   });

});


Consolidated Post Failover & Failback Script (Node.JS) - example for multiple Zone Access

The following node.js script is the example for handling multiple zone access:

// 1st Part refresh name resolution

var exec = require('child_process').exec;

var child;

var mycmd1 = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failover-z01.sh"';

var mycmd2 = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failover-z03.sh"';

var mycmd3 = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failback-z01.sh"';

var mycmd4 = 'echo 41d7297b7c79651bb94dcf676538f9b3b5ed6e8ed25e04c6ee38d14269e022cc | sudo -S su root -c "sh /opt/superna/sca/failback-z03.sh"';



if (process.env.SOURCE.indexOf('cluster20') !== -1)

{

  console.log("Failover");

  if (process.env.ZONE_DATA.indexOf('zone01') !== -1)

  {

  child = exec(mycmd1, function (error, stdout, stderr)

  {

  console.log('result output: ' + stdout);

  console.log('result errors: ' + stderr);

  }

  );

  }

  if (process.env.ZONE_DATA.indexOf('zone03') !== -1)

  {

  child = exec(mycmd2, function (error, stdout, stderr)

  {

  console.log('result output: ' + stdout);

  console.log('result errors: ' + stderr);

  }

  );

  }

}


if (process.env.SOURCE.indexOf('cluster21') !== -1)

{

  console.log("Failback");

  if (process.env.ZONE_DATA.indexOf('zone01') !== -1)

  {

  child = exec(mycmd3, function (error, stdout, stderr)

  {

  console.log('result output: ' + stdout);

  console.log('result errors: ' + stderr);

  }

  );

  }

  if (process.env.ZONE_DATA.indexOf('zone03') !== -1)

  {

  child = exec(mycmd4, function (error, stdout, stderr)

  {

  console.log('result output: ' + stdout);

  console.log('result errors: ' + stderr);

  }

  );

  }

}

// 2nd Part refresh mount


var node_ssh = require('node-ssh');

var ssh = new node_ssh();


var cmd1 = "ls -l /proc/*/cwd | grep /mnt/z01-nfs01 | awk '{print $9}' | grep -o '[0-9]*' | xargs kill -s 9",

   cmd2 = 'umount -fl /mnt/z01-nfs01',

   cmd3 = 'mount -t nfs -o vers=3 cluster20-z01.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01',

   cmd4 = "ls -l /proc/*/cwd | grep /mnt/z03-nfs01 | awk '{print $9}' | grep -o '[0-9]*' | xargs kill -s 9",

   cmd5 = 'umount -fl /mnt/z03-nfs01',

   cmd6 = 'mount -t nfs -o vers=4 cluster20-z03.ad1.test:/ifs/data/zone03/z03-nfs01 /mnt/z03-nfs01',

   host = '172.16.81.161',

   user = 'root',

   pass = 'GoSuperna!';


console.log(' On host: ' + host);


if (process.env.ZONE_DATA.indexOf('zone01') !== -1)

  {

   console.log('Executing command: ' + cmd1);

   console.log('Executing command: ' + cmd2);

   console.log('Executing command: ' + cmd3);   

   ssh.connect({

       host: host,

       username: user,

       password: pass

   }).then(function() {

   ssh.execCommand( cmd1, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

   }).then(function() {

   ssh.execCommand( cmd2, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

       

   }).then(function() {

   ssh.execCommand( cmd3, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

       console.log('ssh operation complete');

       ssh.end();

   });

});

   });

   });

}


if (process.env.ZONE_DATA.indexOf('zone03') !== -1)

  {

   console.log('Executing command: ' + cmd4);

   console.log('Executing command: ' + cmd5);

   console.log('Executing command: ' + cmd6);   

   ssh.connect({

       host: host,

       username: user,

       password: pass

   }).then(function() {

   ssh.execCommand( cmd4, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

   }).then(function() {

   ssh.execCommand( cmd5, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

       

   }).then(function() {

   ssh.execCommand( cmd6, {

       stream: 'both'

   }).then(function(result) {

       console.log('result output: ' + result.stdout);

       console.log('result errors: ' + result.stderr);

       console.log('ssh operation complete');

       ssh.end();

   });

});

   });

   });

}


Superna Eyeglass API Guide

It is now possible to failover using external applications such as VMware srm or a script called from an application, web page or curl command.  The API Guide covers api explorer to automatically build CURL commands that allows a single command failover over a policy or entire access zone.  This also allows script engine logic to run if enabled at the end of failover.  

The API and example VMWare integration for failover is explained in the Superna Eyeglass API guide.

Services

An Icon used for remote services connecting to Eyeglass. As more agent and external solutions are developed, the services window allows API token based authentication to show which services are using Eyeglass data, allows port state and deletion  services no longer required.

  1. The first service is UIM probe for sending DR and Isilon Alarm data to CA UIM.  UIM CA Isilon Probe for Eyeglass - follow this product documentation link for installation and configuration details http://documentation.superna.net/eyeglass-uim-probe

  2. Eyeglass clustered agent used with Ransomware and Easy Auditor products

Network Visualization

Overview

A new way to view Isilon clusters, DR status, and jump to the DR Dashboard.  The Network Visualization feature allows you to visualize DR and cluster replication. This first release offers the first in several enhancements aimed at visualizing data, data flows, and storage across one or more Isilon clusters.

This view indicates which clusters are replicating to each other and direction.  For each cluster, failover readiness status for all failover types is summarized. Any failover readiness error will show as a red arrow from the source to target cluster (failover direction).  Warnings are displayed with an orange arrow.  In the case where there are not errors or warnings the arrow will be green for active replication direction (failover direction) and grey for the failed over (inactive) direction. This simplifies monitoring many clusters replicating.

To view Network Visualization:

  1. Open the Icon

Screen Shot 2017-03-24 at 7.29.29 PM.png

  1. Zoom in or out to navigate depth of view

  2. Click and hold to drag the visual view objects

  3. Click a cluster to get view of active Sync Data on the cluster viewed by Failover mode and status

  4. Click on hyperlink to hump to DR Dashboard Directly from the Network Visualization window


Screen Shot 2016-04-15 at 7.28.21 PM.png

RPO Reporting and Trending Feature Guide

See Eyeglass Isilon Edition Recovery Point Objective Trending and Reporting

End User Offline Detection

The Eyeglass web page detects when the browser does not have internet access and does not attempt to connect to Twitter or the Superna support (superna.help) site.  In this case the Twitter icon on the Eyeglass web page is removed and the Help button is inactive.

Eyeglass Reporting

Eyeglass offers these reports

  1. Cluster reports that cover configuration

  2. RPO reports that include syncIQ performance and backup details on SyncIQ performance for backup monitoring of policies used for backup only versus DR.

    1. See RPO guide for details

  3. Cluster Usage reporting (requires Cluster Storage Monitor License keys).

    1. See Cluster Storage Monitor admin guide for details

Reports are now available from the Reports on Demand Icon.

Screen Shot 2017-03-24 at 6.27.59 PM.png


Eyeglass Cluster Config Reports

Eyeglass collects key cluster settings for the Isilon clusters that it is managing, and makes them available in an html report format by email or on demand. Use the contents of this report to automate and keep up to date the documentation of cluster configuration for DR and your DR run book.  This information is also valuable for IT planning, cluster expansion, networking and security



What’s New

  1. New in 1.9 cluster reports includes:

    1. DNS settings for onefs 8 now added to cluster reports

    2. Pools/Subnets in all groupnets shows up (with access zones) in onefs 8 cluster reports. Also added zone name to pools in onefs 7 reports



Cluster Configuration Email Report

By default, the full cluster html report containing key cluster settings is emailed daily, at midnight, to all users who have been configured for email notification.  

Cluster Configuration Report On-Demand

To generate a Cluster Configuration Report on-demand:

  1. Login to the Eyeglass web page.

  2. Open the Reports on Demand window.

  3.  Select Create New Report/Cluster Report.

  4.  A confirmation dialog is displayed once the request has been submitted.  

  5.  To monitor the progress of the report generation from the Running Jobs window, you can select View Running Jobs in the confirmation dialog.



  6. When the report has been generated it will appear in the Cluster Reports list.  Report Name format includes the name of the cluster that the report is about <cluster name>_cluster_report_<timestamp>.html.

  7. Select the Open link to view the report.


8. The report opens in a new Cluster Report Viewer window with a table of contents linked to each section of the report.

9. To save the report as PDF, select the Print/Save button in the Cluster Report Viewer window and then select the Save as PDF option.

To remove old reports:

  1. Use the check box to select reports for deletion

  2. Click Delete Selected.

Cluster Networking For Failover and Cluster Reports

Cluster networking is now added to cluster reports and can also be viewed in the zone inventory tree (see examples below).  This information is key to assisting with Access Zone failover. Each Access Zone maintains a mapping to subnet pools and configuration data.

See the Failover Design Guide  (Eyeglass Failover Design Guide) for instructions on creating hints to allow Eyeglass to failover networking with smartconnect zone migration, SPN updates, and pool mapping.  This information is also needed to assist with Run Book Robot configuration within an Access zone and will allow customers to verify complete failover and failback operations between sites using Eyeglass automation.


Screen Shot 2015-08-07 at 8.59.38 PM.png

Screen Shot 2015-08-07 at 8.59.49 PM.png

Role Based Access Controls RBAC

This new feature is found under the User Roles icon on the desktop.  It allows user or groups to be mapped to Eyeglass roles. This feature can be used  to create various user roles within Eyeglass from read-only access to backup to DR failover roles.  

The complete guide on how to configure can be found in  Role Based Access Control And Authentication.

Configure Email, Twitter and Slack for  Notifications  of Eyeglass Monitoring Events

The Steps to configure Email Notification are found in the Quickstart Installation Guide here.

Additional information for Microsoft Exchange can be found in the tech note here.

The Steps to configure secure Twitter Notification for alerts can be found in the tech note here.

To configure Slack channel Isilon events AND eyeglass events:

  1. Goto https://superna.slack.com/apps/manage/custom-integrations   (for your domain) as admin

  2. Select Incoming WebHooks

  3. Configure as per screenshot pick the channel to receive alarms

  4. Cut and past webhook url for configuration in eyeglass

Screen Shot 2016-06-12 at 8.11.49 PM.png

  1. Login to eyeglass

  2. Open notification center

  3. Select Slack tab

  4. Set the webhook url created above

  5. Set the level alarms that will be sent to the channel

Note: This includes DR events from eyeglass and Isilon events that are polled from the cluster

Screen Shot 2016-06-12 at 8.14.52 PM.png

DR Design Guides with Eyeglass

The Steps to configure, plan and implement DR operations with Eyeglass is located here in a dedicated chapter on DR options including Run Book Robot implementation and design.  

  1. Eyeglass Start Here first

  2. Eyeglass Failover Design Guide

Eyeglass Shell Key Features

The Eyeglass WebShell console provides the ability to open an OS shell right from the Eyeglass web page.  You can now reach through to the Eyeglass appliance Linux OS command line and execute Eyeglass scripts and OS commands.

This feature will be used for future automation and custom scripts to automate tasks on Isilon clusters.   Advanced configuration changes to the Eyeglass appliance will also be made through the CLI.  Each update or release will expand the CLI capabilities.

Open the Eyeglass Shell

  1. From the Eyeglass web page, open the Eyeglass Main Menu.

  2. Select Eyeglass Shell.

  3. The Eyeglass Shell window opens with a login prompt for the OS login.

  4. Enter a valid local OS user name and password at the prompt (except root).

  5. Once logged in, you can perform any OS linux commands you could execute when ssh to the appliance.

Note: It is not possible to copy into or from the Eyeglass Shell.


Eyeglass CLI Commands

The following Eyeglass CLI commands are available and can be executed directly from the Eyeglass shell or any ssh session to the appliance.  


Ransomware CLI commands


Use this command to enable SMB shutdown if root Isilon user is detected with ransomware behaviour.  Cluster wide shutdown of all SMB IO.  NOTE: Root user should never be used to access data since it can access all shares regardless of permissions

igls admin lockroot --lock_root


igls adv failover mode

This command is for a large number of policy failovers.  It changes the default behavior of sequential make writeable and resync prep commands to allow up to 10 parallel make write commands or resync prep commands to be  issued to the cluster at once.  If a job on the cluster finishes, another is sent with the goal of keeping 10 jobs always running on the cluster until failover is complete.

High Speed Failover - Parallel Failover Flag Notes :

  1. Allows make write step and resync prep to run in parallel with up to 10 threads, ensures that 10 policies are submitted to be processed at all times.

  2. Testing has shown these steps for large quantity policy failover can improve failover times 3x to 4x.

  3. Risk of a policy failure increases and new flag will NOT stop the failover in progress. The process will continue to issue api calls to submit all syncIQ policies in the failover job until all have been submitted. This runs the risk of more complex recovery if more than one policy fails to complete its step (Allow Writes OR resync Prep)

igls adv failovermode set --parallel=true    (defaults to false)

igls adv failovertimeout

Display the per step timeout for failover tasks. Advanced setting.  Default 60 minutes.  For very large policies (see this link) can be increased to suggest value of 180 minutes

igls adv failovertimeout get  (returns current value)

igls adv failovertimeout set --minutes 180  (sets)

igls adv full sync

This advanced option should be enabled only after consulting with Superna support first.  It overcomes a scenario where NFS exports are created with FQDN for client lists and the FQDN values are NOT resolvable by the DR or target cluster.  This scenario happens when DHCP leases expire DNS resolution, OR if FQDN values do not resolve any longer, and it's not possible to clean up this condition.    Onefs 8 api behaviour denies the creation of the exports with unresolved FQDN client list entries, and requires the force flag to override cluster rules on export creation.   The force create override flag is disabled by default in Eyeglass to avoid conditions where duplicate exports are created.   

Behaviour

This sync mode will delete all shares and exports found on the target cluster that DO NOT exist on the source.  This creates a full sync.  The default option in Eyeglass will leave any shares or exports found that do not exist on the source.  With this option enabled, all extra config will be deleted to make an exact copy on the DR/target cluster.

igls adv fullsync set --fullsync=<true/false>

Default is false

igls adv runbookrobot

Allows a mode where the export auto create and update is disabled and can be manually created on the Robot policy path, set the export settings with Eyeglass appliance ip address as root client, and other settings can be enabled manually.  Each robot run will no longer create or update the export.

Default is true.

Note: If set to false AND export is not created manually, the runbookrobot mount step must also be set to false as documented here.

igls adv runbookrobot set --createExport=false

igls admin health

Display the overall health status of the Eyeglass appliance.


~> igls admin health                          

                                              

{                                                              

"success": true                                            

}



igls admin appid

Display the appliance id of the Eyeglass appliance.


~> igls admin appid                           

                                                           

{                                                              

"applianceCode": [                                         

    ""                                                     

]                                                          

}

                              


         


igls admin version

Display the Eyeglass component versions.


~> igls admin version                         

                                                           

                                                           

[                                                              

{                                                          

    "release": [                                           

        "38"                                               

    ],                                                    

    "version": [                                           

        "1.3"                                              

    ],                                                    

    "name": [                                              

        "eyeglass_ui"                                      

    ]                                                      

},                                                        

{                                                          

    "release": [                                           

        "34"                                               

    ],                                                    

    "version": [                                           

        "1.3"                                              

    ],                                                    

    "name": [                                              

        "eyeglass_rest"                                    

    ]                                                      

},                                                        

{                                                          

    "release": [                                           

        "64"                                               

    ],                                                    

    "version": [                                           

        "1.3"                                              

    ],                                                    

    "name": [                                              

        "eyeglass_sca"                                     

    ]                                                      

}                                                          

]      




igls alarm active

Retrieve the current active alarm list.  


~> igls alarm active                          

{                                                          

    "sync_key": "Share3-SystemZone",                      

    "code": "SCA0002",                                    

    "severity": "Critical",                               

    "timestamp": 1430350806854,                           

    "source": "Share3-SystemZone",                        

    "message": "Found a replication job where either the sou

rce or destination is not a managed network element.",        

    "extra_data": "{\"info\":\"The replication job for polic

y 'Share3-SystemZone' cannot be created because the target host

cannot be identified.\"}"                                      

}    


Note: To view this list incrementally, you can use the command:


~> igls alarm active | more



igls alarm all

Display the total alarms received in “results”.


~> igls alarm all                             

                                                           

{                                                              

"rows": [],                                               

"alarmsPerPage": "50",                                    

"results": "889"                                           

}                   


igls app restore

Restore Eyeglass data and configuration from Eyeglass Archive.

Note: must be logged in as admin or root user.                                                  


~> igls app restore                           

Usage: igls app restore [OPTIONS] F

igls appliance rediscover


This command should be used when directed by support. It will rebuild the eyeglass database and preserve job status in the jobs icon with release 1.8 or later.     The command will prompt yes to continue.   

Upon completion refresh the UI login screen.  Go to running jobs to see initial discovery job is running to repopulate cluster information in the database inventory icon.

Once completed the job definition screen will show the jobs in previous state and show as pending , where they will run again on next scheduled interval or force run them with run now option.

igls appliance report

(diagnostic log parsing tool run command)  This command is for dark or secure sites where on site log analysis is required.  Summarizes all api, ssh and other errors, config sync analysis, failover analysis of each attempt and success or failure.

  1. Run command: igls appliance report

  2. Wait for the report to complete

  3. See logs report on: https://<eyeglass IP address>/report/

Please refer to document: Eyeglass Backup and Restore



igls adv initialstate

Display and update initial state when Eyeglass creates a new Job.  This command supports changing the initial state for the following Eyeglass Job types: ZONES, AUTO, CUSTOM, QUOTAS.  

~> igls adv initialstate help              

                                                          

show(default):                                                

Displays the initial states for new jobs.                      

                                                           

set --<type>=<state>:                                          

sets a job type to have a specific initial state.             

Valid states are: enabled, disabled.                           

Valid types are: zone,auto,custom,quota

Default: is shown below


Examples:

~> igls adv initialstate show

{

 "ZONES": "USERDISABLED",

 "AUTO": "ENABLED",

 "QUOTA": "ENABLED",

 "CUSTOM": "ENABLED"



~> igls adv initialstate set --custom=disabled                                                              

{                                                              

"success": true                                            

}                 



igls adv PolicyFailover

Enable and disable Eyeglass Configuration Replication task during SyncIQ Policy Failover.


~> igls adv PolicyFailover set --disablereplication=<state>              

                                                          

Valid states are: true, false                           


Examples:

Disable Eyeglass Configuration Replication task during SyncIQ Policy Failover

~> igls adv PolicyFailover set --disablereplication=true

{                                                              

"success": true                                            

}



Enable Eyeglass Configuration Replication task during SyncIQ Policy Failover

~> igls adv PolicyFailover set --disablereplication=false

{                                                              

"success": true                                            

}

Igls adv quotas

  1. igls adv  quotas help

  2. Igls adv quotas (see current values)

  3. Igls adv quota set --quotasync=true  (this enables the feature, false to disable)

  4. Igls adv quota set set --quotasyncdelete=true (defaults disabled, valid values are enabled/disabled/advanced)

    1. Disabled; means no quotas will be deleted after share or export delete, they must be deleted manually

    2. Enabled; when a share is deleted, this mode deletes all quotas in that path unless another share exists in the same path. NOTE: do not enable quota sync delete unless you are sure you can.  This mode will detect a share being deleted and delete all quotas at this path and below in the file system.

    3. Advanced (recommended mode);  when a share is deleted, this mode deletes only eyeglass created quotas in that path unless another share exists in the same path.

Igls adv rundedupe

Disable dedupe setting process while allowing LiveOPS snapshot jobs to execute.  All clusters global command.

igls adv rundedupe set --rundedupe=true/false  (default true)

igls admin schedules

Display and update schedule for Eyeglass tasks. This command supports enabling, disabling and updating the schedule for the following tasks:  Configuration Replication, Eyeglass Reports, Zone Readiness, Runbook Robot

~> igls admin schedules list



[

   {

       "interval": "0 0 * * *",

       "enabled": true,

       "id": "InventoryReport",

       "label": "Eyeglass Reports"

   },

   {

       "interval": "*/15 * * * *",

       "enabled": false,

       "id": "PrintInventoryToSyslog",

       "label": "Print Inventory to Syslog"

   },

   {

       "interval": "*/15 * * * *",

       "enabled": false,

       "id": "Readiness",

       "label": "Zone Readiness"

   },

   {

       "interval": "*/5 * * * *",

       "enabled": true,

       "id": "Replication",

       "label": "Configuration Replication"

   },

   {

       "interval": "0 0 * * *",

       "enabled": true,

       "id": "RunbookRobot",

       "label": "Runbook Robot"

   }

]


Eyeglass Reports

Enable/Disable

To enable/disable the schedule for the Eyeglass Reports, use this command:

igls admin schedules set --id InventoryReport --enabled <true|false>

Examples:

~> igls admin schedules set --id InventoryReport --enabled false

{                                                              

"success": true                                            

}             



~> igls admin schedules set --id InventoryReport --enabled true                                            

{                                                              

"success": true                                            

}                  


Update Schedule

To change the schedule for the Eyeglass Reports use this command.  Valid intervals for reporting are: 1M, 2M, 3M, 4M, 5M, 6M, 10M, 15M, 20M, 30M, 1H, 2H, 3H, 4H, 6H, 8H, 12H, 1D, 7D, 31D.


~> igls admin schedules set --id InventoryReport --interval <interval>


Example:

~> igls admin schedules set --id InventoryReport --interval 7D                                             

{                                                              

"success": true                                            

}                  

Configuration Replication

Enable/Disable

To enable/disable the schedule for Configuration Replication, use this command:


igls admin schedules set --id Replication --enabled <true|false>


Examples:


~> igls admin schedules set --id Replication --enabled false                                                

{                                                              

"success": true                                            

}


~> igls admin schedules set --id Replication --enabled true                                                

{                                                              

"success": true                                            

}   


Update Schedule

To change the schedule for Configuration Replication use this command.  Valid intervals for replication are: 1M, 2M, 3M, 4M, 5M, 6M, 10M, 15M, 20M, 30M, 1H, 2H, 3H, 4H, 6H, 8H, 12H, 1D, 7D, 31D.

igls admin schedules set --id Replication --interval <interval>

Example:


~> igls admin schedules set --id Replication --interval 10M                                                 

{                                                              

"success": true                                            

}    

Runbook Robot Schedule Interval

Use this CLI command to change the interval from once per day.

Enable/Disable

To enable/disable the schedule for Runbook Robot, use this command:

igls admin schedules set --id RunbookRobot --enabled <true|false>

Examples:

~> igls admin schedules set --id RunbookRobot --enabled false                                                

{                                                              

"success": true                                            

}


~> igls admin schedules set --id RunbookRobot --enabled true                                                

{                                                              

"success": true                                            

}   


Update Schedule

To change the schedule for RunbookRobot use this command.  Valid intervals for Configuration Replication are: 1M, 2M, 3M, 4M, 5M, 6M, 10M, 15M, 20M, 30M, 1H, 2H, 3H, 4H, 6H, 8H, 12H, 1D, 7D, 31D.

igls admin schedules set --id RunbookRobot --interval <interval>

Example:

~> igls admin schedules set --id RunbookRobot --interval 10M                                                 

{                                                              

"success": true                                            

}    

Zone Readiness

Enable/Disable

To enable/disable the schedule for the Zone Readiness job, use this command:

igls admin schedules set --id Readiness --enabled <true|false>

Examples:

~> igls admin schedules set --id Readiness --enabled false

{                                                              

"success": true                                            

}             



~> igls admin schedules set --id Readiness --enabled true                                            

{                                                              

"success": true                                            

}                  

Update Schedule

To change the schedule for the Zone Readiness job use this command.  Valid intervals for reporting are: 1M, 2M, 3M, 4M, 5M, 6M, 10M, 15M, 20M, 30M, 1H, 2H, 3H, 4H, 6H, 8H, 12H, 1D, 7D, 31D.

~> igls admin schedules set --id Readiness --interval <interval>

Example:

~> igls admin schedules set --id Readiness --interval 2H                                             

{    

Runbook Robot Mount Export Enable Disable

Default is enabled to mount the cluster and create the test file:

igls adv runbookrobot show (show current value)

igls adv runbookrobot set --mount=true  (default)

igls adv runbookrobot set --mount=false

Advanced commands

igls adv requesttimeout

Description: Sets rest API timeout when cluster or wan responses take longer to return, this value can be increased.

igls adv requesttimeout (displays the timeout value)

igls adv requesttimeout set --inventory <time> (sets the timeout value to <time>)

Example:

igls adv requesttimeout set --inventory 300

igls adv spndelay

Description: used to increase the delay between SPN failover commands, that require domain controller to replicate the delete before the add spn can succeed.   Release 1.8.3 removes the need for this command, by pinning spn failover coammds Toma single node and domain controller.

igls adv spndelay (displays the current setting)

igls adv spndelay set --seconds=<seconds>  (set a delay between delete and create SPN during failover)

Example:

igls adv spndelay set --seconds=10

Cluster Storage Monitor CLI commands

These commands are used to add Smartconnect FQDN to the quota portal authentication UI for end users.

  1.  For checking fqdn list:   igls admin auth

  2.  For adding a new fqdn: igls admin auth add --fqdn <name>

  3.  For changing a fqdn:     igls admin auth modify --fromfqdn <name> --tofqdn <newName>

  4.  For deleting a fqdn:       igls admin auth delete --fqdn <name>

  5.  For deleting all fqdn's:   igls admin auth delete --all true

Cluster Storage Reports schedule CLI commands

See setting and getting schedules in the CLI section above, to specify job id, get and set functions, and enable and disable actions on scheduled reports

Igls admin sched (lists schedules)

Used to enable or disable the daily report for cluster disk usage:

{

       "interval": "0 0 * * *",

       "enabled": true,

       "id": "StorageMonitorReport",

       "label": "Storage Monitor Report"

   }

Used to schedule the daily summary of quota requests and actions for the daily summary:

{

       "interval": "0 0 * * *",

       "enabled": true,

       "id": "QuotaRequestsReport",

       "label": "Quota Requests Report"

   },

RPO Reporting CLI commands

This section contains Eyeglass CLI commands related to the RPO Reporting feature.

igls adv runreports --report_type=rpo

Use this command to manually generate the SyncIQ Job Report and have it emailed. The time that the command is run is the starting time for the report and associated calculations.   Each command example below are the type options

igls adv runreports --report_type=rpo

igls adv skipscreenshots

Use this command to enable or disable RPO chart screenshots in the RPO Report.

To disable screenshots:

igls adv skipscreenshots set --skip=true

To enable screenshots:

igls adv skipscreenshots set --skip=false


CSM Reporting CLI commands

This section contains Eyeglass CLI commands related to the CSM Reporting feature.

igls adv runreports --report_type=csm

Use this command to manually generate CSM Report and have it emailed.

igls adv runreports --report_type=csm


Eyeglass Appliance Administration

Eyeglass Processes

Eyeglass has 2 main processes:

sca.service

scadb.service

ssh to the Eyeglass appliance or open the Eyeglass shell and sudo su- to root, to view service status and/or stop, start, restart these services using the commands below:

systemctl start sca.service

systemctl stop sca.service

systemctl status -l sca.service

systemctl start scadb.service

systemctl stop scadb.service

systemctl status -l scadb.service

yast

Launch the yast Eyeglass Linux operation system setup and configuration tool from the Eyeglass shell.

  1. Open the Eyeglass Shell.

  2. Login as admin and sudo su - to root.

  3. Enter ‘yast2’.

  4. The YaST2 menu opens.

  5. Use the arrow, tab, enter and function keys to navigate the menu.

Eyeglass Appliance Time Synchronization Best Practice

The appliance is a VMware appliance which allows the ESX host and Vcenter NTP time source to flow through to the appliance.  In most environments, the virtual environment NTP source is distributed to each host and each VM, on that host, receives a good time source via BIOS time set.    The Isilon cluster should be set to the same time source as the VMware environment to ensure time of events for DR logs are synced to the same source.   If the cluster is using a different time source then the procedure below can be used to add NTP to Eyeglass appliance.

  1. SSH to appliance as admin then sudo -s to root

  2. type 'yast'

  3. Select Network Services and then NTP Configuration




4.   Using tab to Start NTP Daemon and select Now and on Boot

5.   Tab to Add to add a server source

6.   Select Add Server using tab to add a new server

7.    In the add server dialog box enter ip or FQDN or hostname (Note: requires DNS to be setup correctly on the appliance) of the NTP source.

8.    Select the Test button to verify reachability and protocol connection for time responds correctly.

9.    Select OK using tab

10.  Done.


Eyeglass Automatic Updates for Recommended Packages

Eyeglass automatic recommended Suse updates are applied weekly only if the appliance has Internet connection.   Manual RPM updates are required for OS updates for customers without Internet access.  The RPM’s will need to be retrieved by customers from Internet hosted Repositories.  

Changing repository URL on the appliance is possible for a customer that rehost OS patches internally.  Use Zypper AR  URL  “name of repo” to add a repo or consult openSuse documentation.

The appliance is setup by default to check weekly for recommended packages and automatically apply them.  To change this setting:

  1. ssh to the Eyeglass appliance.

  2. Login as admin and sudo su to root or login as root.

  3. Type 'yast'.  The YaST2 menu opens with Software selected by default.





4.   Use the right arrow key to move to the right hand menu and then the down arrow to highlight Online Update Configuration.





5.    Hit Enter to select Online Update Configuration.  The Online Update Configuration window opens.



6.   To change the Interval, Tab until the Interval is highlighted.  Then use the arrow key to see Interval options.  Use the arrow key again to highlight the interval you would like and Enter to select.






7.   To enable/disable Automatic Updates, Tab until Automatic Online Update is highlighted.  Then Enter to select/deselect this option.



8.   Tab to OK and Enter to save your changes.


Update Eyeglass Appliance Network Settings

  1. ssh to the Eyeglass appliance (if access this way is available) or console to the Eyeglass Appliance Virtual Machine from vSphere.

  2. Login as admin and sudo su to root or login as root.

  3. Type 'yast'.  The YaST2 menu opens with Software selected by default.

  4. Use the down arrow key and navigate to Network Devices.  Then use the right arrow key to move to the right hand menu followed by the down arrow key to navigate to Network Settings.

  5. Use the Enter key to select Network Settings.

To change the Eyeglass appliance IP address:

  1. Start at the Network Settings window.

  2. Use the Tab key to highlight the Edit option on the Network Settings window and then Enter.

  3. In the Network Card Setup window, use the Tab key to navigate to the field you want to change and make the required update.

  4. Once you have made all of the required changes, use the Tab key to navigate to the Next Option and Enter.  This will return you to the Network Settings window.

  5. If no further updates are required use the Tab key to navigate to OK and Enter to save your changes.  If further updates are required, follow the steps in the appropriate section.


To change the Eyeglass appliance DNS settings:

  1. Start at the Network Settings window.

  2. Use the right arrow key to highlight the Hostname/DNS option.

  3. Use the Tab key to navigate to the field that needs to be updated and make the required change.

  4. Then Tab to OK to complete.


To change the Eyeglass appliance Routing settings:

  1. Start at the Network Settings window.

  2. Use the right arrow key to highlight the Routing Option.

  3. Use the Tab key to navigate to the field that needs to be updated and make the required change.

  4. Then Tab to OK to complete.

Eyeglass Root Password

If it is required to have the root password to the appliance follow the procedure below.

  1. Login as admin using ssh

  2. then execute 'sudo -s'

  3. then 'passwd'

  4. enter new and re-type new password

You can also execute commands as root without changing root password by following steps 1 and 2 above.

Configuring  Eyeglass for WAN with dual NIC environments


This solution can be used for customers that do not allow VM’s to access both clusters from a single network IP interface due to security or network design.  The solution below allows Eyeglass to be modified to support dual NIC solution and static routes to reach remote clusters.

Prerequisites:

  1. SSH to Eyeglass as admin, sudo -s

  2. shutdown

  3. Follow vmware steps with vcenter vsphere to add a 2nd nic to Eyeglass, use e1000 nic type

  4. Attach to 2nd virtual switch

  5. start up the vm

  Note:  Eyeglass must be able to login via PAPI API and SSH over these ip networks.






  1. SSH to eyeglass appliance login as root . (Note: you can login as admin then use the command sudo -s to root user, if root account is the default random password)

  2. Assume a single Eyeglass with two NICs, one configured for the address 10.105.16.100; the other for 10.105.17.101 to added the other route please follow the next steps to add your route

  3. Type the command yast2 > then tap to reach Network Settings > then  tap to reach Routing

  4. On Routing please tap to reach Add button and hit Enter .

  5. Add your destination IP , Subnet Mask (Hint : Use \24 instead of 255.255.255.0 or your own subnet mask in the same format otherwise you’ll have an error)  then tap to reach Ok then hit Enter .

  6. Now you can see the routes on the table. (Note: These are persistent routes)

  7. Change the cluster network setting as needed for your environment.  

Note: this simple network has the cluster on the same subnet as the Eyeglass NIC’s. Typically a router would be the next top address in the static routes for one cluster and the default route 0.0.0.0 would be used to reach the other cluster.

  1. Add the clusters to Eyeglass appliance - (Hint: go to add managed device on Eyeglass and set the IP address and authentication )


Appliance Security updates and Eyeglass updates with HTTP Proxy

Many customers use proxy access to the Internet and require configuration of HTTP proxy on devices that need access to the Internet on port 80.

Eyeglass updates are hosted for online updates and Operating System updates are also reachable with HTTP for security and package adds with zypper.

Steps to configure HTTP proxy for Eyeglass

  1. Login via SSH and admin password

  2. sudo -s (switch to root)

  3. yast  (run yast)

  4. Navigate to the screenshots below to enter proxy configuration details

    1. URL

    2. user id and password

Note: This is untested with various proxy software solutions and not supported under maintenance contracts.

Screen Shot 2015-10-26 at 1.46.23 PM.png

Screen Shot 2015-10-26 at 1.46.07 PM.png


Eyeglass - How to install VMware Native tools


Use this guide to remove 3rd party open VMware tools and replace with Native VMware tools for Linux.

Checking open-vm-tools installed package:


Login to Eyeglass appliance and use the command zypper to install and remove software:


eyeglass02-sandra:~ # zypper se open-vm*

Loading repository data...

Reading installed packages...


S | Name              | Summary                          | Type

--+-------------------+----------------------------------+--------

i | open-vm-tools     | Open Virtual Machine Tools       | package

 | open-vm-tools-gui | Open Virtual Machine Tools - GUI | package


eyeglass02-sandra:~ # zypper info open-vm-tools

Loading repository data...

Reading installed packages...

Information for package open-vm-tools:

--------------------------------------

Repository: openSUSE_13.1_OSS

Name: open-vm-tools

Version: 9.2.3-3.2.1

Arch: x86_64openSUSE

Vendor:

Installed: Yes

Status: up-to-date

Installed Size: 589.7 KiB

Summary: Open Virtual Machine Tools

Description:

Open Virtual Machine Tools (open-vm-tools) are the open source

implementation of VMware Tools. They are a set of guest operating

system virtualization components that enhance performance and user

experience of virtual machines. As virtualization technology rapidly

becomes mainstream, each virtualization solution provider implements

their own set of tools and utilities to supplement the guest virtual

machine. However, most of the implementations are proprietary and are

tied to a specific virtualization platform.

With the Open Virtual Machine Tools project, we are hoping to solve

this and other related problems. The tools are currently composed of

kernel modules for Linux and user-space programs for all VMware

supported Unix-like guest operating systems. They provide several

useful functions like:

* File transfer between a host and guest

* Improved memory management and network performance under

  virtualization

* General mechanisms and protocols for communication between host and

guests and from guest to guest


eyeglass02-sandra:~ #  zypper info open-vm-tools-gui

Loading repository data...

Reading installed packages...

Information for package open-vm-tools-gui:

------------------------------------------

Repository: openSUSE_13.1_OSS

Name: open-vm-tools-gui

Version: 9.2.3-3.2.1

Arch: x86_64

Vendor: openSUSE

Installed: No

Status: not installed

Installed Size: 377.7 KiB

Summary: Open Virtual Machine Tools - GUI

Description:

GUI Toolbox for Open Virtual Machine Tools

eyeglass02-sandra:~ #



Removing open-vm-tools package:


If you automatically want to remove any packages that become unneeded after removing the specified package, use the --clean-deps option:

zypper remove --clean-deps <package-name>


Example:


eyeglass02-sandra:~ # zypper remove --clean-deps open-vm-tools

Loading repository data...

Reading installed packages...

Resolving package dependencies...


The following 5 packages are going to be REMOVED:

 libdnet1 libicu51_2 libicu51_2-data libvmtools0 open-vm-tools


5 packages to remove.

After the operation, 26.8 MiB will be freed.

Continue? [y/n/? shows all options] (y): y

(1/5) Removing open-vm-tools-9.2.3-3.2.1 .........................................................................................[done]

Additional rpm output:

redirecting to systemctl stop vmtoolsd.service




(2/5) Removing libvmtools0-9.2.3-3.2.1 ...........................................................................................[done]

(3/5) Removing libdnet1-1.12-19.1.2 ..............................................................................................[done]

(4/5) Removing libicu51_2-51.2-6.1.2 .............................................................................................[done]

(5/5) Removing libicu51_2-data-51.2-6.1.2 ........................................................................................[done]

eyeglass02-sandra:~ #


Verifying on vCenter if the open-vm-tools have been removed, click VM in the virtual machine menu, then on the summary TAB check the VMware tools is running:

vm-tools-zypper-remove-open-vm-tools.PNG


.

Installing VMware Tools from the Command Line with the Tar Installer

  1. Click VM in the virtual machine menu, then click Guest > Install/Upgrade VMware Tools and click OK.


2.   To create a mount point, run:


mkdir /mnt/cdrom


3.  To mount the CDROM, run:

mount /dev/cdrom /mnt/cdrom

cd /tmp

Note: If you have a previous installation, delete the previous vmware-distrib directory before installing. The default location of this directory is  </tmp/vmware-tools-distrib>.

4.  Untar the VMware Tools tar file:

tar zxpf /mnt/cdrom/VMwareTools-5.0.0-<xxxx>.tar.gz

Where <xxxx> is the build/revision number of the VMware Workstation release.

Note: If you attempt to install a tar installation over an rpm installation — or the reverse — the installer detects the previous installation and must convert the installer database format before continuing.

Example:


eyeglass02-sandra:/ # cd /mnt/cdrom/

eyeglass02-sandra:/mnt/cdrom # ls -l

total 61894

-r--r--r-- 1 root root 61978217 Aug 17  2013 VMwareTools-9.4.0-1280544.tar.gz

-r-xr-xr-x 1 root root     1961 Aug 17  2013 manifest.txt

-r--r--r-- 1 root root     1847 Aug 17  2013 run_upgrader.sh

-r-xr-xr-x 1 root root   693484 Aug 17  2013 vmware-tools-upgrader-32

-r-xr-xr-x 1 root root   702400 Aug 17  2013 vmware-tools-upgrader-64

eyeglass02-sandra:/mnt/cdrom #


eyeglass02-sandra:/tmp # tar zxpf /mnt/cdrom/VMwareTools-9.4.0-1280544.tar.gz

eyeglass02-sandra:/tmp # ls -l

total 36

drwxrwxrwt 2 root root 4096 Dec 22 20:52 VMwareDnD

drwx------ 2 root root 4096 Dec 22 21:08 YaST2-07198-U8yhoU

drwx------ 2 root root 4096 Dec 22 21:08 YaST2-07198-p3WD1w

drwxr-xr-x 2 root root 4096 Dec 22 20:52 hsperfdata_root

drwxr-xr-x 2 root root 4096 Dec 22 20:53 jna-root

drwxr-xr-x 3 root root 4096 Dec 23 09:03 passenger.1.0.7000

drwxr-xr-x 2 root root 4096 Nov 27 15:26 tomcat

drwx------ 2 root root 4096 Dec 22 20:52 vmware-root

drwxr-xr-x 7 root root 4096 Aug 17  2013 vmware-tools-distrib

eyeglass02-sandra:/tmp # cd vmware-tools-distrib

eyeglass02-sandra:/tmp/vmware-tools-distrib #


5.  Run the VMware Tools tar installer:

cd vmware-tools-distrib

./vmware-install.pl

Respond to the configuration questions on the screen. Press Enter to accept the default value.

Example:

eyeglass02-sandra:/tmp/vmware-tools-distrib # ./vmware-install.pl

Creating a new VMware Tools installer database using the tar4 format.


Installing VMware Tools.


In which directory do you want to install the binary files?

[/usr/bin]


What is the directory that contains the init directories (rc0.d/ to rc6.d/)?

[/etc/init.d]


What is the directory that contains the init scripts?

[/etc/init.d]


In which directory do you want to install the daemon files?

[/usr/sbin]


In which directory do you want to install the library files?

[/usr/lib/vmware-tools]


The path "/usr/lib/vmware-tools" does not exist currently. This program is

going to create it, including needed parent directories. Is this what you want?

[yes]


In which directory do you want to install the documentation files?

[/usr/share/doc/vmware-tools]


The path "/usr/share/doc/vmware-tools" does not exist currently. This program

is going to create it, including needed parent directories. Is this what you

want? [yes]


The installation of VMware Tools 9.4.0 build-1280544 for Linux completed

successfully. You can decide to remove this software from your system at any

time by invoking the following command: "/usr/bin/vmware-uninstall-tools.pl".


Before running VMware Tools for the first time, you need to configure it by

invoking the following command: "/usr/bin/vmware-config-tools.pl". Do you want

this program to invoke the command for you now? [yes]


Initializing...



Making sure services for VMware Tools are stopped.


Stopping VMware Tools services in the virtual machine:

  Guest operating system daemon:                                      done

  Unmounting HGFS shares:                                             done

  Guest filesystem driver:                                            done



The module vmci has already been installed on this system by another installer

or package and will not be modified by this installer.


The module vsock has already been installed on this system by another installer

or package and will not be modified by this installer.


The module vmxnet3 has already been installed on this system by another

installer or package and will not be modified by this installer.


The module pvscsi has already been installed on this system by another

installer or package and will not be modified by this installer.


The module vmmemctl has already been installed on this system by another

installer or package and will not be modified by this installer.


The VMware Host-Guest Filesystem allows for shared folders between the host OS

and the guest OS in a Fusion or Workstation virtual environment.  Do you wish

to enable this feature? [no]


The vmxnet driver is no longer supported on kernels 3.3 and greater. Please

upgrade to a newer virtual NIC. (e.g., vmxnet3 or e1000e)


The vmblock enables dragging or copying files between host and guest in a

Fusion or Workstation virtual environment.  Do you wish to enable this feature?

[no]


VMware automatic kernel modules enables automatic building and installation of

VMware kernel modules at boot that are not already present. This feature can be

enabled/disabled by re-running vmware-config-tools.pl.


Would you like to enable VMware automatic kernel modules?

[no]


No X install found.


Creating a new initrd boot image for the kernel.


Kernel image:   /boot/vmlinuz-3.11.10-21-default

Initrd image:   /boot/initrd-3.11.10-21-default

KMS drivers:     vmwgfx

Root device:    /dev/sda1 (mounted on / as ext3)

Kernel Modules: thermal_sys thermal processor fan usb-common usbcore ehci-hcd ohci-hcd uhci-hcd xhci-hcd vmxnet3 vmw_pvscsi scsi_dh scsi_dh_alua scsi_dh_hp_sw scsi_dh_rdac scsi_dh_emc drm ttm vmwgfx scsi_transport_spi mptbase mptscsih mptspi usbhid hid-logitech-dj hid-generic hid-holtek-kbd hid-lenovo-tpkbd hid-ortek hid-roccat hid-roccat-common hid-roccat-arvo hid-roccat-isku hid-samsung ehci-pci ohci-pci

Features:       acpi kms plymouth block usb resume.userspace resume.kernel

  Checking acpi hot plug                                              done

Starting VMware Tools services in the virtual machine:

  Switching to guest configuration:                                   done

  Guest operating system daemon:                                      done

The configuration of VMware Tools 9.4.0 build-1280544 for Linux for this

running kernel completed successfully.


You must restart your X session before any mouse or graphics changes take

effect.


You can now run VMware Tools by invoking "/usr/bin/vmware-toolbox-cmd" from the

command line.


To enable advanced X features (e.g., guest resolution fit, drag and drop, and

file and text copy/paste), you will need to do one (or more) of the following:

1. Manually start /usr/bin/vmware-user

2. Log out and log back into your desktop session; and,

3. Restart your X session.


Enjoy,


--the VMware team


Found VMware Tools CDROM mounted at /mnt/cdrom. Ejecting device /dev/sr0 ...

eyeglass02-sandra:/tmp/vmware-tools-distrib #


6.  To end the VMware Tools install, click VM in the virtual machine menu, then click Guest > End VMware Tools Install.

7.  To check if the VMware tools has been installed, click VM, then summary Tab:


vm-tools-installation-done.PNG


8.  To check the vmware tools installed on the appliance:

/etc/init.d/vmware-tools status

vmware-toolbox-cmd -v


Example:

eyeglass02-sandra:~ # /etc/init.d/vmware-tools status

vmtoolsd is running

eyeglass02-sandra:~ # vmware-toolbox-cmd -v

9.4.0.25793 (build-1280544)

eyeglass02-sandra:~ #

Upgrading the Eyeglass Appliance

Please refer to the Eyeglass Upgrade Guide for procedure to upgrade the Eyeglass Appliance.

Diagnostic Tools for Dark Sites

This feature allows diagnostic tool to process logs on the appliance and summarize errors from config sync, time to sync, failover jobs.

To execute the log analysis

  1. SSH to appliance as admin user

  2. Run command: igls app report

  3. Wait for the report to complete

  4. See logs report on: https://<eyeglass IP address>/report/

  5. Alternate location is  /srv/www/htdocs/report with file name index.html can be downloaded via scp to review and open in a browser


Eyeglass Remote Logging Service to Syslog Server

Overview


Remote Logging Service feature provides the ability to push the contents of the Eyeglass Appliance /var/log/messages logs received from managed devices to 3rd party log consumers such as vmware Log Insight, Splunk or other syslog servers  using a customized logging feature on the appliance.  

Based on this configuration as soon as the appliance receives a syslog message it applies tagging to the incoming messages based on the device type to include name and serial number information for each log message locally and then sends the message on port 514 via UDP to the configured 3rd party log consumers once configured.

This allows dashboards and analysis in logging tools to be done on serial number, device name and inventory data collected from the device and logged.   

The Eyeglass log include information from various data sources collected from DR status data and output to syslog locally on Eyeglass.  For example inventory information, alarms, events, custom Eyeglass events or failure tasks are all tagged to a managed device and sent to syslog.

This capability allows logging analysis tools to get much more information about the device than typical syslog only events and build rules to trigger on data status log messages example DR not Ready on Access zone name xxxx is possible.

Setting Up the Remote Logging

  1. Login to the Eyeglass UI and access the Log View.

To access Log View, Single Click on Log View Icon from Eyeglass Desktop or on the bottom left of the page and select logging option

  1. The Logging window should open. click on Remote Logging Services option: then Add Remote Log Consumer. Fill in the required Connection Parameters and Select the Log Consumer type you wish to access. The default port is 514. You can replace it with the port number as required.

  1. When done, Single click on Submit. The Remote Log Consumer you just added will be displayed on Remote Logging Services Interface.

  1. SSH to Eyeglass as admin and sudo su root , cd /etc/syslog-ng then edit syslog-ng.conf file

  2. Find in that file  the line with setting for:  destination logserver

  3. Edit the IP address for the server and put the same REMOTE LOG CONSUMER IP that you added into Eyeglass UI

  4. Remove the # for both lines in the file example below

  1. Save and close the file.

  2. Restart syslog service after changing the configuration in syslog-ng.conf. Use  systemctl restart syslog-ng command

  3. The remote server should start receiving a syslog packets from Eyeglass after setting the configuration on the server.

  4. Note: In your syslog server make sure that the port you are using for remote logging is available and not blocked by firewall.


Example of Syslog Messages received from Eyeglass


Syslog Messages for Replication Jobs status

6/21/2017 2:45:21 AM 172.22.4.89 [12218]: 2017-06-21 02:45:21,460 [pool-4-thread-488] DEBUG MAIN Replicationjobs:getAllJobs [69] - [{"lastsuccessts":1498027512582,"jobstatus":"ENABLED","name":"rnsm04-c03_InsightIQ-NFSDS","destination":"rnsm04-c04","lastrunts":1498027512582,"policy_name":"InsightIQ-NFSDS","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-nfs01","jobtype":"AUTO","destinationpath":"/ifs/data/zone01/z01-nfs01"},{"lastsuccessts":1497608339227,"jobstatus":"ENABLED","name":"rnsm04-c03_rnsm04-c04","destination":"rnsm04-c04","lastrunts":1497608339227,"policy_name":"InsightIQ-NFSDS","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-nfs01","jobtype":"READINESS","destinationpath":"/ifs/data/zone01/z01-nfs01"},{"lastsuccessts":1498027512568,"jobstatus":"ENABLED","name":"rnsm04-c03_InsightIQ-NFSDS-ZONES","destination":"rnsm04-c04","lastrunts":1498027512568,"policy_name":"InsightIQ-NFSDS","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-nfs01","jobtype":"ZONES","destinationpath":"/ifs/data/zone01/z01-nfs01"},{"jobstatus":"USERDISABLED","name":"rnsm04-c03_InsightIQ-NFSDS_FILESYSTEM","destination":"rnsm04-c04","policy_name":"InsightIQ-NFSDS","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-nfs01","jobtype":"FILESYSTEM","destinationpath":"/ifs/data/zone01/z01-nfs01"},{"jobstatus":"ENABLED","name":"rnsm04-c03_InsightIQ-NFSDS_quotas","destination":"rnsm04-c04","policy_name":"InsightIQ-NFSDS","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-nfs01","jobtype":"QUOTA","destinationpath":"/ifs/data/zone01/z01-nfs01"},{"lastsuccessts":1498027512587,"jobstatus":"ENABLED","name":"rnsm04-c03_z01-smb01-synciq","destination":"rnsm04-c04","lastrunts":1498027512587,"policy_name":"z01-smb01-synciq","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-smb01","jobtype":"AUTO","destinationpath":"/ifs/data/zone01/z01-smb01"},{"jobstatus":"USERDISABLED","name":"rnsm04-c03_z01-smb01-synciq-ZONES","destination":"rnsm04-c04","policy_name":"z01-smb01-synciq","source":"rnsm04-c03","sourcepath":"/ifs/data/zone01/z01-smb01","jobtype":"ZONES", rnsm04-igls-03 06-21 02:45:21 6 bash 3


Syslog Messages for Policy Readiness

6/21/2017 3:29:36 AM 172.22.4.89 [12218]: 2017-06-21 03:29:36,587 [pool-4-thread-492] DEBUG MAIN Policies:getAllPolicies [56] - [{"policy_name":"InsightIQ-NFSDS","policy_enabled":true,"policy_last_success":1497605568000,"policy_last_run":1497605568000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"OK","job_status":"SUCCESS","job_name":"rnsm04-c03_InsightIQ-NFSDS","job_last_run":1498029912540,"job_last_success":1498029912540,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498029916470},{"policy_name":"z01-smb01-synciq","policy_enabled":true,"policy_last_success":1497933205000,"policy_last_run":1497933205000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"OK","job_status":"SUCCESS","job_name":"rnsm04-c03_z01-smb01-synciq","job_last_run":1498029912546,"job_last_success":1498029912546,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498029916474}]

6/21/2017 2:00:24 AM 172.22.4.89 [12218]: 2017-06-21 02:00:24,266 [pool-4-thread-473] DEBUG MAIN Policies:getAllPolicies [56] - [{"policy_name":"InsightIQ-NFSDS","policy_enabled":true,"policy_last_success":1497605568000,"policy_last_run":1497605568000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"WARNING","job_status":"SUCCESS","job_name":"rnsm04-c03_InsightIQ-NFSDS","job_last_run":1498024815649,"job_last_success":1498024815649,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498024821235},{"policy_name":"z01-smb01-synciq","policy_enabled":true,"policy_last_success":1497933205000,"policy_last_run":1497933205000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"WARNING","job_status":"SUCCESS","job_name":"rnsm04-c03_z01-smb01-synciq","job_last_run":1498024817389,"job_last_success":1498024817389,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498024821240}] rnsm04-igls-03 06-21 02:00:24 6 bash 3

6/21/2017 12:05:30 AM 172.22.4.89 [12218]: 2017-06-21 00:05:30,310 [pool-4-thread-439] DEBUG MAIN Policies:getAllPolicies [56] - [{"policy_name":"InsightIQ-NFSDS","policy_enabled":true,"policy_last_success":1497605568000,"policy_last_run":1497605568000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"WARNING","job_status":"SUCCESS","job_name":"rnsm04-c03_InsightIQ-NFSDS","job_last_run":1498017912818,"job_last_success":1498017912818,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498017915591},{"policy_name":"z01-smb01-synciq","policy_enabled":true,"policy_last_success":1497933205000,"policy_last_run":1497933205000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"WARNING","job_status":"SUCCESS","job_name":"rnsm04-c03_z01-smb01-synciq","job_last_run":1498017912823,"job_last_success":1498017912823,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498017915595}]


Syslog Messages for Zone Readiness Jobs

6/21/2017 2:58:30 AM 172.22.4.89 [12218]: 2017-06-21 02:58:30,340 [pool-4715-thread-1] DEBUG MAIN ReadinessTask:lambda$run$957 [69] - ReadinessTask is done. rnsm04-igls-03 06-21 02:58:30 6 bash 3

6/21/2017 2:58:30 AM 172.22.4.89 [12218]: 2017-06-21 02:58:30,327 [pool-4713-thread-1] DEBUG MAIN ReadinessJobResultHandler:handleResult [64] - JOB rnsm04-c03_rnsm04-c04: Status: {"state":"FINISHED","jobStatus":"OK","started":1498028299796,"finished":1498028310325,"duration":10529,"name":"MediateSPN rnsm04-c03_rnsm04-c04","info":"SPN check","children":[],"modified":1498028310325} rnsm04-igls-03 06-21 02:58:30 6 bash 3

6/21/2017 2:58:30 AM 172.22.4.89 [12218]: 2017-06-21 02:58:30,326 [pool-4713-thread-1] DEBUG MAIN ReadinessJobResultHandler:handleResult [64] - JOB rnsm04-c03_rnsm04-c04: Status: {"state":"FINISHED","jobStatus":"OK","started":1498028302405,"finished":1498028302672,"duration":267,"name":"AccessZoneValidation rnsm04-c03_rnsm04-c04","info":"Access Zone Validation","children":[],"modified":1498028302672} rnsm04-igls-03 06-21 02:58:30 6 bash 3

6/21/2017 2:58:30 AM 172.22.4.89 [12218]: 2017-06-21 02:58:30,325 [pool-4713-thread-1] DEBUG MAIN ReadinessJobResultHandler:handleResult [56] - JOB NAME: rnsm04-c03_rnsm04-c04 rnsm04-igls-03 06-21 02:58:30 6 bash 3

6/21/2017 2:58:30 AM 172.22.4.89 [12218]: 2017-06-21 02:58:30,325 [pool-4713-thread-1] DEBUG MAIN ReadinessJobResultHandler:handleResult [55] - ***PRINTING READINESS JOB STATUS ******* rnsm04-igls-03 06-21 02:58:30 6 bash 3


Syslog Messages for FAILOVER

6/21/2017 3:56:00 AM 172.22.4.89 [12218]: 2017-06-21 03:56:00,219 [cron4j-task-10] DEBUG MAIN AlarmDataManager:executeSave [2817] - >> Keys: Sync-Key: 'rnsm04-c03_Policy Failover 2017-06-21_03-41-54', Severity: 'INFORMATIONAL', Description: 'Failover Succeeded' rnsm04-igls-03 06-21 03:56:00 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,808 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,805 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:appendReportsToLog [66] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,805 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:appendReportsToLog [65] - SyncIQ Reports For Policy: z01-smb01-synciq_mirror rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,804 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [70] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,804 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,800 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [70] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,800 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,796 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [70] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,791 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [70] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,795 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,791 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,784 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [70] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: "SIQ-Failover-z01-smb01-synciq-2017-06-21_03-52-25" rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [INFO] SYSLOG:154 - Eyeglass, , Event: 2017-06-21 03:55:33.814, AID:rnsm04-c03_Policy Failover 2017-06-21_03-41-54, Port:Nil, Type:null, EntityType:, Extra Data:{"Status":"Success","Finished":1498031733809,"Started":1498030914946,"URL":"https://172.22.4.89/failover_logs/Policy_Failover__rnsm04-c03__2017-06-21_03-41-54__SUCCESS/Policy_Failover__rnsm04-c03__2017-06-21_03-41-54__SUCCESS.json"}, Description:Failover Succeeded , NSA, Severity:INFORMATIONAL, Impact:false, Category:SCA0040 localhost 06-21 03:55:33 6 20

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,784 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:lambda$appendReportsToLog$617 [69] - { rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,777 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:appendReportsToLog [66] - ********************** rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,776 [pool-4817-thread-1] INFO com.superna.nde.jobengine.failover.operations.AddReportsToLogs AddReportsToLogs:appendReportsToLog [65] - SyncIQ Reports For Policy: z01-smb01-synciq rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,773 [pool-4817-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting Fetch reports for all policies rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:34 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,772 [pool-4817-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE Post Failover Inventory rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:33 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,163 [Config-rnsm04-c03] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:33 AM 172.22.4.89 [12218]: 2017-06-21 03:55:33,018 [Config-rnsm04-c03] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:33 6 bash 3

6/21/2017 3:55:30 AM 172.22.4.89 [12218]: 2017-06-21 03:55:30,421 [Config-rnsm04-c04] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:30 6 bash 3

6/21/2017 3:55:30 AM 172.22.4.89 [12218]: 2017-06-21 03:55:30,277 [Config-rnsm04-c04] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:30 6 bash 3

6/21/2017 3:55:12 AM 172.22.4.89 [12218]: 2017-06-21 03:55:12,227 [pool-4846-thread-1] DEBUG MAIN InventoryTreeSync:doSyncTree [227] - Deleting rnsm04-c04_0050569fabe8ed5f43599d12125d95893177_Configuration/snapshot/snapshots/snapshots_SIQ-Failover-z01-smb01-synciq-2017-06-21_03-52-25 rnsm04-igls-03 06-21 03:55:12 6 bash 3

6/21/2017 3:55:12 AM 172.22.4.89 [12218]: 2017-06-21 03:55:12,071 [pool-4846-thread-1] DEBUG MAIN DBAccess:save [51] - Saving new: rnsm04-c03_0050569f2381b25c4359c015dbaaa5e20044_Configuration/snapshot/snapshots/snapshots_SIQ-Failover-z01-smb01-synciq_mirror-2017-06-21_03-55-00 rnsm04-igls-03 06-21 03:55:12 6 bash 3

6/21/2017 3:55:12 AM 172.22.4.89 [12218]: 2017-06-21 03:55:12,070 [pool-4846-thread-1] DEBUG MAIN InventoryTreeSync:doSyncTree [148] - Saving rnsm04-c03_0050569f2381b25c4359c015dbaaa5e20044_Configuration/snapshot/snapshots/snapshots_SIQ-Failover-z01-smb01-synciq_mirror-2017-06-21_03-55-00 rnsm04-igls-03 06-21 03:55:12 6 bash 3

6/21/2017 3:55:12 AM 172.22.4.89 [12218]: 2017-06-21 03:55:11,861 [Config-rnsm04-c03] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:11 6 bash 3

6/21/2017 3:55:12 AM 172.22.4.89 [12218]: 2017-06-21 03:55:11,711 [Config-rnsm04-c03] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:11 6 bash 3

6/21/2017 3:55:11 AM 172.22.4.89 [12218]: 2017-06-21 03:55:11,438 [Config-rnsm04-c04] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:11 6 bash 3

6/21/2017 3:55:11 AM 172.22.4.89 [12218]: 2017-06-21 03:55:11,289 [Config-rnsm04-c04] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:11 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:10,265 [pool-4817-thread-1] DEBUG MAIN RunInventory:execute [35] - Executing Post Failover Inventory rnsm04-igls-03 06-21 03:55:10 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:10,255 [pool-4817-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting Post Failover Inventory rnsm04-igls-03 06-21 03:55:10 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:10,250 [pool-4804-thread-2] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 delete all quotas from /ifs/data/zone01/z01-smb01 rnsm04-igls-03 06-21 03:55:10 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:10,055 [pool-4813-thread-2] DEBUG MAIN FailoverStep:call [132] - DONE Set configuration replication for policies to ENABLED rnsm04-igls-03 06-21 03:55:10 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,937 [pool-4804-thread-2] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 delete all quotas from /ifs/data/zone01/z01-smb01 rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,927 [pool-4813-thread-2] DEBUG MAIN FailoverStep:call [118] - Starting Set configuration replication for policies to ENABLED rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,917 [pool-4814-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE Run Quota Jobs Now rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,911 [pool-13-thread-1] DEBUG MAIN QuotaFailoverJobResultHandler:handleResult [37] - Status: {"state":"FINISHED","jobStatus":"OK","started":1498031709697,"finished":1498031709898,"duration":201,"name":"Quota Failover z01-smb01-synciq 1498031709697","children":[{"state":"FINISHED","jobStatus":"OK","started":1498031709699,"finished":1498031709898,"duration":199,"name":"Quota Failover z01-smb01-synciq 1498031709697","children":[{"state":"FINISHED","jobStatus":"OK","started":1498031709701,"finished":1498031709708,"duration":7,"name":"Quota Failover z01-smb01-synciq","children":[{"state":"FINISHED","jobStatus":"OK","started":1498031709706,"finished":1498031709706,"duration":0,"name":"Quota Failover for directory, user and group type quotas - z01-smb01-synciq","children":[],"modified":1498031709706},{"state":"FINISHED","jobStatus":"OK","started":1498031709708,"finished":1498031709708,"duration":0,"name":"Quota Failover for default-user and default-group type quotas - z01-smb01-synciq","children":[],"modified":1498031709708}],"modified":1498031709708},{"state":"FINISHED","jobStatus":"OK","started":1498031709708,"finished":1498031709898,"duration":190,"name":"Quota Audit Job for policy rnsm04-c03_z01-smb01-synciq_quotas","children":[{"state":"FINISHED","jobStatus":"OK","started":1498031709714,"finished":1498031709898,"duration":184,"name":"Start Quota Audit Job for policy rnsm04-c03_z01-smb01-synciq_quotas","children":[],"modified":1498031709898},{"state":"FINISHED","jobStatus":"OK","started":1498031709887,"finished":1498031709892,"duration":5,"name":"z01-smb01-synciq Quota Audit1498031709887","children":[{"state":"FINISHED","jobStatus":"OK","started":0,"finished":1498031709892,"duration":1498031709892,"name":"z01-smb01-synciq audit","info":"[]","children":[],"modified":1498031709892}],"modified":1498031709892}],"modified":1498031709898}],"modified":1498031709898}],"modified":1498031709898} rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,910 [pool-13-thread-1] DEBUG MAIN QuotaFailoverJobResultHandler:handleResult [36] - JOB NAME: rnsm04-c03_z01-smb01-synciq_quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,907 [pool-13-thread-1] DEBUG MAIN QuotaFailoverJobResultHandler:handleResult [35] - ***PRINTING QUOTA FAILOVER JOB STATUS ******* rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,876 [pool-4857-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,874 [pool-4857-thread-2] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,718 [pool-4857-thread-2] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,717 [pool-4857-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,716 [pool-4853-thread-1] DEBUG MAIN QuotaJobFactory:runPrepJob [77] - Is controlled failover? true rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,680 [pool-4814-thread-1] INFO MAIN QuotaFailoverJobFactory:createJob [96] - Adding policy rnsm04-c03_z01-smb01-synciq_quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,661 [pool-4849-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c04 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:10 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,625 [pool-4849-thread-2] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:55:09 AM 172.22.4.89 [12218]: 2017-06-21 03:55:09,463 [pool-4849-thread-2] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 get quotas rnsm04-igls-03 06-21 03:55:09 6 bash 3

6/21/2017 3:41:56 AM 172.22.4.89 [12218]: 2017-06-21 03:41:56,434 [pool-4805-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 run z01-smb01-synciq rnsm04-igls-03 06-21 03:41:56 6 bash 3

6/21/2017 3:41:56 AM 172.22.4.89 [12218]: 2017-06-21 03:41:56,431 [pool-4802-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 unset schedule for policy z01-smb01-synciq rnsm04-igls-03 06-21 03:41:56 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:55,404 [pool-4802-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 unset schedule for policy z01-smb01-synciq rnsm04-igls-03 06-21 03:41:55 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:55,403 [pool-4802-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE rnsm04-c03 get z01-smb01-synciq info rnsm04-igls-03 06-21 03:41:55 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:54,970 [pool-4802-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting rnsm04-c03 get z01-smb01-synciq info rnsm04-igls-03 06-21 03:41:54 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:54,966 [pool-4806-thread-1] DEBUG MAIN FailoverStep:call [132] - DONE Wait for other failover jobs to complete rnsm04-igls-03 06-21 03:41:54 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:54,961 [pool-4806-thread-1] DEBUG MAIN FailoverStep:call [118] - Starting Wait for other failover jobs to complete rnsm04-igls-03 06-21 03:41:54 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:54,767 [pool-4-thread-492] INFO MAIN PolicyFailoverJobFactory:createJob [83] - in policy failover rnsm04-igls-03 06-21 03:41:54 6 bash 3

6/21/2017 3:41:55 AM 172.22.4.89 [12218]: 2017-06-21 03:41:54,714 [pool-4-thread-492] DEBUG MAIN RequestDispatcher:getPlugin [180] - retrieving plugin: com.superna.scaapi.plugins.dr.PolicyFailover rnsm04-igls-03 06-21 03:41:54 6 bash 3

6/21/2017 3:41:46 AM 172.22.4.89 [12218]: 2017-06-21 03:41:46,672 [pool-4-thread-492] DEBUG MAIN RequestDispatcher:getPlugin [180] - retrieving plugin: com.superna.scaapi.plugins.dr.PolicyFailover rnsm04-igls-03 06-21 03:41:46 6 bash 3

6/21/2017 3:41:40 AM 172.22.4.89 [12218]: 2017-06-21 03:41:39,792 [pool-4-thread-492] DEBUG MAIN RequestDispatcher:getPlugin [180] - retrieving plugin: com.superna.scaapi.plugins.dr.PolicyFailover rnsm04-igls-03 06-21 03:41:39 6 bash 3


Syslog Messages for ALARM

6/21/2017 3:56:00 AM 172.22.4.89 [12218]: 2017-06-21 03:56:00,219 [cron4j-task-10] DEBUG MAIN AlarmDataManager:executeSave [2817] - >> Keys: Sync-Key: 'rnsm04-c03_Policy Failover 2017-06-21_03-41-54', Severity: 'INFORMATIONAL', Description: 'Failover Succeeded' rnsm04-igls-03 06-21 03:56:00 6 bash 3

6/21/2017 3:56:00 AM 172.22.4.89 [12218]: 2017-06-21 03:56:00,218 [cron4j-task-10] INFO MAIN AlarmDataManager:executeSave [2815] - Sending alarm from '' to DB rnsm04-igls-03 06-21 03:56:00 6 bash 3

6/21/2017 3:56:00 AM 172.22.4.89 [12218]: 2017-06-21 03:56:00,206 [cron4j-task-10] INFO MAIN AlarmManagementTask:run [75] - Handling possible alarms. rnsm04-igls-03 06-21 03:56:00 6 bash 3

6/21/2017 3:56:00 AM 172.22.4.89 [12218]: 2017-06-21 03:56:00,063 [cron4j-task-5] INFO MAIN AlarmPollTask:run [70] - Polling network elements for alarms. rnsm04-igls-03 06-21 03:56:00 6 bash 3


Syslog Messages for READINESS with Overall DR Status INFO

6/21/2017 4:10:25 AM 172.22.4.89 [12218]: 2017-06-21 04:10:24,731 [pool-4-thread-510] DEBUG MAIN Policies:getAllPolicies [56] - [{"policy_name":"InsightIQ-NFSDS","policy_enabled":true,"policy_last_success":1497605568000,"policy_last_run":1497605568000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"OK","job_status":"SUCCESS","job_name":"rnsm04-c03_InsightIQ-NFSDS","job_last_run":1498032615457,"job_last_success":1498032615457,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498032619245},{"policy_name":"z01-smb01-synciq","policy_enabled":false,"policy_last_success":1498030917000,"policy_last_run":1498031683000,"policy_last_status":"finished","policy_status":"DISABLED","overall_dr_status":"FAILED_OVER","job_status":"DISABLED","job_name":"rnsm04-c03_z01-smb01-synciq","job_last_run":1498031576731,"job_last_success":1498031576731,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":false,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498032619249},{"policy_name":"z01-smb01-synciq_mirror","policy_enabled":true,"policy_last_success":1498031691000,"policy_last_run":1498031691000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"INFO","job_status":"SUCCESS","job_name":"rnsm04-c04_z01-smb01-synciq_mirror","job_last_run":1498032615462,"job_last_success":1498032615462,"job_source":"rnsm04-c04","job_destination":"rnsm04-c03","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498032623757}] rnsm04-igls-03 06-21 04:10:24 6 bash 3


Syslog Messages for READINESS with Overall DR Status OK

6/21/2017 4:25:15 AM 172.22.4.89 [12218]: 2017-06-21 04:25:14,778 [pool-4-thread-516] DEBUG MAIN Policies:getAllPolicies [56] - [{"policy_name":"InsightIQ-NFSDS","policy_enabled":true,"policy_last_success":1497605568000,"policy_last_run":1497605568000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"OK","job_status":"SUCCESS","job_name":"rnsm04-c03_InsightIQ-NFSDS","job_last_run":1498033514694,"job_last_success":1498033514694,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498033216402},{"policy_name":"z01-smb01-synciq","policy_enabled":false,"policy_last_success":1498030917000,"policy_last_run":1498031683000,"policy_last_status":"finished","policy_status":"DISABLED","overall_dr_status":"FAILED_OVER","job_status":"DISABLED","job_name":"rnsm04-c03_z01-smb01-synciq","job_last_run":1498031576731,"job_last_success":1498031576731,"job_source":"rnsm04-c03","job_destination":"rnsm04-c04","job_enabled":false,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498033216406},{"policy_name":"z01-smb01-synciq_mirror","policy_enabled":true,"policy_last_success":1498031691000,"policy_last_run":1498031691000,"policy_last_status":"finished","policy_status":"SUCCESS","overall_dr_status":"OK","job_status":"SUCCESS","job_name":"rnsm04-c04_z01-smb01-synciq_mirror","job_last_run":1498033514699,"job_last_success":1498033514699,"job_source":"rnsm04-c04","job_destination":"rnsm04-c03","job_enabled":true,"job_has_policy":true,"audit_status":"AUDITSUCCEEDED","policy_readiness_last_success":1498033219830}] rnsm04-igls-03 06-21 04:25:14 6 bash 3


Syslog Messages for Ransomware Events

6/20/2017 2:51:27 AM 172.22.4.89 [INFO] SYSLOG:154 - Eyeglass, , Event: 2017-06-20 02:51:26.746, AID:RNSM04\rnsm04-t32, Port:Nil, Type:null, EntityType:, Extra Data:{"severity":"WARNING","user name":"RNSM04\\rnsm04-t32","files":["\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\6871.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\3556.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\4179.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9172.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9281.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9811.txt"],"explanation":"New ransomware event created","sid":"S-1-5-21-4205747320-2446522354-1604720750-11190"}, Description:Ransomware signal received. , NSA, Severity:CRITICAL, Impact:false, Category:SCA0061 localhost 06-20 02:51:26 6 20

6/20/2017 2:51:27 AM 172.22.4.89 [12218]: 2017-06-20 02:51:26,749 [Thread-31] INFO SYSLOG AlarmHandlerTask:run [154] - Eyeglass, , Event: 2017-06-20 02:51:26.746, AID:RNSM04\rnsm04-t32, Port:Nil, Type:null, EntityType:, Extra Data:{"severity":"WARNING","user name":"RNSM04\\rnsm04-t32","files":["\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\6871.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\3556.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\4179.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9172.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9281.txt","\\\\rnsm04-c03\\zone01\\ifs\\data\\zone01\\z01-smb01\\Data02\\9811.txt"],"explanation":"New ransomware event created","sid":"S-1-5-21-4205747320-2446522354-1604720750-11190"}, Description:Ransomware signal received. , NSA, Severity:CRITICAL, Impact:false, Category:SCA0061 rnsm04-igls-03 06-20 02:51:26 6 bash 3


APPENDIX 1 Update Isilon sudoer file for Eyeglass Service Account & Isilon Clusters version 7.1.1.0 and 7.1.1.1 for SyncIQ Policy or DFS Mode Failover

Eyeglass SyncIQ Policy Failover requires some CLI commands that must run with root level access.  Many customers also run the cluster in STIG or compliance mode for Smartlock WORM features and root user account is not allowed to login and run commands. To allow this user permissions across the cluster nodes the following commands must have sudo privileges.

OneFS CLI commands that require Elevated Permissions to run as root:

  1. For Isilon Clusters version 7.1.1.0 and 7.1.1.1 ONLY an additional entry is also required.

To add sudo privileges to the Eyeglass cluster service account:

  1. ssh to the Isilon cluster as root user

  2. Edit the sudoer file using the Isilon isi_visudo command.

  3. Sudoers file opens in vi editor.

  4. Add a line where ‘~’ is present for the user used in Eyeglass that was used to provision the Isilon clusters

For example - if the user is ‘eyeglass’ , add the line

eyeglass ALL=(ALL) NOPASSWD: /usr/bin/isi_linmap_mod

  1. To save changes and close the file, press the Esc key on the keyboard to exit the shell’s insert mode and then type in :wq. This will not appear on the screen while being typed. Pressing the Enter key will close the file.

  2. Repeat for each Cluster managed by Eyeglass for failover.