How to - Setup Active Active Appliance Procedures





This procedure is to protect the production Eyeglass appliance using a 2nd licensed Eyeglass appliance.  This solution offers same time sync of DR data to a second appliance that can take over operations under the following conditions:

  1. Controlled failover - switching the active appliance from one data center to the other

  2. Uncontrolled failover protection - the 2nd appliance can be used for failover operations since it has a current near real time synced status and copy of all policies and configuration data (shares, exports and quotas) needed to complete a failover to the surviving cluster.


Definitions:

  1. Active Sync Appliance - Responsible for syncing configuration data and is the primary appliance for all failover operations

  2. Active Monitoring Appliance - Responsible for inventory of configuration data needed for syncing, monitors syncIQ, collects syncIQ job reports


Notes on this solution:

  1. This process means Two Eyeglass appliance have clusters added.

  2. This requires two sets of license keys (license keys are appliance specific)

  3. One appliance has active sync jobs and the other has disabled policies but fully synced view of both clusters






Eyeglass Isilon Edition - SyncIQ DR Orchestration Appliance Overview v28.png



Procedure Overview  below:

-----------------------------------------------------------------------


You will need a second Eyeglass appliance set up. However, it will have clusters added and must be able to reach the clusters overs required ports.  This appliance will have license keys installed and must be purchased separately for the hot hot appliance solution.


Setup 2nd Appliance (Monitor Appliance) procedure:


  1. Install 2nd appliance at the remote data center location

  2. Install license keys (2nd set for the new appliance equal to the number of managed clusters)

  3. Disable the Runbook Robot DR Automation Task (it should only run on the Sync Appliance)

    1. igls admin schedules set --id RunbookRobot --enabled false
  4. Default mode in Eyeglass post installation puts all jobs in user disabled state in the jobs window

  5. Mirror any DFS configuration on the Monitor appliance.  Select all policies that are DFS enabled on the Active Sync appliance and ensure the policies are enabled for DFS mode on the Monitor appliance

    1. NOTE: Any new or updated DFS mode policies on the Active Sync appliance are NOT synced to the Monitor appliance and must be manually updated after any changes are made.

  6. This is the state the jobs will stay in as the Active Monitoring appliance, inventory task will run on this appliance as per default schedule and sync all configuration changes to the Eyeglass appliance.

  7. Email should be configured on the 2nd appliance so that inventory failures can be sent and corrected if the 2nd appliance can not get inventory data from the clusters.  This will require enabling Major alarm filter at a minimum for alarm notifications.

  8. NOTE: with email enabled cluster configuration, reports, RPO reporting will also generate emails from the 2nd appliance

  9. WARNING CAUTION: Never enable any Configuration sync jobs on this appliance, two appliances should never have jobs enabled and by monitoring the same clusters. This can lead to errors, failure to sync, orphan configuration data and is unsupported.



Transfer Active sync function from Production appliance to the Active Monitor Appliance procedure:

  1. To change the Active sync appliance to the Active Monitor appliance

  2. Login and open jobs window select all Configuration sync jobs AND all DFS Enabled jobs, → then bulk actions menu and select disable.  

    1. This ensures all jobs will not perform replication tasks

  3. On the the Active Monitoring appliance:

    1. Login and open the jobs window select all Configuration sync jobs, → then bulk action menu and select Enable.

    2. Now select any DFS enabled policies that were enabled on the Active Sync appliance and verify on the Monitor appliance the policies are DFS enabled and  appear in the DFS Enabled Replication Section in the Jobs Definitions UI,

      1. To re-enable a policy select  bulk action menu and select Enable DFS mode on ALL jobs that should be DFS enabled.

    3. Now Select any job, → bulk action menu and select Run now.  This will start a configuration replication cycle to Sync, compare , audit and update DR dashboard.  This may take time depending on the size of the cluster configuration.

    4. Monitor from the running jobs window until it completes

    5. Verify all jobs are green ok in Jobs definition window

    6. Open DR dashboard and verify DR status for all failover types are showing Green ok.

  4. Optional:

    1. Configure runbook robot again on this appliance in basic or advanced mode (see Failover guide)


Failover procedures from Active Monitor appliance


Use this procedure when: The Active sync appliance is down, unavailable, destroyed and you are confident it will not come back on the network AND you need to failover now AND the Active sync appliance CANNOT be recovered.


Controlled Failover

  1. This process means both source and target clusters are healthy and a failover action needs to be performed AND the Active Sync appliance is down and not recoverable

  2. Login to Monitor appliance → select all Configuration sync jobs with check box and Bulk action Enable

  3. Now switch refer to steps above for installation for DFS mode policies on the Monitor appliance.  They are manually synced between Active Sync and the Monitor appliance so double check all policies are correct before attempting to failover using the monitor appliance.

  4. Select any job, → bulk action menu Run now to force run configuration sync tasks (this will process all jobs not just the one selected).

  5. Monitor from running jobs window to ensure no errors

  6. Verify jobs window jobs show green

  7. Verify DR dashboard shows green for all policies for all modes of failover in use (syncIQ, DFS mode or access zone)

  8. Open DR Assistant → execute failover of the policy that requires failover, ensure the controlled checkbox is enabled

  9. Monitor Failover from DR Assistant

  10. NOTE: Do NOT allow original Active Sync Appliance to resume operations.  

    1. How to recover original Active Sync Appliance?   Delete the VM,  deploy new VM, open support case to get keys re-issued for new Active Monitor appliance.

Uncontrolled Failover


  1. This process means the source cluster is down and a failover action needs to be performed AND the Active Sync appliance is down and not recoverable.

  2. Login to Monitor appliance → select all Configuration sync jobs with check box and Bulk action Enable

  3. Now switch any jobs that were DFS mode (you need to have records of which policy is DFS mode, use naming convention to self document)

  4. Open DR Assistant → execute failover of the policy that requires failover (unselect Controlled failover check box)

  5. Monitor Failover from DR Assistant

  6. NOTE: Do NOT allow original Active Sync Appliance to resume operations.  

    1. How to recover original Active Sync Appliance?   Delete the VM,  deploy new VM, open support case to get keys re-issued for new Active Monitor appliance.


Comments