Isilon Upgrade Procedure with Eyeglass

Isilon Upgrade Procedure with Eyeglass







Product Name - Superna Eyeglass

Revision Changes to this Document - December 2017


.



Introduction to this Guide

Overview

This document provides the steps for upgrading Isilon clusters that are using Eyeglass.

Help

Should you have any issues that are not addressed in this guide, Superna offers support in several forms; on line, voicemail, Email, or live on line chat.

  1. The support site provides online ticket submission and case tracking.  Support Site link - support.superna.net 

  2. Leave a voicemail at 1 (855) 336-1580

In order to provide service you must have an account in our system. When calling in leave; customer name, email, description of question or issue, and primary contact for your company. We will  assign the case to primary contact for email followup.

  1. Email eyeglasssupport@superna.net

  2. To download license keys please go to the following  license keys.

  3. You can also raise a case right from in Eyeglass desktop using the help button, search for your issue and if want to raise a case or get a question answered, click the “leave us a message”  with your name, email and appliance ID and a case is opened directly from Eyeglass.

 http://site.superna.net/_/rsrc/1472870726155/support/LeaveUsAMessage.png?height=200&width=167

  1. Or get Support Using Chat M-F 9-5 EDT  (empty box?  we are not online yet)

  2. Eyeglass Live Chat 

  3. You should also review our support agreement here.

Prior to Upgrade

Check the Feature Release Compatibility table in the appropriate Eyeglass Release Notes to ensure cluster release compatibility listing is supported for the cluster configuration upgrade for the source and destination cluster.

    • If your upgrade process will always have the source and destination cluster in a supported configuration AND upgrade does not change version first 2 digits (for example 7.2.1.1 upgrade to 7.2.1.2), follow instructions for Supported Feature Release Compatibility maintained during Upgrade - no major version change.

    • If your upgrade process is for OneFS 7.2.x to OneFS 8.0.x follow instructions for Supported Feature Release Compatibility maintained during Upgrade - OneFS 7.2.x to OneFS 8.0.x.

    • If your upgrade process will result in source and destination cluster being in an unsupported configuration at any point follow instructions for Supported Feature Release Compatibility NOT maintained during Upgrade.

Supported Feature Release Compatibility maintained during Upgrade - no major version change

  1. Login to Eyeglass web page prior to upgrade and confirm that configuration replication is in a good state.

  2. Begin the upgrade process.  Eyeglass is in a running state during upgrade process.

  3. If source or destination cluster is not reachable at any point during upgrade you may see failed Eyeglass Configuration Replication Jobs and their related alarms. Subsequent Eyeglass configuration replication Job execution with source and target clusters in reachable state should clear these alarms.

Eyeglass Alarms during Isilon Upgrade

The following provides a summary of the Eyeglass alarms that you might encounter during an Isilon upgrade where source and/or target cluster are unreachable:

One or both clusters not reachable when Eyeglass Replication Task is starting

  • Replication jobs failed.

  • DR Dashboard has Error status.

  • 3 Alarms raised for each scheduled replication job.





One or both cluster not available when Eyeglass making an add/update/delete against a configuration object (share/export/quota)

  • Some jobs succeeded and some failed.

  • DR Dashboard has Error status.

  • Alarm raised for failed jobs and failed audits.



One or both cluster are not available when Eyeglass performing post-replication discovery task

  • Jobs succeeded and audits failed.

  • DR Dashboard shows Error status.

  • Alarm raised for failed audits.




Post Upgrade

It is recommended to leave Eyeglass running for 30 minutes after the upgrade procedure has been completed and then check the status of the Eyeglass Jobs to confirm that system has returned to pre-upgrade state.  Any alarms that occurred can be seen in the Alarms History.

Supported Feature Release Compatibility maintained during Upgrade - OneFS 7.2.x to OneFS 8.0.x

Description

The following is the procedure for performing site cluster rolling upgrade from OneFS 7.2.x to OneFS 8.0.x.  This simplifies the process and reduces the risk to migrate to OneFS 8 at the DR cluster, and allow failback to OneFS 7.

How To Video

Multi Data Center Rolling Site Upgrade OneFS 7 to OneFS 8 with Eyeglass

Upgrade Prerequisite

Both sites’ clusters are on the OneFS 7.2.x and Eyeglass has been configured properly to manage both clusters with Eyeglass Access Zone and DFS Mode Failover settings.

Upgrade Procedure

To upgrade both site’s clusters to OneFS 8.0.x (8.0.0.1) by performing upgrade on 1 site at a time. The plan is to upgrade the Target Cluster first and then follow by upgrade the Source Cluster.

Upgrade Target Cluster first:

  1. Verify that prior to upgrade, the Eyeglass has been configured properly and Eyeglass Jobs statuses are OK. No error.

  2. Prepare Target Cluster for Upgrade. Refer to EMC Isilon documentation for detailed instructions, including:

    1. Pre-upgrade checks

    2. Supported upgrade paths from current OneFS release to the target release. Might need to upgrade to intermediate release before final upgrade to the target release.

    3. Download and store the Isilon installer to the Cluster (e.g. /ifs/data folder).

  3. Disable Eyeglass Jobs temporary for upgrade process.

  4. Perform OneFS upgrade on Target Cluster. (Take note that OneFS upgrade from 7.2.x to 8.0.0.1 is simultaneous upgrade. Cluster nodes are rebooted simultaneously). Eyeglass will report that the cluster is unreachable.

  5. Once upgrade process of Target Cluster has completed successfully, perform the following tasks to let Eyeglass manage the Target Cluster that now has new OneFS Release 8.0.0.1 (Refer to Eyeglass online documentation):

    1. Update the Eyeglass Admin role privileges on this Target Cluster by adding the additional permissions list in the  document below:

      1. Isilon Cluster User Minimum Privileges for Eyeglass

    2. Update the sudo privileges for the Eyeglass service account on this Target Cluster:

      1. As per the document above for OneFS 8 sudoer settings

  6. Restart Eyeglass sca service:

    1. SSH as admin user.

    2. Then sudo -s.

    3. Enter admin password.

    4. To stop: systemctl stop sca.service.

    5. To start: systemctl start sca.service.

    6. Verify: systemctl status sca.service.

  7. Re-enable the Eyeglass Jobs.

  8. Manually Run with Run Now bulk actions menu from Jobs Window or wait until the next Replication Job has started. Once Jobs have completed, verify Eyeglass status is OK. No error.

  9. Verify all Jobs are green and DR Dashboard audit and readiness for zones status

  10. NOTE: At this point in the site cluster upgrade path, Eyeglass is now in mixed mode, which means OneFS 7 API takes priority when syncing configuration information from source to target cluster.

  11. NOTE: This mixed mode means only OneFS 7 object schema will be used when syncing to OneFS 8 clusters.

What you need to know in Mixed mode

In this mixed OneFS7/OneFS8 mode:

  • Config is supported between clusters running 7 to 8 OneFS releases. To ensure you are running a supported 7 to 8 release, consult the release notes for exact versions supported.

  • Failover is supported.

  • Failback is supported.

  • Cluster reporting is supported but only OneFS 7 data will be retrieved from OneFS 8 clusters .

  • Runbook Robot is supported.

  • RPO reporting is supported.

Its expect this mixed mode will be a temporary scenario before upgrading the source cluster (expectation is a period would be weeks to months at most).  It is not intended to be a normal operating mode between replicating clusters

Upgrade Source Cluster second:

  1. Verify that prior to upgrade, the Eyeglass Jobs statuses are OK. No error.

  2. Prepare Target Cluster for Upgrade. Refer to EMC Isilon documentation for detailed instructions, including:

    1. Pre-upgrade checks

    2. Supported upgrade paths from current OneFS release to the target release. Might need to upgrade to intermediate release before final upgrade to the target release.

    3. Download and store the Isilon installer to the Cluster (e.g. /ifs/data folder).

  3. Perform Eyeglass Failover to Target Cluster. (see documentation for the mode of failover and follow the Failover Planning Guide and checklist).

  4. Verify Failover has completed successfully. No errors in the failover log.

  5. By default, all Eyeglass Jobs for Mirror Policies are in User Disabled state.  If not disabled (e.g, have done failover and failback before), manually disable all Eyeglass Jobs temporary upgrade process.

  6. Perform OneFS upgrade on Source Cluster. (Take note that OneFS upgrade from 7.2.x to 8.0.0.1 is simultaneous upgrade. Cluster nodes are rebooted simultaneously). Eyeglass will report that the cluster is unreachable.

  7. Once upgrade process of Source Cluster has completed successfully, perform the following tasks to let Eyeglass manage the Source Cluster that now has new OneFS Release 8.0.0.1 (Refer to Eyeglass online documentation Isilon Cluster User Minimum Privileges for Eyeglass)):

    1. Update the Eyeglass Admin role privileges on this Source Cluster by adding additional permissions list in the  document below:

      1. Isilon Cluster User Minimum Privileges for Eyeglass

    2. Update the sudo privileges for the Eyeglass service account on this Source Cluster:

      1. Additional sudor for OneFS 8 in the  document below:

        1. Isilon Cluster User Minimum Privileges for Eyeglass

  8. Restart Eyeglass sca service:

    1. SSH as admin user.

    2. Then sudo -s.

    3. Enter admin password.

    4. To stop: systemctl stop sca.service.

    5. To start: systemctl start sca.service.

    6. Verify: systemctl status sca.service.

  9. Re-enable the Eyeglass Jobs.

  10. Run or wait till the next Replication Jobs have started. Once Jobs have completed, verify Eyeglass status is OK. No error.

NOTE: if the only pair of clusters managed by Eyeglass are OneFS 8 at this point mixed mode exists automatically and API support is now OneFS 8 allowing all new attributes on shares, exports, quotas from OneFs 8 schema to sync normally.

Supported Feature Release Compatibility NOT maintained during Upgrade

  1. If you are using the Eyeglass 1.4 Runbook Robot feature, disable it from the Eyeglass appliance command line as described in Eyeglass Administration Guide.  The Eyeglass CLI command can be executed directly from the Eyeglass shell or any ssh session to the Eyeglass appliance.

  2. Login to Eyeglass web page prior to upgrade and confirm that configuration replication is in a good state.

  3. Open the Jobs window. Make a list of all Jobs that are in the User Disabled State. (You will need this list post upgrade to ensure that these Jobs remain disabled)

  4. Disable ALL Eyeglass configuration replication Jobs:

    1. Open the Jobs window.

    2. Select the checkbox at the top of the Job Name column to select all Jobs.

    3. Select the Select a bulk action button and then select the Enable/Disable option.

    4. All Jobs will be disabled (User Disabled unless already Policy Disabled) and will not be executed on the regular Replication Task schedule.

!IMPORTANT

At this point, configuration is not being replicated between the source and target clusters.  Any configuration changes made on the source cluster are not replicated to the target.

  1. Begin the Isilon Upgrade Process.

  2. Once both nodes are at a compatible release lineup as per the Feature Release Compatibility matrix, Eyeglass configuration replication Jobs may be enabled again:

If there were NO User Disabled Jobs prior to the Upgrade:

  1. Open the Jobs window.

  2. Select the checkbox at the top of the Job Name column to select all Jobs.

  3. Select the Select a bulk action button and then select the Enable/Disable option.

  4. All Jobs will be enabled and will display their last known replication state.

If there were User Disabled Jobs prior to the Upgrade:

  1. Open the Jobs window.

  2. Select check box for each Job manually EXCEPT for those Jobs that were User Disabled State prior to the upgrade.

  3. Select the Select a bulk action button and then select the Enable/Disable option.

  4. All selected Jobs will be enabled and will display their last known replication state.

If you were using Runbook Robot and had disabled it prior to the Isilon Upgrade:

  1. ssh to the Eyeglass appliance and login as root or sudo su - to root.

  2. Enable the Runbook Robot feature from the Eyeglass appliance command line as described Eyeglass Administration Guide.   

Post Upgrade

It is recommended to leave Eyeglass running for 30 minutes after the upgrade procedure has been completed and then check the status of the Eyeglass Jobs to confirm that system has returned to pre-upgrade state.