Isilon Upgrade Procedure with Eyeglass





This document provides the steps for upgrading Isilon clusters that are using Eyeglass.

Prior to Upgrade

  1. Check the Feature Release Compatibility table in the appropriate Eyeglass Release Notes to ensure cluster release compatibility listing is supported for the cluster configuration upgrade for the source and destination cluster.

    • If your upgrade process will always have the source and destination cluster in a supported configuration AND upgrade does not change version first 2 digits (for example 7.2.1.1 upgrade to 7.2.1.2), follow instructions for Supported Feature Release Compatibility maintained during Upgrade - no major version change.


    • If your upgrade process is for OneFS 7.2.x to OneFS 8.0.x follow instructions for Supported Feature Release Compatibility maintained during Upgrade - OneFS 7.2.x to OneFS 8.0.x.


    • If your upgrade process will result in source and destination cluster being in an unsupported configuration at any point follow instructions for Supported Feature Release Compatibility NOT maintained during Upgrade.


Supported Feature Release Compatibility maintained during Upgrade - no major version change

  1. Login to Eyeglass web page prior to upgrade and confirm that configuration replication is in a good state.

  2. Begin the upgrade process.  Eyeglass is in a running state during upgrade process.

  3. If source or destination cluster is not reachable at any point during upgrade you may see failed Eyeglass Configuration Replication Jobs and their related alarms. Subsequent Eyeglass configuration replication Job execution with source and target clusters in reachable state should clear these alarms


Eyeglass Alarms during Isilon Upgrade


The following provides a summary of the Eyeglass alarms that you might encounter during an Isilon upgrade where source and/or target cluster are unreachable:


One or both clusters not reachable when Eyeglass Replication Task is starting


  • Replication jobs failed.

  • DR Dashboard has Error status.

  • 3 Alarms raised for each scheduled replication job.








One or both cluster not available when Eyeglass making an add/update/delete against a configuration object (share/export/quota)


  • Some jobs succeeded and some failed.

  • DR Dashboard has Error status.

  • Alarm raised for failed jobs and failed audits.






One or both cluster are not available when Eyeglass performing post-replication discovery task


  • Jobs succeeded and audits failed.

  • DR Dashboard shows Error status.

  • Alarm raised for failed audits.




Post Upgrade

It is recommended to leave Eyeglass running for 30 minutes after the upgrade procedure has been completed and then check the status of the Eyeglass Jobs to confirm that system has returned to pre-upgrade state.  Any alarms that occurred can be seen in the Alarms History.





Supported Feature Release Compatibility maintained during Upgrade - OneFS 7.2.x to OneFS 8.0.x

Description

This document is the procedure for performing site cluster rolling upgrade from OneFS 7.2.x to OneFS 8.0.x.  This simplifies the process and reduces the risk to migrate to OneFS 8 at the DR cluster, and allow failback to OneFS 7.


How To Video


Multi Data Center Rolling Site Upgrade OneFS 7 to OneFS 8 with Eyeglass


Scenario

Current setup prior to upgrade

Both sites’ clusters are on the OneFS 7.2.x and Eyeglass has been configured properly to manage both clusters with Eyeglass Access Zone and DFS Mode Failover settings.


Upgrade Plan

To upgrade both site’s clusters to OneFS 8.0.x (8.0.0.1) by performing upgrade on 1 site at a time. The plan is to upgrade the Target Cluster first and then follow by upgrade the Source Cluster.



Procedures:


Upgrade Target Cluster:


  1. Verify that prior to upgrade, the Eyeglass has been configured properly and Eyeglass Jobs statuses are OK. No error.

  2. Prepare Target Cluster for Upgrade. Refer to EMC Isilon documentation for detailed instructions, including:

    1. Pre-upgrade checks

    2. Supported upgrade paths from current OneFS release to the target release. Might need to upgrade to intermediate release before final upgrade to the target release.

    3. Download and store the Isilon installer to the Cluster (e.g. /ifs/data folder).


  1. Disable Eyeglass Jobs temporary for upgrade process.

  2. Perform OneFS upgrade on Target Cluster. (Take note that OneFS upgrade from 7.2.x to 8.0.0.1 is simultaneous upgrade. Cluster nodes are rebooted simultaneously). Eyeglass will report that the cluster is unreachable.

  3. Once upgrade process of Target Cluster has completed successfully, perform the following tasks to let Eyeglass manage the Target Cluster that now has new OneFS Release 8.0.0.1 (Refer to Eyeglass online documentation:

    1. Update the Eyeglass Admin role privileges on this Target Cluster by adding the additional permissions list in the  document below:

      1. http://documentation.superna.net/eyeglass-isilon-edition/tech-notes/isilon-cluster-user-minimum-privileges-for-eyeglass


    1. Update the sudo privileges for the Eyeglass service account on this Target Cluster:

      1. As per the document above for OneFS 8 sudoer settings


  1. Restart Eyeglass sca service:

    1. SSH as admin user

    2. Then sudo -s

    3. Enter admin password

    4. To stop: systemctl stop sca.service

    5. To start: systemctl start sca.service

    6. Verify: systemctl status sca.service

  2. Re-enable the Eyeglass Jobs

  3. Manually Run with run now bulk actions menu from jobs window or wait until the next Replication Job has started. Once Jobs have completed, verify Eyeglass status is OK. No error.

  4. Verify all Jobs are green and DR Dashboard audit and readiness for zones status

  5. NOTE: At this point in the site cluster upgrade path, Eyeglass is now in mixed mode, which means OneFS 7 API takes priority when syncing configuration information from source to target cluster.

  6. NOTE: This mixed mode means only OneFS 7 object schema will be used when syncing to OneFS 8 clusters.


What you need to know in Mixed mode


In this mixed mode:

  • config is supported between clusters running 7 to 8 OneFS releases for exact versions supported consult the release notes for the latest release to ensure you are running a supported 7 to 8 release

  • Failover is supported

  • Failback is supported

  • Cluster reporting is supported but only OneFS 7 data will be retrieved from OneFS 8 clusters  

  • Runbook Robot is supported

  • RPO reporting is supported


Its expect this mixed mode will be a temporary scenario before upgrade the source cluster and this period would be weeks to months at most.  It is not intended to be a normal operating mode between replicating clusters



Upgrade Source Cluster:


  1. Verify that prior to upgrade, the Eyeglass Jobs statuses are OK. No error.

  2. Prepare Target Cluster for Upgrade. Refer to EMC Isilon documentation for detailed instructions, including:

    1. Pre-upgrade checks

    2. Supported upgrade paths from current OneFS release to the target release. Might need to upgrade to intermediate release before final upgrade to the target release.

    3. Download and store the Isilon installer to the Cluster (e.g. /ifs/data folder).


  1. Perform Eyeglass Failover to Target Cluster. (see documentation for the mode of failover and follow the planning guide)

  2. Verify Failover has completed successfully. No errors in the failover log.

  3. By default, all eyeglass jobs for Mirror Policies are in User Disabled state.  If not disabled (e.g, have done failover and failback before), manually disable all Eyeglass Jobs temporary upgrade process.

  4. Perform OneFS upgrade on Source Cluster. (Take note that OneFS upgrade from 7.2.x to 8.0.0.1 is simultaneous upgrade. Cluster nodes are rebooted simultaneously). Eyeglass will report that the cluster is unreachable.

  5. Once upgrade process of Source Cluster has completed successfully, perform the following tasks to let Eyeglass manage the Source Cluster that now has new OneFS Release 8.0.0.1 (Refer to Eyeglass online documentation: http://documentation.superna.net/eyeglass-isilon-edition/tech-notes/isilon-cluster-user-minimum-privileges-for-eyeglass)

    1. Update the Eyeglass Admin role privileges on this Source Cluster by adding additional permissions list in the  document below:

      1. http://documentation.superna.net/eyeglass-isilon-edition/tech-notes/isilon-cluster-user-minimum-privileges-for-eyeglass



    1. Update the sudo privileges for the Eyeglass service account on this Source Cluster:

      1. additional sudor for OneFS 8 in the  document below:

        1. http://documentation.superna.net/eyeglass-isilon-edition/tech-notes/isilon-cluster-user-minimum-privileges-for-eyeglass


  1. Restart Eyeglass sca service:

    1. SSH as admin user

    2. Then sudo -s

    3. Enter admin password

    4. To stop: systemctl stop sca.service

    5. To start: systemctl start sca.service

    6. Verify: systemctl status sca.service

  2. Re-enable the Eyeglass Jobs

  3. Run or wait till the next Replication Jobs has started. Once Jobs have completed, verify Eyeglass status is OK. No error.

  4. NOTE: if the only pair of clusters managed by Eyeglass are OneFS 8 at this point mixed mode exists automatically and API support is now OneFS 8 allowing all new attributes on shares, exports, quotas from OneFs 8 schema to sync normally





Supported Feature Release Compatibility NOT maintained during Upgrade


  1. If you are using the Eyeglass 1.4 Runbook Robot feature, disable it from the Eyeglass appliance command line as described here.  The Eyeglass CLI command can be executed directly from the Eyeglass shell or any ssh session to the Eyeglass appliance.

  2. Login to Eyeglass web page prior to upgrade and confirm that configuration replication is in a good state.

  3. Open the Jobs window. Make a list of all Jobs that are in the User Disabled State. (You will need this list post upgrade to ensure that these Jobs remain disabled)

  4. Disable ALL Eyeglass configuration replication Jobs:

    1. Open the Jobs window.

    2. Select the checkbox at the top of the Job Name column to select all Jobs.

    3. Select the Select a bulk action button and then select the Enable/Disable option.

    4. All Jobs will be disabled (User Disabled unless already Policy Disabled) and will not be executed on the regular Replication Task schedule.

!IMPORTANT

At this point, configuration is not being replicated between the source and target clusters.  Any configuration changes made on the source cluster are not replicated to the target.







  1. Begin the Isilon Upgrade Process.

  2. Once both nodes are at a compatible release lineup as per the Feature Release Compatibility matrix, Eyeglass configuration replication Jobs may be enabled again:


If there were NO User Disabled Jobs prior to the Upgrade:

  1. Open the Jobs window.

  2. Select the checkbox at the top of the Job Name column to select all Jobs.

  3. Select the Select a bulk action button and then select the Enable/Disable option.

  4. All Jobs will be enabled and will display their last known replication state.


If there were User Disabled Jobs prior to the Upgrade:

  1. Open the Jobs window.

  2. Select check box for each Job manually EXCEPT for those Jobs that were User Disabled State prior to the upgrade.

  3. Select the Select a bulk action button and then select the Enable/Disable option.

  4. All selected Jobs will be enabled and will display their last known replication state.


If you were using Runbook Robot and had disabled it prior to the Isilon Upgrade:

  1. ssh to the Eyeglass appliance and login as root or sudo su - to root.

  2. Enable the Runbook Robot feature from the Eyeglass appliance command line as described here.   


Post Upgrade

It is recommended to leave Eyeglass running for 30 minutes after the upgrade procedure has been completed and then check the status of the Eyeglass Jobs to confirm that system has returned to pre-upgrade state.