Failover Process and Customer Role


Contents

  1. 1 Overview
    1. 1.1 Roles and Responsibilities during Failover with Superna Eyeglass
      1. 1.1.1 Superna Eyeglass Support Role in Failover
      2. 1.1.2 Customer Expectations and Role in Failover
    2. 1.2 Day of Failover Support Process
      1. 1.2.1 Objective:
    3. 1.3 Superna Eyeglass Failover Planning Process
      1. 1.3.1 Failover Planning Checklist
        1. 1.3.1.1 1) Mandatory Step: We recommend the latest Eyeglass release be used for planned failover. We allow N-2 release under conditions covered in the Failover Release Notes: http://documentation.superna.net/eyeglass-isilon-edition/plan/failover-release-notes
        2. 1.3.1.2 1A) Mandatory Step: We also require the Failover release notes have been reviewed and all risks to your environment has been assessed. If you have questions, now is the time to ask.
        3. 1.3.1.3 1B) Mandatory Step: If also planning OneFS 7 to 8 upgrade review and execute this procedure after upgrade to onefs 8 for ANY cluster http://documentation.superna.net/eyeglass-isilon-edition/operate/isilon-upgrade-procedure-with-eyeglass
        4. 1.3.1.4 1C) Mandatory - Are you failing over and failback on the same day? yes or no
        5. 1.3.1.5 1D) Mandatory - How long a maintenance window have you scheduled?
        6. 1.3.1.6 1E) Mandatory - How many  Access zones or policies will be failed over?
        7. 1.3.1.7 1F) Mandatory - Is this planned failover an uncontrolled failover where you are simulating source cluster unavailable? Yes or No
        8. 1.3.1.8 2) Mandatory Step: If requesting health check on your installation or planning a failover. Support requires this document planning checklist is completed. This document that has links to all key documents that you need to review and steps to execute before running your DR test:
        9. 1.3.1.9 2A) Do you have contact list in place for failover day?
        10. 1.3.1.10 2B) Do you plan to stop IO to the source cluster before failover?
        11. 1.3.1.11 2C) Have you run domain mark jobs on all SyncIQ policies?
        12. 1.3.1.12 3) Mandatory Step: Operational steps and procedures for the day of failover
        13. 1.3.1.13 4) Mandatory Step: All failover types review document
        14. 1.3.1.14 5) Mandatory Step: Please upload a full Eyeglass Backup 7 days prior to failover for our review of your failover readiness.
        15. 1.3.1.15 6) What is covered by readiness check
    4. 1.4 Note: What about unplanned DR event?



Overview

This document describes:

  • Roles and Responsibilities during Failover with Superna Eyeglass

  • Day of Failover Support Process

  • Superna Eyeglass Failover Planning Process - requires 7 days notice


Roles and Responsibilities during Failover with Superna Eyeglass

Superna Eyeglass Support Role in Failover

Superna Eyeglass End User License Agreement  includes support for the Superna Eyeglass product.  As related to failover this includes:

  • Failover planning process including readiness health check and remediation prior to a planned event

  • Failover log analysis during and post failover health check and failback assessment

  • Root cause of issues during a failover

  • Next steps required to complete a failover under all conditions, all recovery steps are documented.


Superna Eyeglass product support entitlement does not include:

  • Assisted failover of customer data.

    • Professional services from 3rd party Eyeglass Certified partners offer hands on failover services. (services page)

  • Support for any external hardware and software vendors

    • Support agreements must be in place for all external hardware and software vendors. Functional recovery steps will be provided that requires customer subject matter experts to execute on the 3rd party vendor products.

    • Customers must have access to support for (AD, DNS, Networking, Hosts, Isilon), Superna Eyeglass support can provide root cause of external component issues but is not primary support replacement for these components.

    • Superna Eyeglass Support is legally not authorized to take control of any devices or make business decisions on behalf of customers during a failover event Or provide specific technical advice that affects a 3rd party vendor where that vendor should be consulted.

    • Superna support cannot provide assistance that would violate customer support agreements with 3rd party vendors.  

    • Superna Eyeglass Support can not take control or join a Webex or phone call to assist with hands on failover procedures or troubleshooting for 3rd party hardware and software for legal reasons

  • Decisions on data protection before, during and post failover.  

    • These decisions reside with customers. Superna Eyeglass Support can not be responsible for legal reasons.


Customer Expectations and Role in Failover

  • Customers must provide or have access to all skills required to complete a failover and debug any issues in their IT environment which includes (AD, DNS resolution and updates if necessary, AD domain edit permissions to computer objects with ADSI Edit, Isilon knowledge on SyncIQ operations, share/export management, Networking, firewalls, Windows logon process, Linux mount requirements, application specific knowledge that uses NAS shares).


  • Customers must have access to the Superna Support Portal for purpose of uploading failover logs and communicating with Superna Support team on any questions or issues that may arise during the failover.


Day of Failover Support Process


Please find below steps for day of failover. NOTE: Support will follow this process below exactly as written, this is fastest process to complete a failover.


Objective:

  • Verify access to data on the target cluster post failover is supports priority in accessing failover logs.

  • Once data access validated and you have provided confirmation of data access to the case.. Support will then move on to assess failback steps if any errors in failover steps.


    1. BEFORE FAILOVER: Onefs known bug with quota scan. Follow these instructions before failover

    2. DURING FAILOVER: Release > 2.5 will post a message to the failover log indicating when each policy is failed over and data access testing can be begin. Monitor the log and start testing as soon as the message is seen per policy. Failover steps for policy: <policy Name> completed in X minute(s). Data access manual testing should be completed, using this guide as a reference url.

    3. AFTER FAILOVER: Attach failover log from eyeglass as a reply to this email.

    4. AFTER FAILOVER: Test data access by following these instructions http://documentation.superna.net/eyeglass-isilon-edition/design/eyeglass-assisted-failover#TOC-How-to-Validate-and-troubleshoot-A-Successful-Failover-WHEN-Data-is-NOT-Accessible-on-the-Target-Cluster

    5. AFTER FAILOVER: Upload full backup from eyeglass appliance.

    6. AFTER FAILOVER: Reply to this email with results from data access testing.

 

If you are planning to failback on same day, could you please upload a new set of eyeglass full backup (Create Full backup) logs by following this procedure to access the failback readiness: This is mandatory.

http://documentation.superna.net/eyeglass-isilon-edition/get-help/how-to-raise-an-eyeglass-isilon-support-request#TOC-Eyeglass-Full-Backup

 

If your file is bigger than 7MB please follow instructions to upload logs:

http://documentation.superna.net/eyeglass-isilon-edition/get-help/how-to-raise-an-eyeglass-isilon-support-request#TOC-ALTERNATE-UPLOAD-INSTRUCTIONS-for-All-Eyeglass-versions




Superna Eyeglass Failover Planning Process

The Failover Planning process is a series of steps designed to eliminate known risks during a failover event.  It includes planning steps to be completed by the customer and a Superna Eyeglass failover readiness health check and remediation prior to a planned event completed by Superna Eyeglass Support.


Step 1 in the process is to open a support case notifying the Superna Support team of the planned failover.  The Superna Eyeglass Support team will post the Failover planning checklist to the case describing planning process and next steps.


Superna Eyeglass Support requires 7 days notice to allow this process to be followed by opening a case.   Customers that do not follow the process are accepting all risks from release notes and known issues the planning process is designed to eliminate.  Support level is reduced for customers that do not follow documented procedures to eliminate known risks.

Failover Planning Checklist

For a planned failover please review the following to maintain support for your installation. We provide a planning guide and checklist excel file here. We provide a complete planning process and customer role.


  1. STEP 1 We require written acknowledgement of the failover mandatory steps below to be posted to the case. This process ensures you have read and understood all steps for failover to ensure a successful failover and ownership of the failover process resides with your company.

  2. STEP 2: Submit support logs for review of your environment (NOTE: Until step 1 is complete. support is unable to review logs.) NOTE: The readiness check only includes items listed at the end of this message.

  3. STEP 3: Support will determine next steps based on review and post to the case.

  4. NOTE1: Customers must have access to support for (AD, DNS, Networking, Hosts, Isilon), Eyeglass support can provide root cause of external component issues but is not primary support replacement for these components.

  5. NOTE2: support can not take control or join a Webex to assist with hands on failover procedures for 3rd party hardware and software for legal reasons.

  6. NOTE3: Support agreements must be in place for all external hardware and software vendors. Functional recovery steps will be provided that requires customer subject matter experts to execute on the 3rd party vendor products.

  7. NOTE4: Decisions on data protection before, during and post failover reside with customers. Support can not be responsible

  8. NOTE5: For a permanent link to policies and support expectations for planned failovers refer to this link.

Technical details for failover planning are provided below. We would like to schedule a 30 minute webex to review these failover planning mandatory steps as well as the day of failover support process and answer any questions you might have. Please provide us with your availability so that we can schedule.



 

 

1) Mandatory Step: We recommend the latest Eyeglass release be used for planned failover. We allow N-2 release under conditions covered in the Failover Release Notes: http://documentation.superna.net/eyeglass-isilon-edition/plan/failover-release-notes

Acknowledged: Yes/No

 

1A) Mandatory Step: We also require the Failover release notes have been reviewed and all risks to your environment has been assessed. If you have questions, now is the time to ask.

Acknowledged: Yes/No

 

>> We provide notice when a release has support removed and requirement is to upgrade .We improve failover from all cluster failovers from all customers each release so without upgrading to latest release you don't benefit from other customers failovers.

 

>> We send all Eyeglass release notifications to @eyeglassisilon or email distribution

 

1B) Mandatory Step: If also planning OneFS 7 to 8 upgrade review and execute this procedure after upgrade to onefs 8 for ANY cluster http://documentation.superna.net/eyeglass-isilon-edition/operate/isilon-upgrade-procedure-with-eyeglass

Acknowledged: Yes/No

 

1C) Mandatory - Are you failing over and failback on the same day? yes or no

NOTE: We recommend at least 4 hours between any failover and failback within a single day.

 

1D) Mandatory - How long a maintenance window have you scheduled?

 

1E) Mandatory - How many  Access zones or policies will be failed over?


1F) Mandatory - Is this planned failover an uncontrolled failover where you are simulating source cluster unavailable? Yes or No

 

If YES, the ONLY supported procedure using Superna Eyeglass is documented here: Eyeglass Simulated Disaster Event Test Procedure.

 

ANY change to the documented procedure is NOT supported.

 

2) Mandatory Step: If requesting health check on your installation or planning a failover. Support requires this document planning checklist is completed. This document that has links to all key documents that you need to review and steps to execute before running your DR test:

http://documentation.superna.net/eyeglass-isilon-edition/plan/failover-planning-guide-and-checklist

 

Acknowledged: Yes/No

 

2A) Do you have contact list in place for failover day?

Acknowledged: Yes/No

 

Customers must provide or have access to all skills required to complete a failover and debug any issues in their IT environment which includes (AD, DNS resolution and updates if necessary, AD domain edit permissions to computer objects with adsiedt, Isilon knowledge on SyncIQ operations, share/export management, Networking, firewalls, Windows logon process, Linux mount requirements, application specific knowledge that uses NAS shares)

 

Superna Eyeglass Support agreement is for the Superna Eyeglass product and is not a replacement for skill or support agreement for all external hardware and software vendors.

 

 

2B) Do you plan to stop IO to the source cluster before failover?

 

Acknowledged: Yes/No

 

If yes, for SMB protocol the 2.0 or later feature can be used to block IO to shares with DR assistant. This inserts a deny read permission dynamically before failover starts and removes after failover completes.

NFS requires the protocol to be disable to guarantee no IO. Exports should be unmounted before disabling the the protocol on the cluster.

 

If no, you will incur data loss.

 

2C) Have you run domain mark jobs on all SyncIQ policies?

Consult explanation here.

 

Acknowledged: Yes/No

 

3) Mandatory Step: Operational steps and procedures for the day of failover

 

 

Acknowledged: Yes/No

 

 

  1. Operational Steps for Eyeglass Assisted Access Zone Failover

    1. How to Monitor the Eyeglass Assisted Failover

    2. How to Validate and troubleshoot A Successful Failover WHEN Data is NOT Accessible on the Target Cluster

    3. Post Access Zone Failover Manual Steps

    4. Post Access Zone Failover Checklist

    5. Troubleshooting Failover

  2. Operational Steps for Eyeglass Microsoft DFS Mode Failover

    1. How to Monitor the Eyeglass Assisted Failover

    2. How to Validate and troubleshoot A Successful Failover WHEN Data is NOT Accessible on the Target Cluster

    3. Post Eyeglass Microsoft DFS Mode Failover Checklist

 

 

4) Mandatory Step: All failover types review document

  1. Failover Recovery Procedures


 

Acknowledged: Yes/No

 

 

5) Mandatory Step: Please upload a full Eyeglass Backup 7 days prior to failover for our review of your failover readiness.

 

Please upload a new set of eyeglass full backup logs (Create Full backup) by following this procedure:

http://documentation.superna.net/eyeglass-isilon-edition/get-help/how-to-raise-an-eyeglass-isilon-support-request#TOC-Eyeglass-Full-Backup

 

If your file is bigger than 7MB please follow instructions to upload logs:

http://documentation.superna.net/eyeglass-isilon-edition/get-help/how-to-raise-an-eyeglass-isilon-support-request#TOC-ALTERNATE-UPLOAD-INSTRUCTIONS-for-All-Eyeglass-versions

 

 

Acknowledged: Yes/No

 

6) What is covered by readiness check

As per your request to review readiness for failover, we will review the following for failover readiness validation:

- Access Zone Failover hints

- Access Zone Readiness status / DFS Readiness status / Policy Readiness Status

- SPN errors

- Eyeglass misconfigurations (example Eyeglass version)

- SyncIQ Domain Mark

- Planned Failover Type

- Client Redirection Guide Followed (DNS Dual delegation/DFS Dual Delegation)

- Eyeglass appliance health check

- Cluster API response health check


Anything other than the above is not checked. Anything else not listed needs to be verified by yourselves in the context of your overall failover plan.


For a full review of readiness, best practices and assessment of failover design, we offer the following purchased service:

http://documentation.superna.net/eyeglass-isilon-edition/get-help/eyeglass-services/services-dr-event-audit


 

Acknowledged: Yes/No

 

Note: What about unplanned DR event?

Unplanned (real dr with source cluster destroyed ) no it's not required to be on latest release. However we always expect the software is upgraded to supported releases published here http://documentation.superna.net/eyeglass-isilon-edition/software-release-download. We provide notifications to @eyeglassisilon or email distribution