Microsoft DFS Mode Failover Guide

Eyeglass SyncIQ Failover and Failback with Microsoft DFS

Product Name - Superna Eyeglass

Revision Changes to this Document

Contents

  1. 1 Product Name - Superna Eyeglass
    1. 1.1 Revision Changes to this Document
    2. 1.2 Video How To - Overview
    3. 1.3 Key Values
    4. 1.4 Windows DFS Client OS Compatibility and Release Notes
    5. 1.5 Release Note
    6. 1.6 DFS Feature Changes and Share names used on DFS synced Shares
      1. 1.6.1 DFS Fast Failover Mode - Superna Eyeglass 1.5.2 and beyond
      2. 1.6.2 DFS Failover Enhancements
    7. 1.7 Help
  2. 2 How DFS mode with Eyeglass Works
    1. 2.1 Process Flow for DFS Failover with Eyeglass
      1. 2.1.1 Normal
      2. 2.1.2 Failover with Eyeglass DFS mode
      3. 2.1.3 Failback with Eyeglass
      4. 2.1.4 Eyeglass Microsoft DFS Mode Failover with NFS Export
  3. 3 Requirements for Eyeglass Microsoft DFS Mode Failover
    1. 3.1 Cluster Version Requirements
    2. 3.2 SyncIQ Policy Requirements - Blocks Failover
    3. 3.3 Failover Target Cluster Requirements - Blocks Failover
    4. 3.4 Eyeglass Quota Job Requirements
    5. 3.5 Active Directory
  4. 4 Considerations for Eyeglass Microsoft DFS Mode Failover
    1. 4.1 SyncIQ Policy Recommendations
  5. 5 Preparing your System for the Eyeglass Microsoft DFS Mode Failover
    1. 5.1 Overview
    2. 5.2 Preparation Steps
  6. 6 Failover Planning and Checklist
  7. 7 Monitoring DR Readiness for Eyeglass Assisted Failover
    1. 7.1 Eyeglass DR Readiness for DFS Enabled SyncIQ Policies
  8. 8 Operational Steps for Eyeglass Microsoft DFS Mode Failover
    1. 8.1 Overview
    2. 8.2 Procedure for Running Eyeglass Microsoft DFS Mode Failover
  9. 9 Post Eyeglass Microsoft DFS Mode Failover Manual Steps for NFS Exports
  10. 10 Post Eyeglass Microsoft DFS Mode Failover Checklist
    1. 10.1 Procedure for Checking your SMB Clients Post DFS Failover:
  11. 11 Advanced DFS mode Setup with Access Zones
    1. 11.1 Configuration Steps

Overview

The Eyeglass DFS solution for Isilon greatly simplifies DR with DFS.  The solution allows DFS to maintain targets (UNC paths) that point at both source and destination clusters.

The failover and failback operations are initiated from Eyeglass and move configuration data to the writeable copy of the UNC target. Grouping of shares by SyncIQ policy allows Eyeglass to automatically protect shares added to the Isilon.  Quotas are also detected and protected automatically.

We recommend using Domain based DFS with Eyeglass as the most highly available solution for clients versus server based DFS roots.

Video How To - Overview

How to setup Microsoft DFS failover Mode with Superna Eyeglass

Key Values

  • Isilon Configuration synchronized to active cluster!

    • Integrated with Isilon SyncIQ and Eyeglass Orchestrated Failover.

    • Supports Quotas on Isilon during failover and failback operations.

  • Supports partial Access zone failover solutions. I.E per application level failover and failback within the Access zone (requires detected IP pool with Eyeglass ignore hints)

  • NO DFS administration MMC  tasks required for failover / failback!

    • DFS referral list contains all targets (production and DR clusters).

    • Automatically detected DFS targets protected.

    • Kerberos ticket caching services failover without needing to clear Kerberos cache on clients.

    • Avoids NTLM fallback authentication.

  • NO CLIENT UNMOUNT needed!

    • Clients should cache locally when selecting a target UNC path.

    • Clients auto select writeable copy of the DFS mount.

  • Apple OS X supports DFS mode!

    • OS X Yosemite supports DFS with Active Directory if joined to the domain

Screen Shot 2015-12-04 at 12.39.12 PM.png



Windows DFS Client OS Compatibility and Release Notes


The following list of Windows OS clients have all been tested.

  1. Windows 7, 8.x, 10

  2. Windows Server 2008, 2012, 2012 R2, 2016


Release Note

  1. Onefs 8 has CA compatibility mode capability advertised called persistent file handles, This is required for Continuous Availability Mode support with Isilon.  Windows servers in cluster mode only advertise this capability when using CA mode.

    1. An issue arise from Onefs 8 when NOT using CA mode this capability is still advertised to clients.

    2. This can trigger the issue described in KB article below “The computer can have more time to determine whether a shared folder is available if there is a failover of the shared folder.

    3. NOTE: This issue only affected client machines with no active connection to Isilon shares mounted over DFS.  A delay was seen when mounting the DFS root that would complete within one minute.  NOTE: Actively mounted shares or mapped to DFS folders directly did not see this delay.

  2. Windows 10 and 2012 server have a registry setting described in the link below to correct this behavior.    Windows 8 and 2012 server require a hotfix to be applied.

    1. https://support.microsoft.com/en-us/help/2820470/delayed-error-message-when-you-try-to-access-a-shared-folder-that-no-l



DFS Feature Changes and Share names used on DFS synced Shares

To streamline how DFS mode with normal configuration sync,  the feature has been enhanced as per the table below.   This change allows new options for customers that require access to DR cluster data in read-only state, and preserves DFS mode functionality.  It also reduces risk of issues during failover.

New feature to hide shares on the DR cluster or read only cluster are now available.  After switching the tag, the next configuration sync cycle will apply the changes to the DR cluster share names.

Eyeglass Version

DFS Mode Sync Mode

DFS mode failover Behavior

Prefix on Shares

Post fix on shares for Security on DR Cluster

1.4.x and earlier

Delete shares of the same name on DR cluster

Create shares on DR cluster and delete on source cluster



1.5 and beyond

Create shares on DR cluster with a renamed prefix added to the share name

Rename share on DR cluster and rename source cluster with a prefix added to the share name

igls-dfs (this default can be changed and be changed to another string by editing the tag <dfsshareprefix>igls-dfs-</dfsshareprefix> in /opt/superna/sca/data/system.xml )

WARNING: If DFS is enabled, changing this tag will NOT clean up shares with the previous name.  Clean up is manual and new shares will be created with the new tag.


1.6 and beyond




New feature to allow the source active  cluster share to be visible BUT a $ post fix can be applied on the Synced igl-dfs-sharename$ to hide the share on the DR cluster.  After failover the share names will flip and hide it on the Source cluster.  This can be enabled by editing /opt/superna/sca/data/system.xml file.  Change the tag to include $ as shown below <dfssharesuffix>$</dfssharesuffix>  NOTE: on an existing installation the old DFS renamed share will need to be manually deleted


NOTE: When 1.5 DFS mode is enabled,  all shares found on source will be created on the target cluster with the prefix applied to the share name.  If upgrading from 1.4.x DFS mode, no action is required and shares will be created using 1.5 logic.

DFS Fast Failover Mode - Superna Eyeglass 1.5.2 and beyond


Release

Speed Improvement

Notes

1.5.2 >

For DFS Failover (Microsoft DFS Mode or DFS enabled Job in an Access Zone Failover), the Renaming shares step occurs after Data sync and before the Policy Failover step (Allow Writes, Resync Prep).  This ensures that the amount of time that DFS clients are directed to the failover source cluster is minimized once the failover has started and that the DFS clients are already directed to the target cluster when the filesystem becomes writeable.  

NOTE:  During failover, clients with open files will now receive a read-only error message if they attempt to save data once redirection has occurred but before the target is writeable.  This is expected and gives the user feedback that writes will not be successful.   Each application has different behaviour in how it returns a read-only file system error to the user.

1.6.0 >

Parallelized Rename - Now the rename process can use up to 10 threads at once to rename 10 shares in parallel across all policies in a failover job.  This will provide a 10x speed improvement to redirect DFS clients faster under all failover conditions.  With large share or policy count failovers getting accelerated by a factor of 10



DFS Failover Enhancements


Release

Enhancement

Notes

1.9 >

For DFS Failover (Microsoft DFS Mode or DFS enabled Job in an Access Zone Failover), following Share Rename Step Enhancement has been made:

  • If share renaming is failed for all the shares for a cluster, then failover status is error and failover is stopped. This aborts the failover and leaves the data accessible on the source cluster.

  • If share renaming is failed only for some shares for a cluster, then failover status is warning and manual recovery on the shares that failed to rename is required.

Summary:  This eliminates the possibility of data access outage from share rename step and ensures if some shares rename the failover will continue.





Help

This document is designed as a guide for Eyeglass SyncIQ Failover and Failback with Microsoft DFS, should you have any issues that are not addressed in this guide, Superna offers support in several forms; on line, voicemail, Email, or live online chat.

  1. The support site provides online ticket submission and case tracking.  Support Site link - support.superna.net 

  2. Leave a voicemail at 1 (855) 336-1580

In order to provide service you must have an account in our system. When calling in leave; customer name, email, description of question or issue, and primary contact for your company. We will  assign the case to primary contact for email followup.

  1. Email eyeglasssupport@superna.net

  2. To download license keys please go to the following  license keys.

  3. You can also raise a case right from in Eyeglass desktop using the help button, search for your issue and if want to raise a case or get a question answered, click the “leave us a message”  with your name, email and appliance ID and a case is opened directly from Eyeglass.

 http://site.superna.net/_/rsrc/1472870726155/support/LeaveUsAMessage.png?height=200&width=167

  1. Or get Support Using Chat M-F 9-5 EDT  (empty box?  we are not online yet)

  2. Eyeglass Live Chat 

  3. You should also review our support agreement here.

How DFS mode with Eyeglass Works

Process Flow for DFS Failover with Eyeglass

Normal

The following diagram displays the process flow in normal condition (before Failover)

Configuration:

DFS Target folder was configured to refer to both Primary Cluster and Secondary Cluster’s SMB Shares, with the priority set to the Primary Cluster. The Active Directory Sites and Subnets were also configured to let the clients resided on the same subnet and site as the Isilon Primary Cluster IP addresses.

SMB Shares were created on Primary and Secondary Clusters.  See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares) for version behaviour.


  1. SMB Client is accessing a domain-based namespace (e.g. \\example.com\dfs01\folder1) . This SMB client computer sends a query to the AD  to discover a list of root targets for the namespace.

  2. AD Controller returns a list of root targets defined for the requested namespace.

  3. SMB client selects the root target from the referral list  and sends a query to the root server for the requested link.

  4. DFS root server constructs a list of folder targets in the referral. Order / priority of targets can be configured. Folder Target to the Primary Cluster is configured as the first path to refer to  (higher priority) in the referral list. The SMB Client and Primary Cluster are also residing in the same Active Directory Site and Subnet. The SMB Share(s) on Secondary Cluster is not active (Deleted). DFS root server sends the referral information to the client. The active path is to the Primary Cluster.

  5. SMB client tries to establish a connection to the selected target (the first priority / active target  in the list).

  6. Isilon (Primary cluster) responds to this SMB connection.

Failover with Eyeglass DFS mode

The following diagram displays the process flow during Failover event.

Flow:

  1. Primary Cluster failure or down detected,

    1. Initiate Eyeglass Failover of DFS syncIQ policy protecting the DFS UNC targets (7a)

    2. Eyeglass deletes or renames SMB share(s) on Primary Cluster (7b) See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares ) for version behaviour.

      1. Eyeglass creates quotas on Secondary cluster and deletes them from the Primary cluster

  2. Eyeglass performs SyncIQ Failover to Secondary Cluster

    1. Eyeglass syncs data changes with SyncIQ to Secondary cluster move schedule over and runs re-sync prep (8)

  3. Eyeglass creates or renames SMB share(s) on Secondary Cluster  See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares )  for version behaviour.

  4. DFS Target folder - path to the Secondary cluster will automatically be activated

  5. SMB Client is connecting to the Secondary cluster

  6. Secondary cluster is responding to SMB client


Failback with Eyeglass

The following diagram displays the process flow during Failback event.

Flow:

  1. Primary cluster is resumed and ready for failback. (13, 14, 15, 16)

    1. Perform Eyeglass Failback of DFS SyncIQ policy protecting the DFS UNC targets (15)

    2. Eyeglass syncs data changes with SyncIQ to Primary cluster (15)

    3. Eyeglass performs SyncIQ Failover to Primary Cluster

    4. Eyeglass creates or renames SMB share(s) on Primary Cluster (16) See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares )  for version behaviour.

    5. Eyeglass deletes or renames SMB share(s) on Secondary Cluster (14)  See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares )  for version behaviour.

    6. Eyeglass creates quotas on Primary cluster and deletes them from the Secondary cluster

    7. DFS Target folder - path to the primary cluster will  automatically be activated.

    8. SMB Client is connecting to the primary cluster

    9. Primary cluster is responding to SMB client

Eyeglass Microsoft DFS Mode Failover with NFS Export

Eyeglass Microsoft DFS Mode Failover can be used with SyncIQ policies that protect NFS exports by the same policy, but in this case manual steps or post-failover scripting must be used to update NFS client mounts.

Requirements for Eyeglass Microsoft DFS Mode Failover


The requirements in this section must be met in order to initiate an Eyeglass Microsoft DFS Mode Failover.  Failure to meet some of these conditions may block the Failover.


Cluster Version Requirements

Clusters participating in an Eyeglass Microsoft DFS Mode Failover must be running the supported Isilon Cluster version for this feature.  See the Feature Release Compatibility matrix in the Eyeglass Release notes specific to your Eyeglass version found here.

SyncIQ Policy Requirements - Blocks Failover

For a Microsoft DFS Mode failover with Eyeglass, it is required that the Eyeglass Configuration Replication Job for that SyncIQ Policy is in the Enabled state.  

Note: If the SyncIQ Policy is disabled in OneFS or the corresponding Eyeglass Configuration Replication Job is disabled in Eyeglass the SyncIQ Policy failover will be blocked.

Failover Target Cluster Requirements - Blocks Failover

For a SyncIQ Policy failover with Eyeglass, it is required that the Isilon Cluster that is the target of the failover be IP reachable by Eyeglass with the required ports open.

Eyeglass Quota Job Requirements

For a SyncIQ Policy failover with Eyeglass, there are no Eyeglass Quota Job state requirements.  Quotas will be failed over whether Eyeglass Quota Job is in Enabled or Disabled state.

Active Directory

  • AD clients must have both paths of the folder target cached post failover.

  • AD clients must be able to contact a Domain Controller.

  • UNC paths to mount the DFS folder must use DFS UNC syntax \\domain name\dfsrootname\dfs folder name.  

  • SmartConnect zone names in the  UNC targets must be delegated and resolvable by clients.

  • SmartConnect zone name SPN’s for folder target UNC paths must be correctly registered in AD.

Considerations for Eyeglass Microsoft DFS Mode Failover


The following are highly recommended to ensure that all automated Eyeglass Microsoft DFS Mode failover steps can be completed.

SyncIQ Policy Recommendations

  • SyncIQ Job in OneFS should have been completed without error and shows green.

    • Impact:

      • Failover will be blocked if SyncIQ policies are in an error stated on the cluster.  Eyeglass will attempt to run the policy which will fail.  Correct this on the OneFS cluster.

      • Data loss due to unreplicated data.

  • Isilon does not support SyncIQ Policy with excludes (or includes) for failover.

    • Impact: Not a supported configuration for failback.

  • Isilon best practices recommend that SyncIQ Policies utilize the Restrict Source Nodes option which requires an IP to be created with target SmartConnect zone..

    • Impact: Subnet pool used for data replication is not controlled all nodes in the cluster can replicate data from all IP pools.  This is hard to manage bandwidth and requires all nodes to have access to the WAN.


Preparing your System for the Eyeglass Microsoft DFS Mode Failover


This section contains steps to configure an Eyeglass Microsoft DFS Mode Failover / Failback solution.  Please refer to Microsoft DFS, Isilon and Eyeglass documentation for their respective detailed configuration steps.

Overview

Step 1 - Configure Isilon Smartconnect Zones

Step 2 - Configure Isilon SyncIQ Policies and SMB Shares (only source cluster needs the share created)

Step 3 - Configure DFS root (domain based)

Step 4 - Configure DFS folder

Step 5 - Configure DFS Target (Primary cluster):  \\SmartConnect Zone\share name

Step 6 - Verify Client Access example: \\domain.name\DFSrootname\

Step 7 - Eyeglass Setup and SyncIQ policy enabled for DFS Mode (enable mirror policy if it exists):

  • run DFS Mode policy and verify its green OK

Step 8 - Configure DFS Target (Secondary cluster):  \\Smartconnect Zone\share name

Step 9 - Verify Client Access:  

  • mount DFS path:  \\domain.name\DFSrootname\folder name

  • Write data successfully


Preparation Steps


Step 1 - Configure Isilon SmartConnect Zones:

DFS Folder Targets will be configured using UNC Path to Primary and Secondary SMB Share(s) by using their SmartConnect Zone names.  If the Isilon clusters do not already have SmartConnect Zone names configured they will need to be setup.

Step 2 - Configure Isilon SyncIQ Policies and Shares:

Create the SyncIQ policies and shares required to protect and access the data if not already existing.  

Design Considerations:

  1. A single SyncIQ policy can have multiple DFS targets underneath it in the file system path.

    1. This means all DFS targets using this policy failover together

  2. To achieve per application failover a single SyncIQ policy for each DFS target is required.

Step 3 - Configure DFS root

Recommend Domain based DFS for higher availability with replicated DFS root between domain controllers.

Step 4 - Create DFS folder


Step 5 - Configure Folder DFS Target (Primary cluster)

Configure the DFS Target for the Primary cluster (cluster which is currently active) by adding UNC path to share created on Primary cluster.  This UNC path must use a SmartConnect Zone on the Primary cluster.

Example:  \\production.example.com\SMBshare1

Step 6 - Verify Client Access

Check access from DFS enabled client and ensure that they have read / write access enabled to the Primary cluster as per the DFS Folder(s) and Target(s) configured.

IMPORTANT: You must use path including fully qualified active directory domain DNS name - for example:  \\ad1.test\DFSRoot\ShareFolder


Tools help debug and plan DFS for DR before going into production.  Test machines should install these tools which can be found here: "Where to find DFSUTIL.EXE for Windows Server "

Verify that the SMB client is accessing the DFS target folder’s SMB Share from the primary cluster. DFS utility command to verify: (note DFS client utilities must be installed into the OS, they are not standard tools)

  • dfsutil cache referral (ensure both targets are listed)

  • dfsutil cache referral flush (clear the cache for debugging)

  • dfsutil  diag viewdfspath \\domainname\dfsroot\dfsfolder

Example:

C:\Windows\system32>dfsutil cache referral

4 entries...

Entry: \DFS-SVR-01\dfs02\folder2

ShortEntry: \DFS-SVR-01\dfs02\folder2

Expires in 270 seconds

UseCount: 1 Type:0x1 ( DFS )

  0:[\cluster11-z01.ad1.test\SMB-Share-002] AccessStatus: 0 ( ACTIVE TARGETSET

)

  1:[\cluster12-z01.ad1.test\SMB-Share-002] AccessStatus: 0xc00000cc ( TARGETSE

T )


Entry: \ad1.test\dfs02

ShortEntry: \ad1.test\dfs02

Expires in 270 seconds

UseCount: 0 Type:0x81 ( REFERRAL_SVC DFS )

  0:[\DFS-SVR-01\dfs02] AccessStatus: 0 ( ACTIVE TARGETSET )


Entry: \AD1.test\sysvol

ShortEntry: \AD1.test\sysvol

Expires in 0 seconds

UseCount: 0 Type:0x1 ( DFS )

  0:[\ad1.AD1.test\sysvol] AccessStatus: 0 ( ACTIVE TARGETSET )


Entry: \DFS-SVR-01\dfs02

ShortEntry: \DFS-SVR-01\dfs02

Expires in 270 seconds

UseCount: 0 Type:0x81 ( REFERRAL_SVC DFS )

  0:[\DFS-SVR-01\dfs02] AccessStatus: 0 ( ACTIVE TARGETSET )



DfsUtil command completed successfully.


C:\Windows\system32>dfsutil diag viewdfspath \\ad1.test\dfs02\folder2


The DFS Path <\\ad1.test\dfs02\folder2> resolves to -> <\\production.example.com\SMBshare1>


Done processing this command.


Step 7 - Eyeglass setup

  1. Install Eyeglass and add clusters as managed devices to Eyeglass

  2. Allow Eyeglass to complete initial discovery and creation of Eyeglass configuration replication Jobs.

  3. Enable Eyeglass DFS mode on the Eyeglass Configuration Replication Jobs for the SyncIQ policies that will be using the Eyeglass DFS Failover solution.

NOTE: If a SyncIQ policy has already been failed over in OneFS and a mirror policy exists, enable Eyeglass DFS Mode for both the active and disabled policy.



Screen Shot 2015-08-13 at 8.22.20 PM.png

    1. DFS Enabled SyncIQ policies will show in a new folder and be of Type AUTODFS.

Screen Shot 2015-08-14 at 7.35.33 AM.png

    1. Initially the state of the Eyeglass AUTODFS job will be pending.  Once the next Eyeglass replication cycle has been executed and the new DFS job has been run the state will be updated.  The state will be OK if no errors were encountered.

NOTE 1: (See DFS Feature changes table for version behaviour.)  With Eyeglass DFS mode, the SMB shares only exist on the active cluster.  This is expected and will not interfere with the failover.  If Eyeglass detects that the share does exist on the standby cluster, Eyeglass DFS mode Jobs (AUTODFS) will delete it on the cluster that is currently the SyncIQ target (read-only) cluster.

NOTE 2:  If the SyncIQ policy associated with an Eyeglass DFS mode (AUTODFS) job is renamed, Eyeglass considers this to a be a new, previously unknown Job and it will be created as a regular Eyeglass Configuration Replication (AUTO) job.  You must re-enable DFS for this job.

NOTE 3: See DFS Feature changes table for version behaviour.   


Step 8 - Configure DFS Target (Secondary cluster)

Configure the DFS Target for the Secondary cluster (cluster which is NOT currently active) by adding UNC path to share created on Secondary cluster.  This UNC path must use a Smartconnect Zone on the Secondary cluster.

Example:  \\dr.example.com\SMBshare1

!! IMPORTANT

You must use exactly the same share name as was used for Primary cluster DFS target

Confirm that both DFS Targets for Primary and Secondary clusters are enabled.

Design Considerations:

Configure Folder Targets with UNC Path to Primary and Secondary SMB Share by using their SmartConnect Zone names.  NOTE:  With Eyeglass DFS mode, the shares only exist on 1 cluster at a time.  You must create both target UNC’s in DFS folder targets.  The adding of the target cluster will not be able to detect the site information since the share does not exist.  This is ok and will not interfere with the failover.  (See DFS Feature changes table for read-only share version behaviour.)  

Set the Target Folder Priorities

Priority for Primary Cluster: First among all targets

Priority for Secondary Cluster: Last among all targets


Active Directory Sites and Subnets for Isilon clusters and SMB Clients (Optional)

NOTE: It is not required for successful Eyeglass DFS mode failover.

Best practice:  

This step is optional and helps optimize the client decision on which cluster to mount using IP subnets.  This requires subnets are configured for all UNC targets and clients in Active Directory.  For large networks this may not be practical.   If not configured clients will pick a target based on the referral list without using sites.  

Use two separate subnets and sites for the Primary cluster (PrimarySite) and Secondary cluster (SecondarySite).  Example:

  1. PrimarySite:Subnet 172.16.80.128/27

  2. SecondarySite:Subnet 172.16.81.128/27


Place the SMB client’s IP address in the Primary Site subnet’s IP address range. With this setting, the Primary Cluster is referred to as the primary folder target path (higher priority).

Step 9 - Verify Client Access

Check access from DFS enabled client and ensure that they still have read / write access enabled to the Primary cluster ONLY as per the DFS Folder(s) and Target(s) configured.

IMPORTANT: You must use path including fully qualified active directory domain DNS name - for example:  \\ad1.test\DFSRoot\ShareFolder



Tools help debug and plan DFS for DR before going into production.  Test machines should install these tools.

DFS util  downloaded by OS type

Now we have DFS folder target paths configured to have access to Primary and Secondary Cluster’s SMB Share. But only the SMB Share on Primary Cluster is active (in normal condition / before failover).  The SMB share on Secondary Cluster was created for DFS Folder Target creation as the second path. After it had been registered to DFS Folder setting, then it was deleted. With this initial state, you can verify that the SMB client is still accessing the DFS target folder’s SMB Share from the primary cluster even though a DFS target for the Secondary cluster exists. DFS utility command to verify: (note DFS client utilities must be installed into the OS, they are not standard tools)

  • dfsutil cache referral

  • dfsutil  diag viewdfspath

Example:

C:\Windows\system32>dfsutil cache referral

4 entries...

Entry: \DFS-SVR-01\dfs02\folder2

ShortEntry: \DFS-SVR-01\dfs02\folder2

Expires in 270 seconds

UseCount: 1 Type:0x1 ( DFS )

  0:[\cluster11-z01.ad1.test\SMB-Share-002] AccessStatus: 0 ( ACTIVE TARGETSET

)

  1:[\cluster12-z01.ad1.test\SMB-Share-002] AccessStatus: 0xc00000cc ( TARGETSE

T )


Entry: \ad1.test\dfs02

ShortEntry: \ad1.test\dfs02

Expires in 270 seconds

UseCount: 0 Type:0x81 ( REFERRAL_SVC DFS )

  0:[\DFS-SVR-01\dfs02] AccessStatus: 0 ( ACTIVE TARGETSET )


Entry: \AD1.test\sysvol

ShortEntry: \AD1.test\sysvol

Expires in 0 seconds

UseCount: 0 Type:0x1 ( DFS )

  0:[\ad1.AD1.test\sysvol] AccessStatus: 0 ( ACTIVE TARGETSET )


Entry: \DFS-SVR-01\dfs02

ShortEntry: \DFS-SVR-01\dfs02

Expires in 270 seconds

UseCount: 0 Type:0x81 ( REFERRAL_SVC DFS )

  0:[\DFS-SVR-01\dfs02] AccessStatus: 0 ( ACTIVE TARGETSET )



DfsUtil command completed successfully.


C:\Windows\system32>dfsutil diag viewdfspath \\ad1.test\dfs02\folder2


The DFS Path <\\ad1.test\dfs02\folder2> resolves to -> <\\production.example.com\SMBshare1>


Done processing this command.


Configuration complete

Failover Planning and Checklist


Failover planning includes extended preparation beyond storage layer failover steps as related to the clients, application owners and any dependent systems such as DNS and Active Directory.  A full Failover Plan is required taking this all into account.  A Failover Planning Guide and checklist document is provided for input into your own Failover plan.

Monitoring DR Readiness for Eyeglass Assisted Failover


In addition to the Assisted Failover functionality, Eyeglass also provides the following features to monitor your Microsoft DFS Mode Failover Readiness:

  • DR Readiness Validation

Eyeglass DR Readiness for DFS Enabled SyncIQ Policies

Eyeglass DFS Readiness DR Status provides a quick and easy way to assess the status on DFS mode enabled SyncIQ policies readiness for failover or failback operations.  DFS Readiness DR Status is “OK” when all of the conditions below are met:

  • Your SyncIQ Policy is enabled.

  • Your SyncIQ Policy Last Started and Last Success timestamp are identical.

  • Your Eyeglass configuration replication Job is enabled.

  • Your Eyeglass configuration replication Job Last Run and Last Success timestamp are identical.

    • This validates that shares only exist on the writeable copy of the SyncIQ data.  This is the normal operating mode for Eyeglass DFS enabled policies.  (See DFS Feature changes table for read-only share version behaviour.)

  • Your Eyeglass configuration replication Job Audit Status is OK.

To check the DFS Readiness DR Status for your SyncIQ Policies in DFS mode:

  1. Login to Eyeglass.

  2. Open the DR Dashboard.

  3. Select the DFS Readiness tab.

The DR Status is displayed per policy.  It also indicates which pair of clusters are used in the DFS configuration and the associated SyncIQ policy.  

Expand a policy to see the details for the SyncIQ Policy and the Eyeglass Configuration Replication status:

  • Last Run time of the SyncIQ policy and the status of the last run.

  • Last Run time of the Eyeglass Configuration Replication job and the status of the last run and audit.

If new shares are created on the DFS mode policy, the next run of the Eyeglass Configuration Replication job in DFS mode will be aware of the new shares and ready to fail them over.  It is important to check this job's status after creating more DFS shares under a policy.


Operational Steps for Eyeglass Microsoft DFS Mode Failover

Overview

The Eyeglass Microsoft DFS Mode Failover solution for Isilon OneFS fully automates execution of these steps:

  1. Performs SyncIQ Failover to Secondary Cluster and sync’s outstanding data.

  2. Creates and Renames ALL SMB Shares protected by the SyncIQ policy to the  Secondary Cluster.

  3. Deletes or Renames ALL SMB shares on the source cluster to ensure DFS clients can not mount the cluster See DFS Feature changes table (DFS Feature Changes and Share names used on DFS synced Shares ) for version behaviour.

  4. Creates ALL quotas on the target cluster protected by the SyncIQ policy.

  5. Deletes ALL quotas on the source cluster to ensure failback and SyncIQ operations from Secondary cluster to primary.

  6. Copies SyncIQ schedule to Secondary cluster.

  7. Runs Sync Prep on Source cluster to become the target of SyncIQ policy replication.

Procedure for Running Eyeglass Microsoft DFS Mode Failover

IMPORTANT: Making any changes to the SyncIQ Policies or related Eyeglass Configuration Replication Jobs being failed over during the failover may result in unexpected results.

IMPORTANT: Eyeglass Assisted Failover has a 1 hour timeout on each failover step.  Any step which is not completed within this timeout period will cause the failover to fail.

IMPORTANT: Deleting configuration data (shares, exports, quotas) or modifying Share name or NFS Alias name or NFS Export path on the target cluster before failing over without running Eyeglass Configuration Replication will incorrectly result in the object being deleted on the source cluster after failover.  You must run Eyeglass configuration replication before the failover OR select the Config Sync checkbox on failover to prevent this from happening.

For detailed steps consult the failover guide table here.

For detailed steps on execution and monitoring consult the Failover Design Guide

NOTE: On completion of the failover the DFS clients will no longer have a share on the primary cluster and will use the referral list to select a new mount which will be an active share on the target cluster.

Post Eyeglass Microsoft DFS Mode Failover Manual Steps for NFS Exports

For the case where the SyncIQ Policies that were failed over are also protecting NFS exports, manual steps, or post failover scripting is required to update NFS client mounts.

Please refer to the Eyeglass Admin Guide Script Engine Overview.

Post Eyeglass Microsoft DFS Mode Failover Checklist

After a DFS failover we recommend the following steps to confirm everything was successful.

IMPORTANT: If the failover was done with the Controlled failover option unchecked (a real DR event failover), this means the share rename step on source was not executed. The source cluster should NOT be allowed to be reachable on the network until the shares are renamed using onefs UI OR DFS referrals are edited to disable the folder target pointing to the source cluster. Consult Failover Recovery Guide.


  1. Open DR Dashboard and select DFS Readiness tab

  2. Make sure the two jobs are shown 1) one for the Mirror policy which should be active and 2) for the original policy which should be disabled now.

Screen Shot 2015-08-14 at 8.34.06 AM.png

  1. Login to the source cluster (the one that you failed over from) and verify that all shares that are part of the failover are renamed with DFS prefix on the cluster. (See DFS Feature changes table for read-only share version behaviour) .

    1. You can get a list of shares that should have been failed over using this procedure

    2. Open the jobs window

    3. Select the DFS Mode SyncIQ policy job and select the check box then bulk actions and edit configuration option

Screen Shot 2015-08-14 at 8.37.05 AM.png

    1. Expand the cluster configuration to look in the SMB folder.  All Shares that were part of the failover will have a blue check mark next to them.

Screen Shot 2015-08-14 at 8.37.57 AM.png

    1. For Eyeglass Release 1.5 and higher, check the Failover Log.  Look for the lines:

      1. INFO Renamed <number> shares on source cluster.

      2. INFO Renamed <number> shares on target cluster.

      3. ERROR Policy: <policy name> Step: "Renaming shares on source and target clusters" Result: FAILURE: Failed to rename SMB shares for policy <policy name>. Failover not complete.

      4. ERROR Cluster: <cluster name> Step: "Renaming shares on source and target clusters" Result: FAILURE: Renaming shares failed.

These numbers should be the same and should match the total number of shares protected by the SyncIQ Policy that was failed over, and there should be no ERROR related to share renaming.   If this is not the case:

  • Search on the failover source cluster for any shares protected by the SyncIQ Policy failed over that do not have “igls-dfs” prefix. Manually update the share name to add the” igls-dfs” prefix

  • Search on the failover target cluster for any shares protected by the SyncIQ Policy failed over that still have “igls-dfs” prefix.  Manually update the share name to remove the “igls-dfs” prefix


Procedure for Checking your SMB Clients Post DFS Failover:

Use this procedure to check that clients have picked up the change from one cluster to another.

  1. Note: This steps requires dfsutil installed.  "Where to find DFSUTIL.EXE for Windows Server"  

  2. On a DFS enabled client (all OS’s after Windows 7)

  3. Open a command prompt

  4. Type: dfsutil diag viewdfspath \\<dfs path>

example:  “dfsutil diag viewdfspath  \\ad1.test\FOdemo\Corpdata”

  1. The output below was run twice and you can see the UNC target changed from DR cluster to production cluster after the failover.   The output tells you if that client detected the share deleted and remounted the alternate UNC path which is now a writeable copy of the data.

Screen Shot 2015-08-14 at 9.04.29 AM.png

  1. The next test is mount the DFS path and write some data to verify writes are succeeding to the resolved UNC path.    This can also be verified with SyncIQ OneFS UI.

Advanced DFS mode Setup with Access Zones

One using DFS mode policies inside an Access zone that you intend to use Access Zone failover for other policies in the Access Zone.  The following additional configuration must be done to ensure the SmartConnect Zones used for DFS folder UNC paths do not get failed over or require Eyeglass hints.

Configuration Steps

  1. Create dedicated IP pool for DFS protected data and make the IP pool a member of the Access Zone

  2. Create shares for DFS UNC paths

  3. Create SyncIQ policy

  4. Discover the policy with Eyeglass and edit the job in the jobs definition window to Enable DFS mode.

    1. Run the DFS mode job in Eyeglass to make sure it completes

  5. Create Eyeglass hints to ignore the DFS IP pool using a hint as a IP pool alias named “igls-ignore’.

  6. Create the hint on both IP pools: source and target cluster

  7. It should look like below after Zone Readiness job runs (it can be run with Run now option to update the DR dashboard

Screen Shot 2015-12-12 at 7.45.04 PM.png