Two node upgrade process when SSO and CID are running on the same machine

The idea is to do node by node upgrade for SSO and CID and have minimal service break during the upgrade.

For this kind of upgrade, you will need to route production traffic only to SSO node 1 and CID node 1, when you are upgrading SSO Node 2 and CID Node 2. 
Once SSO Node 2 and CID Node 2 are upgraded then route production traffic to upgraded nodes. After this start upgrading SSO Node1 and CID Node1 . 

High-level plan is : 

• Route production traffic only to SSO node 1 and CID node 1 (Active line of servers) 
• Upgrade SSO Node 2  – Make this New active node 
• Upgrade CID Node 2 (slave) – Make this new Master 
• Data migration from SSO Node 1 to SSO Node 2
• Route production traffic only to SSO node 2 and CID node 2 upgraded in the above steps (These will act as Active Nodes from here onwards) 
• Upgrade old SSO Node 1 
• Upgrade old CID Node 1 (old master) - Make this new slave
• Restore proxy setup to configure HA setup

Let's use letters A for Active and P for Passive to avoid confusion in our current setup. 

1) Traffic Switchover: Make sure all traffic is going only to SSO Node A And CID Node A (old master)    

2) SSO Node P upgrade : 

Kindly follow all the steps mentioned below :   
You can refer to this page for details information but you need to consider SSO node1 in this document as SSO P for you. So perform all the steps for SSO Node 1 mentioned in this document (until step 6), on your SSO Node P.  - Clustered upgrade on Windows - SSO  

1. Find out which SSO node is the schema master
   1. Use the Active Directory Schema snap-in to connect an AD LDS instance
   2. Identify the schema master , make the configuration on Node P as schema master
2. Configure the reverse proxy server to route the production traffic only to the SSO A node
   1. Setup both SSO nodes for the activity test
   2. Optional: Test through the reverse proxy server which SSO node is active
   3. Disable traffic from the reverse proxy server to the SSO P node
   4. Test through the reverse proxy server which SSO node is active
3. Disable the replication
   1. Stop the UbiloginServer process from the SSO P node
   2. Disable outbound and inbound replication from the SSO A node
   3. Disable outbound and inbound replication from the SSO P node
4. Configure the metadata clean up for the retired AD LDS instances in the SSO P node
   1. Clean up the metadata from the SSO P node using the dsmgmt tool
   2. Remove the SSO A node server object from the SSO P node using the ADSI Edit tool 
5. Update the SSO P node
   1. Use document: Upgrade on Windows - SSO

Once SSO P is upgraded, move to CID P (slave) upgrade. 

3) CID Node P  upgrade : 

Follow the steps in order. Issue all commands in the Windows command prompt using the Administrator user account.

a. Stop ubiloginserver service in SSO Node P and stop wildfly sevice in CID Node P. 

b.  It is a good idea to take periodic backups of running installations. 

1. Back up Ubisecure CustomerID. See instructions from Backup And Restore - CustomerID. 

     You should do this regularly but for the upgrade process we advise you in this step to move the existing installation to a specific location customerid-old which is used in this document. 

   cd "%PROGRAMFILES%\Ubisecure"
   move customerid customerid-old

c. Perform on Ubisecure CustomerID  P Node (slave) :

1. Unpack the distribution package. See instructions from Distribution package unpacking on Windows - CustomerID. 
2. Your installation folder e.g. %PROGRAMFILES%\Ubisecure\customerid must be empty before doing this step, Deployment template extraction on Windows - CustomerID
3. Check Java. See instructions from Java check on Windows - CustomerID.
4. Install WildFly. See instructions from Two node WildFly installation on Windows - CustomerID. (Ensure to remove the old Wildfly service before installing the new one)
5. Import configuration settings from the previous installation.
   In order to resume the service after the upgrade it is necessary to copy some configuration settings from the previous installation. These settings are contained in the file %PROGRAMFILES%\Ubisecure\customerid-old\application\win32.config.

   In the new installation the win32.config file is located further down the similar path in the config sub-folder and must first be copied to the application folder.

   cd /D "%PROGRAMFILES%\Ubisecure\customerid\application"
   copy config\win32.config 

   The values must be copied carefully from the previous installation as the configuration options may have changed. You should copy only those values that have the same keys on both old and new win32.config files. Refer to page Setup template on Windows - CustomerID for more information about the current configuration options.

   NOTE: If you have made configuration changes to any of the win32.config parameters directly in eidm2.properties file after previously running the setup.cmd ensure that these changes are included in the new win32.config file. For example, if you have changed the REST credentials in eidm2.properties file, make sure the same values are now present in the win32.config file.

   5. Run the setup script.

   cd /D "%PROGRAMFILES%\Ubisecure\customerid\application"
   setup.cmd

6. Prepare WildFly for domain configuration. See instructions from Two node WildFly preparation on Windows - CustomerID. (In case of following error: "Failed to connect to the controller. The controller is not available at localhost:9990: java.net.ConnectException: WFLYPRT0053". Ensure wildfly is started on Node P )
7. Configure HTTPS, See Two node SSL configuration on Windows - CustomerID -  This generates certificate files and Keystore (default location: %WILDFLY_HOME%\domain\configuration\keystore.pfx) is registered to WildFly in the next step  - master node WildFly configuration. 
8. Configure WildFly on CustomerID Master Node. See instructions from  Two node master WildFly configuration on Windows - CustomerID
9. Configure the singleton subsystem. See instructions from  Two node singleton subsystem configuration on Windows - CustomerID
10. Perform database schema update depending on versions.

    • See Database Changes.

11. When upgrading from CustomerID version 5.6.x or older update CustomerID LDAP entries to facilitate REST API OAuth2 authentication. See LDAP import instructions from SSO management web applications creation on Windows - CustomerID.
    

    • See also Configuring OAuth2 authentication for REST API.

     10. Create JDBC data source to WildFly. See instructions from Two node JDBC data source creation on Windows - CustomerID

     11. Create a Mail Session configuration for WildFly. See instructions from  Two node mail session creation on Windows - CustomerID

     12. Configure logging for CustomerID. See instructions from  Two node logging configuration on Windows - CustomerID

     13. Register "customerid.home" system property to WildFly. See instructions from WildFly system property registration on Windows - CustomerID

    d. Perform on Ubisecure SSO Node P :

1.  Upgrade PostgreSQL JDBC driver to SSO node(s). See instructions from PostgreSQL JDBC driver installation to SSO on Windows - CustomerID.
2.  Upgrade Ubisecure CustomerID SSO Adapter to SSO node(s). See instructions from CustomerID SSO Adapter extension upgrade

    e. Perform on the CustomerID new Master Node (Node P) :

1. Deploy the Worker Enterprise Archive, CustomerID Enterprise Archive to the WildFly domain. See instructions from Two node deployment to WildFly on Windows - CustomerID.
2. Restart Ubisecure CustomerID. See instructions from  Restart on Windows - CustomerID.

   f. Perform on Ubisecure SSO Node P:

        1.Restart Ubisecure SSO. See instructions from  Installation related SSO restart on Windows - CustomerID.

Data Migration :  

// We are working on exact instructions to move data in ADLDS from one node to another, below are high level actions need to be taken in this part

• You can take service break at this point to take data backup from Node A and restore in Node P before taking it into use. 
• Stop UbiloginServer and wildfly service in SSO Node A and CID Node A. 
• Perform Data migration from Node A to Node P 
• Verify the application to confirm you can access SSO Management and CustomerID (Basic flows like login works and no problems loading data on screen).  
• Now SSO Node P and CID Node P are upgraded and have upto date data and will be ready to take traffic. 

Traffic Switchover : After this step switch traffic to upgraded nodes SSO Node P  and CID Node P (New Master Node). 

1)  SSO Node A Upgrade : 

Kindly follow all the steps mentioned below :   

Detailed steps are here in this document. You need to consider SSO node 2 in this document as SSO A for you. So perform all the steps for SSO Node 2 mentioned in this document on your Node A:  Clustered upgrade on Windows - SSO   (From Step 7 )

1. Remove the services and the AD LDS from the SSO A node
   1. Check Java 8 environment variables
   2. Stop SSO and Accounting Service Windows services
   3. Remove SSO and Accounting Service Windows service configurations
   4. Remove the AD LDS from the SSO A node
2. Enable the replication in the SSO B node
3. Update the SSO A node
   1. Install Java 11 to the SSO A node
   2. Execute the steps in the document  AD LDS clustering setup (node 2) - SSO
   3. Install the new SSO version to the SSO A node
   4. Update Tomcat configuration by reinstalling it to the SSO A node
   5. Check Single node installation finalization
   6. Test the functionality of the updated SSO A node
4. Enable traffic from the Reverse Proxy to route production traffic to SSO A node in addition to the SSO P node

Once SSO node A is up and running with upgraded software, move to CID Node A upgrade. 

2) CID Node A  Upgrade :
Perform the next steps to upgrade CID node A which is the old master node and after upgrade, it will be a new slave node. 

a. Perform on Ubisecure SSO Node A:

1. Unpack the distribution package. See instructions from Distribution package unpacking on Windows - CustomerID.
2. Upgrade PostgreSQL JDBC driver to SSO node(s). See instructions from PostgreSQL JDBC driver installation to SSO on Windows - CustomerID.
3. Upgrade Ubisecure CustomerID SSO Adapter to SSO node(s). See instructions from CustomerID SSO Adapter extension upgrade.

Note : After this step you can start Ubiloginserver service in SSO Node A to restore SSO HA setup.

b. Perform on CustomerID Node A :

1. Check Java. See instructions from  Java check on Windows - CustomerID.
2. Install WildFly. See instructions from Two node WildFly installation on Windows - CustomerID. (Ensure to remove the old Wildfly service before installing the new one)
3. Transfer the installation folder from master node. See instructions from Two node installation folder transfer on Windows - CustomerID.
4. Prepare WildFly for domain configuration. See instructions from Two node WildFly preparation on Windows - CustomerID.
5. Configure HTTPS, See Two node SSL configuration on Windows - CustomerID -  This generates certificate files and Keystore (default location: %WILDFLY_HOME%\domain\configuration\keystore.pfx) is registered to WildFly in the next step  - slave node WildFly configuration. Note- If cert keys are not changing, generation of new cert on slave node can be skipped and keystore.pfx can be copied from master node to slave node.
6. Configure WildFly on CustomerID Slave Node. See instructions from Two node slave WildFly configuration on Windows - CustomerID

      6. Restart Ubisecure CustomerID. See instructions from Restart on Windows - CustomerID

c. Perform on Ubisecure SSO Node A:

1. Restart Ubisecure SSO. See instructions from Installation related SSO restart on Windows - CustomerID.

d. Finalize Upgrade : 

Kindly verify customerid_diag.log file to see if both CID nodes are running and logs are getting printed for both CID nodes in logs.