Two node in-place upgrade on Linux - SSO Draft

The idea is to keep one node of SSO running and perform the upgrade on the passive node first.

While performing an upgrade, treat the passive node as an active node and complete the upgrade process for this node.

Then switch over traffic to this upgraded node and start performing upgrade on the remaining (old active node) node to make it passive.

Follow the steps in order. Issue all commands using the root user account.

Upgrade SSO Node 2 (Passive Node)

Please note, we will be upgrading NODE2 to make it a new NODE1.

Preliminary Tasks

For the version to be removed, make sure you still have Java 8 installed and JRE_HOME and JAVA_HOME set

Stop the daemons that are running, ubisecure-accounting is a new service since 8.4:

/etc/init.d/ubilogin-server stop
/etc/init.d/ubilogin-directory stop
/etc/init.d/ubisecure-accounting stop

Remove SSO and Accounting Service daemon configurations

cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./config/tomcat/remove.sh

Take a backup from Ubisecure Directory of the old SSO

/usr/local/ubisecure/ubilogin-sso/openldap/libexec/slapd -T cat -f "/usr/local/ubisecure/ubilogin-sso/openldap/etc/openldap/slapd.conf" -l /home/ubilogin/database.ldif

Backup the existing Ubisecure SSO installation and OpenLDAP:
```
cd /usr/local/ubisecure
mv ubilogin-sso ubilogin-sso-old
```

Upgrade Process :

Replace Java installation with Java 11
1. Remove Java 8 installation and Delete JRE_HOME environment variable , as we do not need that with Java 11 installation.
2. Install Java 11 and set JAVA_HOME according to Installation requirements - SSO
Extract the archive sso-x.x.x-unix.tar.gz to directory /usr/local/ubisecure use the full path to the archive you have downloaded
```
tar -xzvf sso-x.x.x-unix.tar.gz
```

Copy unix.config file from the older version

cp /usr/local/ubisecure/ubilogin-sso-old/ubilogin/unix.config /usr/local/ubisecure/ubilogin-sso/ubilogin/unix.config

Add the following lines if they do not exist in the file /usr/local/ubisecure/ubilogin-sso/ubilogin/unix.config, see further instructions for openldap specific values from here The Macro language - SSO
```
tomcat.instancename = ubilogin-server
openldap.instancename = ubilogin-directory
openldap.root= uid=System,ou=System,@suffix@
openldap.maxsize = 10737418240
openldap.idlexp = 20
```

When upgrading from version 8.3.x or older, add the Accounting Service related settings if they do not exist in the file /usr/local/ubisecure/ubilogin-sso/ubilogin/unix.config. Modify the settings according to these guidelines.

# Accounting configuration
accounting.url = https://localhost:8442
accounting.proxy.local.url = @accounting.url@
accounting.instancename = ubisecure-accounting
accounting.datasource.url = jdbc:postgresql://localhost:5432/accountingdb
accounting.datasource.username =
accounting.datasource.password =
accounting.secret-key-location-uri = file:///${user.dir}/config/accounting-service.secret
accounting.actuator.username = accounting_admin
accounting.actuator.password =
accounting.jms.broker.port = 36161
accounting.jms.broker.socket-timeout-ms = 10

When upgrading from version 8.4 or newer, depending of the location of your Accounting Service secret key you may need to copy the file from the older version. NOTE: The secret key must be the same during the entire reporting period which is a month, see Accounting Service security. Example (use the path you have set in the configuration):
```
cd /usr/local/ubisecure/ubilogin-sso-old
cp --parents accounting/config/accounting-service.secret ../ubilogin-sso
```
Copy the following files and directories (recursively) from the previous installation to the matching ubilogin-sso directory. Note that Tomcat, Ubisecure SSO, and Accounting Service logs are retained. Let overwrite existing files or add flags -fn for the cp commands.
When upgrading from versions 8.4 or newer, please notice that the Accounting Service custom configuration file:
```
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/custom/accounting/config/application.yaml
```
which will copied with the copy statements below is not compatible with the newer version located at
```
/usr/local/ubisecure/ubilogin-sso/ubilogin/config/accounting/config/application.yaml
```
You need to check the following settings and manually convert them the corresponding new ones
Since version 8.7: server.use-forward-headers → server.forward-headers-strategy
logging.file → logging.file.name
Since version 9.0: logging.file.max-history → logging.logback.rollingpolicy.max-history
See also Accounting Service additional configuration.
```
cd /usr/local/ubisecure/ubilogin-sso-old
cp -r --parents ubilogin/custom/* ../ubilogin-sso
cp --parents ubilogin/config.index ../ubilogin-sso
cp -r --parents ubilogin/methods/* ../ubilogin-sso
cp -r --parents ubilogin/logs/* ../ubilogin-sso
cp -r --parents accounting/logs/* ../ubilogin-sso
cp -r --parents tomcat/logs/* ../ubilogin-sso
cp --parents ubilogin/webapps/cdc/WEB-INF/config.properties ../ubilogin-sso
cp --parents ubilogin/webapps/ROOT/robots.txt ../ubilogin-sso
```
Check Password application
NOTE:
Password: Check from the current installation if Password application is enabled. To check, examine the file
```
/usr/local/ubisecure/ubilogin-sso-old/tomcat/conf/server.xml
```
If the path /password is not commented out, Password application has been enabled in the previous installation.
Skip this step if the Password application is not enabled.
Copy the following files and directories from the previous installation to the matching ubilogin-sso directory:
```
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/webapps/password/WEB-INF/password.properties
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/webapps/password/WEB-INF/saml2
```
Edit /usr/local/ubisecure/ubilogin-sso/ubilogin/config/tomcat/conf/server.xml and uncomment following line:
<Context path="/password" docBase="${catalina.base}/webapps/password"/>
Also check /usr/local/ubisecure/ubilogin-sso-old/ubilogin/webapps/password/WEB-INF/web.xml for mail.smtp.host and mail.smtp.from configuration and copy those to new web.xml (/usr/local/ubisecure/ubilogin-sso/ubilogin/webapps/password/WEB-INF/web.xml)
Check Common Domain Cookie Discovery
NOTE:
Common Domain Cookie Discovery
Check from the previous installation if Common Domain Cookie Discovery has been enabled.
To check, examine the file
```
/usr/local/ubisecure/ubilogin-sso-old/tomcat/conf/server.xml
```
If the path /cdc is not commented out, Common Domain Cookie Discovery has been enabled in the previous installation.
If Common Domain Cookie Discovery has been enabled prior to the upgrade, re-enable the settings after upgrade according to the Common Domain Cookie Discovery document.
Run the setup script:
Tip
Before running the setup script check if you want to preserve some of the settings that may otherwise be regenerated, see: Preserve essential configuration settings in upgrade.
```
cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./setup.sh
```
After the setup script, you may still need to check some files from the backup folder if you have customised them. Compare the files under /usr/local/ubisecure/ubilogin-sso-old with the ones under /usr/local/ubisecure/ubilogin-sso and copy the necessary changes from:
```
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/webapps/uas/WEB-INF/uas.properties
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/webapps/totp/WEB-INF/application.yaml
```
When upgrading from version 8.3.x or older, install and prepare PostgreSQL server. Since SSO version 8.4 with Accounting Service feature access to PostgreSQL database is required for the service to run. If you have already installed Ubisecure CustomerID you can use the existing PostgreSQL installation but you need to create a specific database for this purpose. The necessary tables are automatically created during the initial startup of the Accounting Service.
See PostgreSQL preparation on Linux for more information and steps to accomplish.
When upgrading from version 8.6 or older, upgrade PostgreSQL server (for supported versions, see System Recommendations) following PostgreSQL official upgrade documentation at https://www.postgresql.org/docs/12/upgrading.html .
We have created Knowledge Base "How-to" article with information how we have tested the upgrade and also include estimated migration times. See Upgrade and migrate to new version of PostgreSQL
For clustered environment check that you have configured OpenLDAP replication in the following files as currently advised: /usr/local/ubisecure/ubilogin-sso-old/ubilogin/ldap/openldap/ldap_server_list.conf and /usr/local/ubisecure/ubilogin-sso-old/ubilogin/ldap/openldap/ldap_peer.conf, see OpenLDAP clustering: Install node 1. If not add the settings into these files before continuing with the OpenLDAP installation. (Please remember that now this node old NODE2 will be new NODE1 , add configuration details keeping this in mind)
You can copy these files from old installation directory and make needed changes as now NODE2 from old configuration will be NODE1 in new upgraded installation.
```
/usr/local/ubisecure/ubilogin-sso-old/ubilogin/ldap/openldap/ldap*.conf
```
If you have a clustered environment repeat the step advised in OpenLDAP clustering: Install node 1 and modify /usr/local/ubisecure/ubilogin-sso/ubilogin/config/settings.sh. Replace <node1-hostname> with your hostname.
```
ADD the following new line below the line reading "esac"
LDAP_LISTEN_URLS="ldap://<node1-hostname>:389 $LDAP_LISTEN_URLS"
```

Remove old OpenLDAP installation and Restore the Ubisecure Directory from the backup

./ldap/openldap/remove.sh
./ldap/openldap/install.sh --no-initdata
su ubilogin -c "/usr/local/ubisecure/ubilogin-sso/openldap/libexec/slapd -T add -f "/usr/local/ubisecure/ubilogin-sso/openldap/etc/openldap/slapd.conf" -l /home/ubilogin/database.ldif"

Start the ubilogin-directory daemon:
```
systemctl start ubilogin-directory
```
Important: Add new entries and update LDAP secrets into OpenLDAP, ignore warnings about e.g. existing entries
```
./ldap/openldap/import-changes.sh
```
When upgrading from version 8.3.x or older, configure Accounting Service
Before continuing with the installation which will start the Accounting Service you need to enter and save the secret key contents in the location referred by accounting.secret-key-location in unix.config. See Accounting Service security about the usage of the key for pseudonymisation. The page contains a suggested script to create a secure enough secret in the default location.
You may also customise other Accounting Service configuration settings for your needs, which is recommended. See Accounting Service additional configuration about the properties to set.
When customising edit this file which is copied from the installation package by the setup script: /usr/local/ubisecure/ubilogin-sso/ubilogin/custom/accounting/config/application.yaml
Reinstall SSO Tomcat and Accounting Service configuration and start the services. Since version 8.4 remove should be done before installation directory is replaced (see step 3.). About Accounting Service (ubisecure-accounting) start see also Linux single node installation.
```
cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./config/tomcat/install.sh
systemctl start ubisecure-accounting
systemctl start ubilogin-server
```
When upgrading from version 8.8.x or older, import initial key.
This operation needs to be done only once when upgrading from version 8.8.x or older to version 8.9.x or newer, and should not be done for any follow-up updates from 8.9.x or newer to newer versions.
Server signing and decryption key management was updated for SSO 8.9 and the initial signing and decryption key generated during SSO setup must be imported manually in the new location in Ubilogin Directory.
```
cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./ldap/openldap/import.sh ./ldap/initial-key.ldif
```
The system upgrade is complete. For the new Java installation you need to import SSO certificate to Java trust store. See also other steps in Single node installation finalization.
NOTE: If you have Ubisecure CustomerID installed, you need to copy the Authorizer files at this point. For instructions, please see Related tasks when upgrading SSO in Linux - CustomerID.
Remove the backed up ubilogin-sso-old directory, or rename and retain it as desired.
Clear your web browser’s cache before accessing the user interface.

This installation is now ready as SSO Active Node (Node1) , now we need to migrate data before we switch traffic to this upgraded SSO Node.

Data Migration : Start Maintenance/Service Window

Stop the daemons that are running in old Active Node (SSO Node1) and Newly upgraded SSO Node (New Active Node) :
```
/etc/init.d/ubilogin-server stop
/etc/init.d/ubilogin-directory stop
/etc/init.d/ubisecure-accounting stop
```

Take a backup from Ubisecure Directory of the old SSO Node1

/usr/local/ubisecure/ubilogin-sso/openldap/libexec/slapd -T cat -f "/usr/local/ubisecure/ubilogin-sso/openldap/etc/openldap/slapd.conf" -l /home/ubilogin/database.ldif

Clean up data directory from New SSO Node1 (Newly upgraded Node)

cd /usr/local/ubisecure/ubilogin-sso/openldap/var/openldapm-db/cn=Ubilogin,dc=sso,dc=example,dc=com  
(delete all files inside)
rm -rf *

Move backup taken in Step2 to New SSO Node1

Import data from backup file to New SSO Node1

/usr/local/ubisecure/ubilogin-sso/openldap/libexec/slapd -T add -f "/usr/local/ubisecure/ubilogin-sso/openldap/etc/openldap/slapd.conf" -l /home/ubilogin/database.ldif

Start the ubilogin-directory daemon:
```
systemctl start ubilogin-directory
```
Important: Add new entries and update LDAP secrets into OpenLDAP, ignore warnings about e.g. existing entries
```
./ldap/openldap/import-changes.sh
```

Now start ubilogin-server and ubisecure-accounting service.

systemctl start ubilogin-server
systemctl start ubisecure-accounting

Verify logs and login into SSO management console
If everything looks ok, - switch traffic to this upgraded node .

End Maintenance/Service Window

Upgrade SSO Old Node1 (Old Active Node)

Please note, we will be upgrading old NODE1 to make it a new NODE2 (Passive Node). We will refer this old Node1 as Node2 here onwards.

Preliminary Tasks

Now all the Service daemons are already in stopped state in SSO Node2.
For the version to be removed, make sure you still have Java 8 installed and JRE_HOME and JAVA_HOME set

Remove SSO and Accounting Service daemon configurations

cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./config/tomcat/remove.sh

Replace Java installation with Java 11
1. Remove Java 8 installation and Delete JRE_HOME environment variable , as we do not need that with Java 11 installation.
2. Install Java 11 and set JAVA_HOME according to Installation requirements - SSO

Upgrade on Node2 (Old Node 1)

Copy SSO node 1 installation(upgraded Node) to Node2

On Node2, create ubisecure folder and copy the ubilogin-sso folder from Node 1 to Node2.

Copy SSO node 1 installation

mkdir -p /usr/local/ubisecure
cd /usr/local/ubisecure/
scp -r <username>@<node1>:/usr/local/ubisecure/ubilogin-sso .

LDAP configuration

On Node2, modify Ubisecure Directory startup script (settings.sh)

LDAP settings file on node 2

vi /usr/local/ubisecure/ubilogin-sso/ubilogin/config/settings.sh

Add node 2 hostname to ldap://node2host:389 to settings.sh.

LDAP settings property on node 2

Modify the line below the line reading "esac"
LDAP_LISTEN_URLS="ldap://node2host:389 $LDAP_LISTEN_URLS"

Install OpenLDAP

Install OpenLDAP service on node 2. A system user ubilogin (default name) will be created automatically by the installation scripts. This user will run the Ubisecure daemons.

Install OpenLDAP on node 2

cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./ldap/openldap/install.sh

If the OpenLDAP install script prompts for LDAP Password, type secret and press return.

If OpenLDAP service were started stop ubilogin-directory on node 2 at this point.

Stop Directory service on node 2

systemctl stop ubilogin-directory

Delete the OpenLDAP database from node 2. It will reappear through replication later. The directory name is based on your LDAP root:

Delete replicated directory on node 2

cd /usr/local/ubisecure/ubilogin-sso/openldap/var/openldap-mdb/<your LDAP root>
rm -f *

Start OpenLDAP service on node 2 with the proper configuration.

Restart directory service on node 2

systemctl start ubilogin-directory

Verify LDAP replication

List OpenLDAP folder on node 2 and verify that database files from node 1 have been copied automatically to node 2. The directory name is based on your LDAP root:

LDAP root directory

ls /usr/local/ubisecure/ubilogin-sso/openldap/var/openldap-mdb/<your LDAP root>

Install Ubisecure SSO Tomcat and Accounting Service

Tune the Accounting Service scheduled job settings in node 2, see Accounting Service additional configuration / Recommended changes.

Tune Accounting Service settings on node 2

cd /usr/local/ubisecure/ubilogin-sso/ubilogin
vi custom/accounting/config/application.yaml

Depending on the Accounting Service secret key location setting you may need to copy a file from node 1 to node 2, see accounting.secret-key-location-uri in SSO Installation Accounting Service settings and Accounting Service security / Pseudonymisation.

Install Ubisecure SSO Tomcat and Accounting Service to node 2:

Install services on node 2

cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./config/tomcat/install.sh

Start Accounting Service and SSO Tomcat

Start services on node 2.

Start services on node 2

systemctl start ubisecure-accounting
systemctl start ubilogin-server

Configuring LDAP failover

Each Ubisecure SSO can be configured to connect to the LDAP directory on the other node in case the local directory cannot be reached. This is recommended if SSO and the directory are run on separate servers. If SSO and directory are run on the same server (default configuration), LDAP failover is not always desired. In this case this chapter can be skipped.

For Ubisecure SSO, LDAP failover is configured in file /usr/local/ubisecure/ubilogin-sso/ubilogin/webapps/uas/WEB-INF/jndi.properties. Add com.ubisecure.util.ldap.server.list setting in the end of the file. An example of such configuration follows:

LDAP failover settings

java.naming.factory.initial = com.ubisecure.util.ldap.jldap.JLDAP
java.naming.provider.url = ldap://localhost:389/cn=Ubilogin,dc=sso,dc=example,dc=com
java.naming.security.authentication = simple
 
java.naming.security.principal = cn=Server,ou=System,cn=Ubilogin,dc=sso,dc=example,dc=com
java.naming.security.credentials = secret
 
com.ubisecure.util.ldap.server.list = ldap://node-1-hostname/ ldap://node-2-hostname/

The order of the servers in the server.list value are insignificant. During startup, both servers are contacted at the same time. The server which responds fastest to the request is used until a failure situation occurs.

For other Ubisecure applications, LDAP failover is configured in the following configuration files:

Ubisecure SSO Management: <installation directory>/ubilogin/webapps/ubilogin/WEB-INF/jndi.properties
Ubisecure Password application: <installation directory>/ubilogin/webapps/password/WEB-INF/ubilogin.jndi.properties
Ubisecure Search: <installation directory>/ubilogin/webapps/search/WEB-INF/jndi.properties
Ubisecure OTP Server: <installation directory>/ubilogin/webapps/otpserver/WEB-INF/jndi.properties
Ubisecure SSO REST API: <installation directory>/ubilogin/webapps/sso-api/WEB-INF/jndi.properties

These changes must be made on both nodes.

After the change, activate the applications on each node:

Activate applications on each node

systemctl stop ubilogin-server
cd /usr/local/ubisecure/ubilogin-sso/ubilogin
./config/tomcat/update.sh
systemctl start ubilogin-server