cancel
Showing results for 
Search instead for 
Did you mean: 

Automating Certificate Updates for the Services Director Communications Channel

Services Director 19.1 introduced a new communications channel to connect to Traffic Managers (vTM) using the REST API of vTM via a secure authenticated link, that is particularly useful in networks behind NATs or firewalls.

 

However, this mutually authenticated link needs care when updating Services Director's server certificate. For information on the Services Director Communications Channel, and how to update the certificates manually, see Introducing the Services Director Communications Channel.


Updating Services Director certificates with a script

Where direct access to the vTM GUI is not available, or estates where breaks in communication cannot be tolerated, it is possible to update the Services Director certificate with a script. The script uses the Services Director's vTM REST API proxy to configure the vTMs using the comms-channel to use a pair of server certificates, both old and new server certificates for Services Director. This means that the vTMs can be primed before the Services Director server certificate is updated, and will be able to reconnect immediately when it has changed.

 

The script is called update_all_vtm_certificates.py, is written for Python 3 and tested on Ubuntu 18.04), and is available for download from https://github.com/pulse-vadc/sd-update-vtm-certificates-script

 

It is highly recommended that customers should follow the instructions in the README.md, as this describes how to use Python's virtualenv functionality to isolate the Python dependencies used for update_all_vtm_certificates.py from the native Python libraries on the machine it is run on (and vice-versa).

 

The script is intended for use as follows:

 

  1. The user generates a new key and server certificate for the Services Director as described in the Advanced User Guide.

  2. The script is called passing in:
    - (A file path to) the new server certificate
    - (A file path to) the current server certificate
    - A URL to the Services Director's REST API
       (the Services Endpoint Address plus port for Services Director virtual appliances)
    - The administrator username for Services Director
    - The administrator password for Services Director
    - Note that if the password is not provided on the command line,
       the script will prompt interactively for it.
    - (A file path to) the new key corresponding to that certificate
       (for checking against the new server certificate only)

  3. With this invocation, the script will:
    - Validate the pairing of the new server certificate and key
    - Disable monitoring (because this could otherwise interfere with the upgrade process)
    - For each comms channel enabled vTM in the Services Director's estate:
    - Use the Services Director's vTM REST API proxy to install the new server certificate,
      with the old server certificate as a secondary server certificate.
    - Wait for the comms channel to be re-established
       (the change above causes an immediate disconnection),
       then check that the certificate change has taken effect
    - Re-enable monitoring
    - Any comms channel enabled vTMs that have not been updated successfully
       are recorded, and their identities output into the vTM-update-failed.txt file.

  4. If any vTMs have failed to be updated according to vTM-updated-failed.txt, step 2 can be repeated, this time passing in an additional parameter --vtms-to-update set to vTM-updated-failed.txt. This will retry the update process for just those instances in the file. This step can be repeated as needed to ensure all vTMs are updated; for any vTMs that continue to fail to be updated, a manual intervention may be required (for more information on manually updating the certificate, see the Manual Update section in Introducing the Services Director Communications Channel)

  5. At this point, the Services Director's server certificate can be updated in Services Director itself. In the Services Director's GUI, select System > Service SSL Certificate, then click the link to update the certificate. Comms channel enabled vTMs in the Services Director's estate should reconnect to Services Director once the certificate has been updated.

  6. Once the comms channels for the affected vTMs have re-established, the script can be run a second time to remove the old Services Director server certificate from the comms channel enabled vTMs in the estate. In this case, the --remove-old-certificates flag must be added to the parameters of the script (and --vtms-to-update removed). This will remove the old server certificate.

 

Example usage session:

In this example, we are updating a Services Director with three registered vTM instances that use the comms channel, one of which is temporarily unavailable.

As a reminder, the script is called update_all_vtm_certificates.py, and is available for download from https://github.com/pulse-vadc/sd-update-vtm-certificates-script.

$ python3 update_all_vtm_certificates.py --new-sd-service-certificate ../../mim/test_certs/cert1.pem --current-sd-service-certificate ../../mim/test_certs/cert.pem --sd-url https://10.62.164.81:8100 --sd-username admin --sd-password mypassword --new-private-key ../../mim/test_certs/key1.pem 
INFO - Disabled monitoring on Services Director
Query vTMs : 100%|####################################################################| 3/3 [00:00<00:00, 23.13it/s]
Update vTMs : 0%| | 0/3 [00:00<?, ?it/s]WARN - Failed to update vtm Instance-P4FJ-CHM8-VEIT-41NG: 'properties'
Update vTMs : 100%|####################################################################| 3/3 [00:06<00:00, 2.09s/it]

The following vTMs were successfully updated:
Instance-ZS9W-5APL-TV5P-1OOA
Instance-52CO-WKF2-65MI-MNSK

There were errors updating the following vTMs.
Instance-P4FJ-CHM8-VEIT-41NG
A list of vTMs which were not updated has been saved in vTM-update-failed.txt to retry these vTMs re-run this script with the paramater --vtms-to-update vTM-update-failed.txt

Summary:
not using comms channel: 0
successfully updated : 2
failed to update : 1

Some vTMs have not had their server_certificate_secondary updated. Please
ensure that the server certificate is updated for these vTMs, either by
retrying this script, or manually updating the server_certificate for each vTM.
Once the server_certificate is updated on all vTMs you can change the SSL
Service Certificate in services director.
INFO - Reset monitoring back to all on Services Director

The script runs, and successfully updates TWO of the vTMs, but for some reason the third vTM was not successfully updated. This session outputs a file vTM-update-failed.txt which contains a list of identifiers for instances that are still to be updated in the form of a JSON list, and shows the third vTM:

["Instance-P4FJ-CHM8-VEIT-41NG"]

 

We run the script a second time, adding this file to the parameter list using --vtms-to-update vTM-update-failed.txt:

 

$ python3 update_all_vtm_certificates.py --new-sd-service-certificate ../../mim/test_certs/cert1.pem --current-sd-service-certificate ../../mim/test_certs/cert.pem --sd-url https://10.62.164.81:8100 --sd-username admin --sd-password password --new-private-key ../../mim/test_certs/key1.pem --vtms-to-update vTM-update-failed.txt 
INFO - Disabled monitoring on Services Director
Update vTMs : 100%|####################################################################| 1/1 [00:04<00:00,  4.86s/it]

The following vTMs were successfully updated:
Instance-P4FJ-CHM8-VEIT-41NG

Summary:
  vtms to update         : 1
  successfully updated   : 1
  failed to update       : 0
INFO - Reset monitoring back to all on Services Director

 

At this point, all three vTMs have been updated, and are showing as being successfully monitored in the Services Director GUI:

cm-sd-cert-4.png

 

A final invocation of the script is then used to remove the old server certificate from the vTM estate, with the parameter --remove-old-certificates:

$ python3 update_all_vtm_certificates.py --new-sd-service-certificate ../../mim/test_certs/cert1.pem --current-sd-service-certificate ../../mim/test_certs/cert.pem --sd-url https://10.62.164.81:8100 --sd-username admin --sd-password password --new-private-key ../../mim/test_certs/key1.pem --remove-old-certificates
INFO - Disabled monitoring on Services Director
Query vTMs : 100%|####################################################################| 3/3 [00:00<00:00, 23.21it/s]
Update vTMs : 100%|####################################################################| 3/3 [00:00<00:00, 3.59it/s]

The following vTMs have had their server_certificate_secondary removed:
Instance-P4FJ-CHM8-VEIT-41NG
Instance-ZS9W-5APL-TV5P-1OOA
Instance-52CO-WKF2-65MI-MNSK

Summary:
vtms to update : 3
successfully updated : 3
failed to update : 0
INFO - Reset monitoring back to all on Services Director

 

 

Version history
Revision #:
4 of 4
Last update:
‎08-05-2019 04:55:AM
Updated by:
 
Contributors