We recently helped a customer that needed to do a rollback of an upgrade. They were operating in a lab environment, practicing to upgrade Nuage software in a separate production. Before executing the production upgrade, they wanted to have a documented procedure to rollback and recover their VSD cluster.
During the upgrade, they had followed the MetroÆ instructions and executed vsd_ha_upgrade_database_backup_and_decouple. The vsd_ha_upgrade_database_backup_and_decouple operation did the following:
- Mounted the upgrade ISO image on the VSD
- Executed the ISO image backup script, which created the backup files on the VSD itself
- Copied the back files from the VSD to the Ansible host
Having preserved the backup files on the Ansible host, this customer was well positioned to execute a simple rollback. Here is the procedure:
- Configure build_vars.yml for the proper VSD hostnames, IP addresses, and so on. This includes making sure that the variable nuage_unzipped_files_dir points to a directory that contains the *old* versions of the VSD images.
- Execute vsd_predeploy to bring up 3 new VMs running the old version
- Manually copy (via scp) the backup files from the Ansible host to all 3 VSDs. The destination directory is /opt/vsd/data/
- Execute vsd_deploy which will install VSD software on each node, pulling in and restoring the backups from /opt/vsd/data
- Execute vsd_postdeploy
With the successful completion of these steps, the VSD cluster (or stand-alone!) should be up and healthy once again.
Note: It is important to copy the backup files to the VSDs and let the vsd_install script restore the data. Restoring the data manually will introduce a mismatch between the certificates that were generated for the new VMs and the certificates stored in the restored DB.