Gitlab Upgrade

2020-07-19

Here is the process I’ve polished during my contracts for various clients, where I was supporting self-hosted version of GitLab.

Pre-upgrade

GitLab create new releases on the 22th of each month.
Sometimes a few days later they issue patch release, if they find any bugs.
So it is recommended to wait a couple of weeks, and for example do Upgrades first Friday of every month.

T - 1 week

• review current versions: https://<GITLAB_SERVER_NAME>/admin
• find new release at: https://about.gitlab.com/releases/
• create a new Issue
• reserve a time slot
• review Release Notes for new release
• review Upgrade recommendations
• pay special attention if multi-step upgrade is needed, see all available versions, example for Debian\Ubuntu: apt-get update && apt list -a gitlab-ee
• review top-level upgrade process
• review versions compatibility between PostgreSQL and Omnibus GitLab, and plan PostgreSQL upgrade if needed and not part of GitLab upgrade

T - 1 day

• announce service down time in Slack \ email
• announce as a Broadcast Message at https://<GITLAB_SERVER_NAME>/admin/broadcast_messages

During Upgrade

• check that instance snapshot is created last night, for example when hosted in GCP see snapshots
• check that there are no remaining migrations
• create backup: sudo gitlab-backup create SKIP=repositories,artifacts,lfs,registry, see doc
• backup /etc/gitlab/{gitlab.rb,gitlab-secrets.json} files separately somewhere, as they contain sensitive data and are not included in the backup
• make sure that sudo gitlab-ctl reconfigure can run successfully
• if PostgreSQL needs to be upgraded separately, follow doc, also per this you might want to specify version as well, e.g.: sudo gitlab-ctl pg-upgrade -V 11
• upgrade GitLab according to GitLab update documentation:
  • sudo apt-get update
  • sudo apt-get install gitlab-ee - to upgrade to the latest version
  • sudo apt-get upgrade gitlab-ee=12.0.12-ee.0 - specify version in case of multi-step upgrade
• [optional] upgrade other packages in the system: sudo apt-get upgrade
• restart server and check if GitLab comes back up

Post-Upgrade

• do some testing (push/pull, major repositories, runners/pipelines/jobs, boards)
• announce in Slack that Upgrade was successful
• add another Broadcast Message at https://<GITLAB_SERVER_NAME>/admin/broadcast_messages for few hours that upgrade was successful