Documentation (2024.2 Enterprise)

# Backups and restoration

Backups are available as part of the Helix TeamHub packages. Backups are activated with configuration flags. The nature of backups changes from one TeamHub component to another:

Component | Method | Schedule | Description
--- | --- | --- | ---
Assets | Archival | Daily at 00:00 | Attachments, Avatars, Logs, Configuration files
MongoDB | Archival | Daily at 00:00 | Mongo dumps
Repositories | Incremental sync | Daily at 02:00 | Repository type specific tools: git, svnsync, hg
Docker Registry | Archival | Daily at 04:00 | Docker images

## Preparation

TeamHub stores backups at /var/opt/hth/backups. For production deployments, the recommendation is to mount a reliable external storage that has enough space to accommodate at least 1.5 times the TeamHub data set (/var/opt/hth/shared).

## Enabling backups

To enable backups, activate them by setting the configuration flags as shown below:

### Standard

1. Merge the following configuration to /var/opt/hth/shared/hth.json and make sure you add the backup settings under the existing keys if they already exist in the configuration.

       {
         "backend": {
           "backups": true
         },
         "mongodb": {
           "backups": true
         },
         "repos": {
           "backups": true
         },
         "docker_registry": {
           "backups": true
         }
       }

2. Apply the changes by reconfiguring TeamHub:

sudo hth-ctl reconfigure

### Enterprise and HA

In Enterprise and HA deployment, the TeamHub services are distributed across the server roles. To enable backups, activate the backup flags in /var/opt/hth/shared/hth.json on the appropriate servers. Make sure you add the backup settings under the existing keys if they already exist in the configuration.

#### Helix TeamHub DB

    {
      "mongodb": {
        "backups": true
      }
    }

By default, mailing is not configured for the DB node. To get notifications if failures occur during DB backups, you need to add the following:

1. Configure the postfix section of hth.json to be identical with the one on your web node.

2. Add the email that will receive the notifications to the app section of hth.json:

       {
         "app": {
           "email": "support@acme.com"

         }
       }

#### Helix TeamHub Web

    {
      "backend": {
        "backups": true
      },
      "repos": {
        "backups": true
      },
      "docker_registry": {
    	"backups": true
      }
    }

Apply the changes by reconfiguring TeamHub on each server:

sudo hth-ctl reconfigure

## Configuring how many backups to keep before oldest backup gets removed

You can configure archival mode backups to prune old backups. The configuration below keeps the 30 latest backup archives and deletes the oldest when a new backup occurs.

    {
      "backups": {
        "keep": 30
      }
    }

A separate setting exists for the Docker Registry:

    {
      "docker_registry": {
        "backups_keep": 5
      }
    }

Unlike with other backups, the Docker Registry backups_keep value should be low because images tend to take up a lot of space, even if they are compressed.

## Restoring backups

Because TeamHub backups are modular, when it comes to restoring the system from a backup, it is important to consider the following:

- Follow the correct order of restoring the backup components:

  1. Assets
  2. Database
  3. Repositories
- Because backups for TeamHub components are taken daily, it is important to restore all components from the same day. Doing otherwise will lead to data inconsistency.

### Stopping services

Before starting the restoration process, it is a good idea to stop all the TeamHub services:

sudo hth-ctl stop

### Restoring assets

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the hth user.
2. Copy the backend_backup.tar from backup storage, which is located at /var/opt/hth/backups/backend_backup/<date>/.

3. Extract the archives and restore:

       sudo su - hth
       tar xvf backend_backup.tar
       cd backend_backup/archives; ls *.tar.gz | xargs -i tar xvf {} -C /

### Restoring MongoDB databases

1. SSH into the TeamHub DB server (or the Combo) and switch to the `hth` user.
2. Copy mongodb_backup.tar from the backup storage /var/opt/hth/backups/mongodb_backup/<date>/.

3. Start MongoDB, extract the archives, and restore:

       sudo su - hth
       sudo hth-ctl start mongodb
       tar xvf mongodb_backup.tar; cd mongodb_backup/databases/; tar xvf MongoDB.tar.bz2

   For Standard deployment without MongoDB authentication, use the following command to restore the database:

   mongorestore --port 4002 --drop MongoDB/

   For Enterprise, HA or Standard deployments with MongoDB authentication, use admin credentials to restore the database:

   mongorestore --port 4002 -u <admin-username> -p <admin-password> --drop MongoDB/

### Restoring repositories

Repositories are stored under the /var/opt/hth/shared/companies/<company>/projects/<repo_type>/repositories directory. To restore backups from the backups store:

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the `hth` user.

2. Run the following script:

       #!/bin/bash

       companies=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/`
       # Loop through companies
       for company in $companies; do
         # Loop through projects
         projects=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/`
         for project in $projects; do
           echo "Restoring repositories for project $project in company $company"
           repos_dest="/var/opt/hth/shared/companies/$company/projects/$project/repositories/"
           if [ ! -d $repos_dest ]; then
             # Company/Project may have been renamed
             echo "Creating $repos_dest"
             mkdir -p $repos_dest
           fi
           rsync -av --delete /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/$project/repositories/ $repos_dest
         done
       done

3. To restore the repositories hooks, execute the following commands to regenerate them:

       sudo su - hth
       sudo hth-ctl start redis
       cd /opt/hth/application/backend/current
       rake hth:restore:all

### Restoring Docker Registry

1. SSH into one of the TeamHub Web servers (or the Combo) and switch to the hth user.
2. Copy docker_registry_backup.tar from the backup storage /var/opt/hth/backups/docker_registry_backup/<date>/.

3. Extract the archives and restore:

       sudo su - hth
       rm -rf /var/opt/hth/shared/storage/docker_registry/docker/
       cd /var/opt/hth/backups/docker_registry_backup/<date>
       tar xvf docker_registry_backup.tar
       cd docker_registry_backup/archives; ls *.tar.gz | xargs -i tar xvf {} -C /

### Starting TeamHub

Start back all TeamHub services:

sudo hth-ctl start

### Reconfiguring TeamHub

If there have been changes to the hth.json restored from backups, apply those changes by running the reconfigure command:

sudo hth-ctl reconfigure

Your search for returned results

Your search for returned results