TrueNAS SCALETrueNAS SCALE Nightly Development Documentation
This content follows experimental early release software. Use the Product and Version selectors above to view content specific to a stable software release.

Cloud Sync Tasks

This section has tutorials for configuring and managing data backups to from TrueNAS to various 3rd party cloud service providers. This article provides instructions on adding a cloud sync task, configuring environment variables, running an unscheduled sync task, creating a copy of a task with a reversed transfer mode, and troubleshooting common issues with some cloud storage providers.

TrueNAS can send, receive, or synchronize data with a cloud storage provider. Cloud sync tasks allow for single-time transfers or recurring transfers on a schedule. They are an effective method to back up data to a remote location.

These providers are supported for Cloud Sync tasks in TrueNAS SCALE:

Using the cloud means data can go to a third-party commercial vendor not directly affiliated with iXsystems. You should fully understand vendor pricing policies and services before using them for cloud sync tasks.

iXsystems is not responsible for any charges incurred from using third-party vendors with the cloud sync feature.

Cloud Sync Task Requirements

You must have:

  • All system storage configured and ready to receive or send data. A cloud storage provider account and location (like an Amazon S3 bucket).

You can create cloud storage account credentials using Credentials > Backup Credentials > Cloud Credentials before adding the sync task or add it when configuring the cloud sync task using Add on the Data Protection > Cloud Sync Task widget to open the Cloudsync Task Wizard. See the Cloud Credentials article for instructions on adding a backup cloud credential.

Creating a Cloud Sync Task

To add a cloud sync task, go to Data Protection > Cloud Sync Tasks and click Add. The Cloudsync Task Wizard opens.

CloudSyncTaskWizardProviderScreen

  1. Select an existing backup credential from the Credential dropdown list. If not already added as a cloud credential, click Add New to open the Cloud Credentials screen to add the credential. Click Save to close the screen and return to the wizard.

  2. Click Next to open the Where and When wizard screen.

    CloudSyncTaskWizardWhatandWhenScreen

  3. Select the option from Direction and in Transfer Mode. Select the location where to pull from or push data to in the Folder field.

  4. Select the dataset location in Directory/Files. Browse to the dataset to use on SCALE for data storage. Click the arrow to the left of the name to expand it, then click on the name to select it.

    If Direction is set to PUSH, click on the folder icon to add / to the Folder field.

    Browsing to select a path

    Click the arrow to the left of the folder icon to expand that folder and show any child datasets and directories. A solid folder icon shows for datasets and an outlined folder for directories. A selected dataset or directory folder and name shows in blue.

  5. Cloud provider settings change based on the credential you select. Select or enter the required settings that include where files are stored. If shown, select the bucket on the Bucket dropdown list.

  6. Select the time to run the task from the Schedule options.

  7. Click Save to add the task.

Use Dry Run to test the configuration before clicking Save or select the option on the Cloud Sync Task widget after you click Save. TrueNAS adds the task to the Cloud Sync Task widget with the Pending status until the task runs on schedule.

Encrypting Cloud Sync Tasks

The option to encrypt data transferred to or from a cloud storage provider is available in the Advanced Options settings.

Select Remote Encryption to use rclone crypt encryption during pull and push transfers. With Pull selected as the Transfer Direction, the Remote Encryption decrypts files stored on the remote system before the transfer. This requires entering the same password used to encrypt data in both Encryption Password and Encryption Salt.

With Push selected as the Transfer Direction, data is encrypted before it is transferred and stored on the remote system. This also requires entering the same password used to encrypt data in both Encryption Password and Encryption Salt.

Filename Encryption is selected by default. When selected, the pull and push transfers encrypt or decrypt file names with the rclone Standard file name encryption mode. The original directory structure of the files is preserved. When disabled, encryption does not hide file names or directory structure, file names can be 246 characters long, use sub-paths, and copy single files. When enabled, file names are encrypted, file names are limited to 143 characters, directory structure is visible, and files with identical names have identical uploaded names. File names can use sub-paths, single copy files, and shortcuts to shorten the directory recursion.

Troubleshooting Transfer Mode Problems

Sync keeps all the files identical between the two storage locations. If the sync encounters an error, it does not delete files in the destination.

Dropbox Issues

One common error occurs when the Dropbox copyright detector flags a file as copyrighted.

BackBlaze B2 Issues

Syncing to a Backblaze B2 bucket does not delete files from the bucket, even after deleting those files locally. Instead, files are tagged with a version number or moved to a hidden state. To automatically delete old or unwanted files from the bucket, adjust the Backblaze B2 Lifecycle Rules.

A directory, deleted in BackBlaze B2 and notated with an asterisk, do not display in the SCALE UI. These folders are essentially empty directories and Backblaze API restricts them so they do not display.

Amazon S3 Issues

Sync cannot delete files stored in Amazon S3 Glacier or S3 Glacier Deep Archive. First restore these files by another means, like the Amazon S3 console.

Using Scripting and Environment Variables

Advanced users can write scripts that run immediately before or after the cloud sync task.

Use either the Advanced Options screen accessed from the Cloudsync Task Wizard or Edit Cloud Sync Task screen, scroll down to the Advanced Options to locate and then enter environment variables in either the Pre-script or Post-script fields. The Post-script field only runs when the cloud sync task succeeds.

Click here for Environment Variables

General Environment Variables

  • CLOUD_SYNC_ID
  • CLOUD_SYNC_DESCRIPTION
  • CLOUD_SYNC_DIRECTION
  • CLOUD_SYNC_TRANSFER_MODE
  • CLOUD_SYNC_ENCRYPTION
  • CLOUD_SYNC_FILENAME_ENCRYPTION
  • CLOUD_SYNC_ENCRYPTION_PASSWORD
  • CLOUD_SYNC_ENCRYPTION_SALT
  • CLOUD_SYNC_SNAPSHOT

Provider-Specific Variables

There also are provider-specific variables like CLOUD_SYNC_CLIENT_ID or CLOUD_SYNC_TOKEN or CLOUD_SYNC_CHUNK_SIZE.

Remote storage settings:

  • CLOUD_SYNC_BUCKET
  • CLOUD_SYNC_FOLDER

Local storage settings:

  • CLOUD_SYNC_PATH

Running an Unscheduled Cloud Sync Task

Saved tasks activate based on the schedule set for the task. Click Run Now on the Cloud Sync Task widget to run the sync task before the saved scheduled time. You can also expand the task on the Cloud Sync Tasks screen and click Run Now on the task details screen.

An in-progress cloud sync must finish before another can begin. Stopping an in-progress task cancels the file transfer and requires starting the file transfer over.

To view logs about a running task, or its most recent run, click on the State oval.

Using Cloud Sync Task Restore

To create a new cloud sync task that uses the same options but reverses the data transfer, select for an existing cloud sync on the Data Protection page. The Restore Cloud Sync Task window opens.

Cloud Sync Restore
Figure 2: Cloud Sync Restore

Enter a name in Description for this reversed task.

Select the Transfer Mode and then define the path for a storage location on TrueNAS scale for the transferred data.

Click Restore.

TrueNAS saves the restored cloud sync as another entry in Data protection > Cloud Sync Tasks.

If you set the restore destination to the source dataset, TrueNAS may alter ownership of the restored files to root. If root did not create the original files and you need them to have a different owner, you can recursively reset their ACL permissions through the GUI or run chown from the Linux CLI.

Cloud Sync Tasks Contents

  • Backing Up Google Drive to TrueNAS SCALE: Provides instructions on adding Google Drive cloud credentials using the Add Cloud Credentials and Add Cloud Sync Task screens, and on working with Google-created content.

    • Adding a Storj Cloud Sync Task: Provides instructions on how to set up a Storj cloud sync task and how to configure a Storj-TrueNAS account to work with SCALE cloud credentials and cloud sync tasks.

      • Adding a Google Photos Cloud Sync Task: Provides instructions on how to set up Google Photos API credentials and use them to create a cloud sync task.