Last updated June 15, 2009 20:01, by Glenn Brunette
Feedicon  


BACKGROUND

ZFS Automatic Snapshots were integrated into OpenSolaris development build 100 allowing for the automatic creation of ZFS snapshots in accordance with various rules limiting which file systems will be impacted and how often snapshots are taken, etc. By default, the snapshots are stored on disk. but wouldn't it be nice if we could store these snapshots in the Cloud?

It turns out that we can. Using the SMF service parameter, backup-save-cmd, we can instruct this service to stream the snapshot to a command of our choice. This is where this project comes in. The goal of this project is to simplify the process of taking this ZFS data stream, compressing and encrypting it, and then storing the result in a Cloud Storage service such as the Sun Cloud Storage Service and Amazon S3. This project does support splitting up of a single ZFS stream into multiple files (if needed to work around a given Cloud providers maximum file limits).

FUNCTIONALITY

This tool supports the following modes of operation:

   * compression
   * encryption
   * splitting

The cryptographic operations performed by these tools are enabled by OpenSSL (or the Solaris Cryptographic Framework on Solaris 10 or OpenSolaris). By default, Solaris is used as it is the primary delivery vehicle for ZFS. Note that on platforms using the UltraSPARC T2 (Niagara 2) processor, these cryptographic operations can be hardware accelerated. To change the default cryptographic provider, use the configuration file defined below.

FUNCTIONAL DIAGRAM

WARNING


DEPENDENCIES

This software module depends on the Cloud Safety Box tool to perform the actual compression, encryption, and splitting operations. The Cloud Safety Box tool must be properly installed before this tool is used. Further the fully-qualified path to the Cloud Safety Box tool must be defined in the configuration file noted below.

Note that the Cloud Safety Box tool has additional dependencies. Those dependencies must also be satisfied prior to using this tool.

DOWNLOAD

Currently, this software is accessible from a Mercurial source code repository and a (tar) bundle.

INSTALLATION

1. Install, configure and test any dependent software packages the Cloud Safety Box tool mentioned above.

2. Install the zfs-backup-to-s3.sh script into a user-specified target directory and has correct ownership and permissions:

      # cp zfs-backup-to-s3.sh [path]/zfs-backup-to-s3.sh
      # chown zfssnap:daemon [path]/zfs-backup-to-s3.sh
      # chmod 755 [path]/zfs-backup-to-s3.sh


3. Install the zfs-backup-to-s3.dfl command into /etc/default and set correct ownership and permissions:

      # cp zfs-backup-to-s3.dfl /etc/default/zfs-backup-to-s3
      # chown zfssnap:daemon /etc/default/zfs-backup-to-s3
      # chmod 600 /etc/default/zfs-backup-to-s3


4. Configure the /etc/default/zfs-backup-to-s3 file as follows:

      * Select the encryption provider to be used (default: Solaris).
      * Select the encryption algorithm to be used (default: AES).
      * Select the encryption key type to be used (default: file).
      * Specify the path to the encryption key file (default: /etc/security/zfs-backup-to-s3.enc_key)
      * Specify the encryption key label (if used and key type is set to "label").
      * Specify the path to the Cloud Safety Box tool.
      * Specify the path to the "Another S3 Bash Interface" tool (required by Cloud Safety Box).
      * Specify the Amazon S3 access key to be used.
      * Specify the path to file containing the Amazon S3 secret key
         (default: /etc/security/zfs-backup-to-s3.s3_key)
      * Specify the Amazon S3 bucket to be used (must already exist).

Example: /etc/default/zfs-backup-to-s3:

      # Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
      # Use is subject to license terms.
      
      # ENC_PROVIDER defines the cryptographic services provider used for
      # encryption operations.  Value values are "solaris" and "openssl".
      ENC_PROVIDER="solaris"
      
      # ENC_ALG defines the algorithm that will be used for the encryption
      # operation.  Valid values are those reported by encrypt(1) (when
      # using the Solaris provider) and openssl(5) (when the OpenSSL 
      # provider is used).
      ENC_ALG=aes
      
      # KEY_TYPE defines the type of key that will be used for the encryption
      # operation.  Valid values include "file" and "label".
      KEY_TYPE=file
        
      # KEY_FILE defines the location of the key used for the encryption
      # operation.  This must be an absolute path to a file that is owned
      # and accessibly only by the "zfssnap" role.  The key contained in
      # this file must be compatible with the ENC_ALG algorithm selected
      # above.  See encrypt(1) and openssl(5) for more information.
      KEY_FILE=/etc/security/zfs-backup-to-s3.enc_key
        
      # KEY_LABEL specifies the label of a symmetric token key stored in 
      # a PKCS#11 token.
      KEY_LABEL=
        
      # MAX_FILE_SIZE specifies the maximum file size that can be sent
      # to the Cloud storage provider without first splitting the file
      # up into chunks (of MAX_FILE_SIZE or less).  This value is specified
      # in Kbytes.  If this variable is 0 or not defined, then this service
      # will _not_ attempt to split the file into chunks.
      MAX_FILE_SIZE=40000000
      
      # S3C_CRYPTO_CMD_NAME defines the fully qualified path to the
      # s3-crypto.ksh program which is used to perform compression,
      # encryption, and file splitting operations.
      S3C_CRYPTO_CMD_NAME="/opt/s3/bin/s3-crypto.ksh"
      
      # S3C_CLI_CMD_NAME defines the fully qualified path to the program
      # used to perform actual upload operations to the Cloud storage
      # provider.  This program is called (indirectly) by the
      # s3-crypto.ksh program defined by the S3C_CRYPTO_CMD_NAME variable
      # above.
      S3C_CLI_CMD_NAME="/opt/s3/bin/s3-suncloud.sh"
         
      # S3_ACCESS_KEY_ID defines the Amazon Simple Storage Service (S3)
      # Access Key to be used for the storage of the encrypted snapshots.
      S3_ACCESS_KEY_ID=1234567890
       
      # S3_SECRET_ACCESS_KEY defines the location of a file containing
      # the Amazon S3 secret access key associated with the access key
      # defined by S3_ACCESS_KEY_ID.
      S3_SECRET_ACCESS_KEY=/etc/security/zfs-backup-to-s3.sun_key
        
      # S3_BUCKET defines the location into which the encrypted snapshot
      # will be stored on Amaozn S3.  This S3 bucket must already exist
      # and be accessible (writable) to the S3_ACCESS_KEY_ID defined
      # above. 
      S3_BUCKET="zfs-snapshots"


5. (If necessary) create a file containing the encryption key. This key can be generated using the Cloud Safety Box tool (keygen command).

6. (If necessary) set the ownership and permissions of the encryption key file. By default, the tool expects this file to exist under /etc/security. The contents of this file must be strongly protected. Ensure that only authorized users have access to the zfssnap role or have administrative privileges on the system.

      # chown zfssnap:daemon [path]/zfs-backup-to-s3.enc_key
      # chmod 400 [path]/zfs-backup-to-s3.enc_key


7. Set the ownership and permissions of the Amazon S3 secret key file. By default, the tool expects this file to exist under /etc/security. The contents of this file must be strongly protected. Ensure that only authorized users have access to the zfssnap role or have administrative privileges on the system.

      # chown zfssnap:daemon [path]/zfs-backup-to-s3.s3_key
      # chmod 400 [path]/zfs-backup-to-s3.s3_key


8. Determine which of the ZFS Automatic Snapshot services you want to use from the following list:

      # svcs auto-snapshot


9. Configure the selected instances (selected in Step 7) to use the zfs-backup-to-s3 command:

      # svccfg -s auto-snapshot:[instance] setprop zfs/backup-save-cmd = [path]/zfs-backup-to-s3.sh
      # svccfg -s auto-snapshot:[instance] setprop zfs/backup = full
      # svcadm refresh auto-snapshot:[instance]


10. Modify the ZFS Automatic Snapshot method to export the LAST_SNAP variable. Note that this is the only step that requires the modification of a file delivered in the Solaris OS. Note that updates to this file may cause these changes to be overwritten so be sure to verify whether the changes need to be reinstated after updates have been applied. There is currently a RFE open against OpenSolaris to address this issue:

   * 6847543 zfs-auto-snapshot should export shell variables when performing backups

After each of the following lines, insert a new line with the text export LAST_SNAP:

      * line 745: print_note "Starting incr. ZFS send of differences between $PREV_SNAP and $LAST_SNAP."
      * line 750: print_note "Starting ZFS send of $LAST_SNAP."

In both cases, the new lines should be placed immediately before a call to $PFZFS send. An OpenSolaris RFE has be filed to make this change so that future versions of this tool will no longer need this workaround.

11. Enable the ZFS Automatic Snapshot instance that was selected in Step 7:

      # svcadm enable auto-snapshot:[instance]


12. (Optional) verify that the ZFS encrypted snapshots are being properly stored into the Cloud. This command should only be executed after the enabled service has had a chance to take and send the snapsho. To simplify this demonstration, the default settings are used for the ZFS Encrypted Backups to the Cloud service and the zfssnap role is used to execute commands in this example.

Configure the essential environment variables:

      zfssnap$ . /etc/default/zfs-backup-to-s3
      zfssnap$ export S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY
      zfssnap$ PATH=/usr/gnu/bin:$PATH:${S3_CMD_PATH}

List the contents of the bucket into which the ZFS encrypted backup should have been stored:

      zfssnap$ s3-crypto.ksh -m ls -b ${S3_BUCKET}

Get the compressed and encrypted backup from the Cloud provider:

      zfssnap$ s3-crypto.ksh -m get -b ${S3_BUCKET} -l [local file name] -r [remote file name]

Decrypt the ZFS encrypted backup:

      zfssnap$ decrypt -a ${ENC_ALG} -k ${KEY_FILE} -i [local file name] -o [decrypted file name].bz2

Decompress the ZFS backup:

      zfssnap$ bzip2 -d [decrypted file name].bz2

Verify that the resulting file is a correct ZFS data stream:

      zfssnap$ file [decrypted file name]
      zfssnap$ strings [decrypted file name] | head -1

Note that the individual s3-crypto.ksh, decrypt, and bzip2 steps could also have been collapsed into a single command:

      zfssnap$ s3-crypto.ksh -C -c -a ${ENC_ALG} -k ${KEY_FILE} -m get -b ${S3_BUCKET} \
         -l [decrypted file name] -r [remote file name]

ADDITIONAL CONSIDERATIONS

1. By default, the ZFS Automatic Snapshot functionality operates on all of the pools and file systems. Since the Storage Cloud can only operate on entire files, the full contents of the ZFS encrypted data stream must be captured before it can be uploaded. This can create a significant burden depending upon how much space is available in TMPDIR. For testing this script, it may be more appropriate to disable Automatic Snapshots on the pools and enable it selectively on those file systems under test as in the following example:

      $ pfexec zfs set com.sun:auto-snapshot=false pool
      $ pfexec zfs set com.sun:auto-snapshot=true pool/home/gbrunett
      $ pfexec zfs get com.sun:auto-snapshot pool/home/gbrunett
      NAME                PROPERTY               VALUE                  SOURCE
      pool/home/gbrunett  com.sun:auto-snapshot  true                   local

ADDITIONAL READING