Last updated June 11, 2009 14:05, by Glenn Brunette
Feedicon  


BACKGROUND

Many of the tools made available to store and retrieve files using Cloud Storage services such as Amazon S3 assume that the content will be stored "as is". There are many scenarios where it would be better to first encrypt the content before storing into the Cloud (and automatically decrypt it upon retrieval).

The goal of this software project is to provide this necessary function, namely to provide front-end encryption and decryption capabilities to augment existing tools that can already store and retrieve files in the Cloud. In addition, this tool will provide useful secondary functions such as compression and support for splitting/merging files that may be larger than a given threshold. Strong security algorithms and providers will be used, and this tool will support a variety of encryption key types (as supported by the underlying providers).

VERSIONS

  • v0.1 - Initial release supporting Amazon S3.
  • v0.2 - Added support for Sun Cloud.
  • v0.3 - Added support for the Cloud Safety Box (CSB) simplified use command.
  • v0.4 - Added support for compression, file splitting, and key labels (Solaris provider only).

FUNCTIONALITY

This tool supports the following modes of operation:

   * encryption
   * compression
   * splitting

in addition to any commands that can be passed through to the back-end functionality (e.g., list, remove, etc.) For "put" operations, compression is done before encryption and splitting is done last (if needed). The reverse process is used for "get" operations.

The cryptographic operations performed by these tools are enabled by OpenSSL (or the Solaris Cryptographic Framework on Solaris 10 or OpenSolaris). By default, OpenSSL is used as it enables the greatest level of portability. To use the Solaris cryptographic operations, use the "-p solaris" command line option. Note that on platforms using the UltraSPARC T2 (Niagara 2) processor, these cryptographic operations can be hardware accelerated.

All of the command line options as well as common use cases are available from the tool's usage message available using the "-h" command line option to the s3-crypto.ksh command. Note that in addition, the Cloud Safety Box, csb, command is also provided to enable a simple, easy to use interface at the expense of some measure of flexibility. If you want more control than what csb provides, simply use the s3-crypto.ksh script directly.

FUNCTIONAL DIAGRAM

DEPENDENCIES

These software modules are just front-end components and must be used with a back-end component that performs the actual operations against the Storage Cloud. To date, this software has been tested against both the Sun Cloud Storage Service and the Amazon Simple Storage Service (S3) using the "Another S3 Bash Interface" tool published by "nescafe5" at:

   http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1081

Thanks to "nescafe5" for posting this great tool. It is expected that this software can be easily adapted to other CLI back-end software components, however. To use with the Sun Cloud, this script must be modified to access object.storage.network.com in place of s3.amazonaws.com.

Note: By default, the "Another S3 Bash Interface" tool is configured to use HTTP not HTTPS. It is strongly recommended that HTTPS be used wherever possible. To enable HTTPS, change the curl command entry from http to https. Also, one of the curl SSL certificate verification steps must also be performed.

Note: By default, the s3-crypto.sh and csb programs will attempt to call the "Another S3 Bash Interface" command as s3. If this functionality was saved under a different name, simply use the S3C_CLI_CMD_NAME environment variable, setting its value to be the name of the program to be executed. For example:

   $ export S3C_CLI_CMD_NAME="s3-suncloud"


DOWNLOAD

Currently, this software is accessible from a Mercurial source code repository and a (tar) bundle.

CSB USAGE

To create a new storage bucket:

      csb put bucket


To remove an empty storage bucket:

      csb rm bucket


To display a listing of storage buckets:

      csb buckets


To display the contents of a specified bucket:

      csb ls bucket


To put a file into a specified bucket:

      csb put bucket local_file [remote_file]


To get a file from a specified bucket:

      csb get bucket remote_file [local_file]


To remove a file from a specified bucket:

      csb rm bucket remote_file


To remove all files from a specified bucket:

      csb rmrf bucket


CSB EXAMPLES


To display a listing of storage buckets:

      $ ./csb buckets
      test-bucket-a
      test-bucket-b
      test-bucket-c


To display the contents of a bucket:

      $ ./csb ls test-bucket-a
      test-file-1
      test-file-2
      test-file-3


To compress, encrypt and put a file (split if necessary) into a bucket:

      $ ./csb put test-bucket-a /export/myfile myfile
      enter aes-256-cbc encryption password:
      Verifying - enter aes-256-cbc encryption password:


To get (reassembling if necessary), decrypt and decompress a file from a bucket:

      $ ./csb get test-bucket-a myfile ./new_myfile
      enter aes-256-cbc decryption password:


To remove a file from a specified bucket:

      $ ./csb rm test-bucket-a myfile


To remove all files from a specified bucket:

      $ ./csb rmrf test-bucket-a


S3-CRYPTO.SH USAGE

To generate a new encryption/decryption key:

      ./s3-crypto.ksh -m genkey -k key_file [-s key_size]


To display a listing of storage buckets:

      ./s3-crypto.ksh -m buckets


To display the contents of a specified bucket:

      ./s3-crypto.ksh -m ls -b bucket


To put a file into a specified bucket:

      ./s3-crypto.ksh -m put -b bucket -l local_file -r remote_file


To compress and put a file into a specified bucket:

      ./s3-crypto.ksh -C -m put -b bucket -l local_file -r remote_file


To encrypt and put a file into a specified bucket:

      ./s3-crypto.ksh -c [ [-a enc_alg] [-p crypto_provider] [-k key_file |-K key_label] ]
         -m put -b bucket -l local_file -r remote_file


To get a file from a specified bucket:

      ./s3-crypto.ksh -m get -b bucket -r remote_file [-l local_file]


To get and decompress a file from a specified bucket:

      ./s3-crypto.ksh -C -m get -b bucket -r remote_file [-l local_file]


To get and decrypt a file from a specified bucket:

      ./s3-crypto.ksh -c [ [-a enc_alg] [-p crypto_provider] [-k key_file | -K key_label] ] -m get -b bucket
         -r remote_file [-l local_file]


To remove a file from a specified bucket:

      ./s3-crypto.ksh -m rm -b bucket -r remote_file


To remove all files from a specified bucket:

      ./s3-crypto.ksh -m rmrf -b bucket


Specific options:

-a. The name of the encryption algorithm to be used. The default is aes (Solaris encryption) and aes-256-cbc for OpenSSL. Valid names are those defined by the encrypt(1) and openssl(5) commands.

-b. The name of the bucket on the Storage Cloud.

-c. Enable encryption (put) / decryption (get).

-C. Enable compression (put) / decompression (get).

-h. Display this message.

-k. The name of the key file used only for cryptographic operations when the -c option is selected. If this parameter is not specified, then the program will prompt the user for a passphrase to be used for encrypt/decrypt operations.

-K. (Solaris cryptographic provider only). Specify the key label of a symmetric token key in a PKCS#11 token.

-l. The name (path) to the local file.

-L. The maximum file size limit (in Kbytes) used to determine if an input file should be split into chunks for upload.

-m. The mode of operation. Values include:

           * buckets.  list (your) Storage Cloud buckets
           * genkey.  generate a key (file) for crypto operations
           * get.  get a file from the Storage Cloud
           * ls.  list the contents of a Storage Cloud bucket
           * put.  put a file into the Storage Cloud
           * rm.  remove a file from a Storage Cloud bucket
           * rmrf.  remove all files from a Storage Cloud bucket

-p. The cryptographic services provider, currently either "openssl" (default) or "solaris".

-r. The name of the remote file.

-s. The size of the key file to be generated (in bytes). This parameter is only used with the genkey command mode.

-S. Split the file into chunks if its size is greater than 4 GB (default) or the size specified by the "-L" (size limit) option.

S3-CRYPTO.SH EXAMPLES

These examples are based upon the use of this tool with the "Another S3 Bash Interface" tool discussed above.

To generate a encryption/decryption key

      $ ./s3-crypto.ksh -m genkey -k ./my_key -s 32


To display a listing of storage buckets:

      $ ./s3-crypto.ksh -m buckets
      test-bucket-a
      test-bucket-b
      test-bucket-c


To display the contents of a bucket:

      $ ./s3-crypto.ksh -m ls -b test-bucket-a
      test-file-1
      test-file-2
      test-file-3


To create a new bucket:

      $ ./s3-crypto.ksh -m put -b new-bucket


To compress a file before storing it in a bucket:

      $ ./s3-crypto.ksh -C -m put -b test-bucket-a \
      -l ./myfile -r cloudfile


To encrypt a file before storing it in a bucket (OpenSSL w/key generated from passphrase):

      $ ./s3-crypto.ksh -c -m put -b test-bucket-a \
      -l ./myfile -r cloudfile
      enter aes-256-cbc encryption password:
      Verifying - enter aes-256-cbc encryption password:


To encrypt a file before storing it in a bucket (OpenSSL w/user-supplied key file):

      $ ./s3-crypto.ksh -c -m put -b test-bucket-a \
      -k ./my_key -l ./myfile -r cloudfile


To encrypt a file before storing it in a bucket (Solaris w/key generated from passphrase):

      $ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
      -l ./myfile -r cloudfile
      Enter passphrase: 
      Re-enter passphrase: 


To encrypt a file before storing it in a bucket (Solaris w/user-supplied key file):

      $ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
      -k ./my_key -l ./myfile -r cloudfile


To encrypt a file before storing it in a bucket (Solaris w/user-supplied key label):

      $ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
      -K my_key_label -l ./myfile -r cloudfile
      Enter PIN for Sun Software PKCS#11 softtoken  : 


To split a file before storing it in a bucket:

      $ ./s3-crypto.ksh -S -m put -b test-bucket-a \
      -l ./myfile -r cloudfile


To compress, encrypt and split a file before storing it in a bucket (OpenSSL w/key generated from passphrase):

      $ ./s3-crypto.ksh -C -c -S -m put -b test-bucket-a \
      -l ./myfile -r cloudfile


To reassemble a file after retrieving it from a bucket:

      $ ./s3-crypto.ksh -S -m get -b test-bucket-a \
      -l ./new_file -r cloudfile


To decrypt a file after retreiving it from a bucket (OpenSSL w/key generated from passphrase):

      $ ./s3-crypto.ksh -c -m get -b test-bucket-a \
      -l ./new_file -r cloudfile
      enter aes-256-cbc decryption password:


To decrypt a file after retreiving it from a bucket (OpenSSL w/user-supplied key file):

      $ ./s3-crypto.ksh -c -m get -b test-bucket-a \
      -k ./my_key -l ./new_file -r cloudfile


To decrypt a file after retreiving it from a bucket (Solaris w/key generated from passphrase):

      $ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
      -l ./new_file -r cloudfile
      Enter passphrase: 


To decrypt a file after retreiving it from a bucket (Solaris w/user-supplied key file):

      $ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
      -k ./my_key -l ./new_file -r cloudfile


To decrypt a file after retreiving it from a bucket (Solaris w/user-supplied key label):

      $ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
      -K my_key_label -l ./new_file -r cloudfile
      Enter PIN for Sun Software PKCS#11 softtoken  : 


To decompress a file after retrieving it from a bucket:

      $ ./s3-crypto.ksh -C -m get -b test-bucket-a \
      -l ./new_file -r cloudfile


To reassemble, decrypt and decompress a file after retrieving it from a bucket (OpenSSL w/key generated from passphrase):

      $ ./s3-crypto.ksh -C -c -S -m get -b test-bucket-a \
      -l ./new_file -r cloudfile


To remove a file from a specified bucket:

      $ ./s3-crypto.ksh -m rm -b test-bucket-a -r cloudfile


To remove all files from a specified bucket:

      $ ./s3-crypto.ksh -m rmrf -b test-bucket-a 


To remove a bucket:

      $ ./s3-crypto.ksh -m rm -b test-bucket-a
  • Mysql
  • Glassfish
  • Jruby
  • Rails
  • Nblogo
Terms of Use; Privacy Policy;
© 2014, Oracle Corporation and/or its affiliates
(revision 20160708.bf2ac18)
 
 
Close
loading
Please Confirm
Close