BACKGROUND
Many of the tools made available to store and retrieve files using Cloud Storage services such as Amazon S3 assume that the content will be stored "as is". There are many scenarios where it would be better to first encrypt the content before storing into the Cloud (and automatically decrypt it upon retrieval).
The goal of this software project is to provide this necessary function, namely to provide front-end encryption and decryption capabilities to augment existing tools that can already store and retrieve files in the Cloud. In addition, this tool will provide useful secondary functions such as compression and support for splitting/merging files that may be larger than a given threshold. Strong security algorithms and providers will be used, and this tool will support a variety of encryption key types (as supported by the underlying providers).
VERSIONS
- v0.1 - Initial release supporting Amazon S3.
- v0.2 - Added support for Sun Cloud.
- v0.3 - Added support for the Cloud Safety Box (CSB) simplified use command.
- v0.4 - Added support for compression, file splitting, and key labels (Solaris provider only).
FUNCTIONALITY
This tool supports the following modes of operation:
* encryption * compression * splitting
in addition to any commands that can be passed through to the back-end functionality (e.g., list, remove, etc.) For "put" operations, compression is done before encryption and splitting is done last (if needed). The reverse process is used for "get" operations.
The cryptographic operations performed by these tools are enabled by OpenSSL (or the Solaris Cryptographic Framework on Solaris 10 or OpenSolaris). By default, OpenSSL is used as it enables the greatest level of portability. To use the Solaris cryptographic operations, use the "-p solaris" command line option. Note that on platforms using the UltraSPARC T2 (Niagara 2) processor, these cryptographic operations can be hardware accelerated.
All of the command line options as well as common use cases are available from the tool's usage message available using the "-h" command line option to the s3-crypto.ksh command. Note that in addition, the Cloud Safety Box, csb, command is also provided to enable a simple, easy to use interface at the expense of some measure of flexibility. If you want more control than what csb provides, simply use the s3-crypto.ksh script directly.
FUNCTIONAL DIAGRAM
DEPENDENCIES
These software modules are just front-end components and must be used with a back-end component that performs the actual operations against the Storage Cloud. To date, this software has been tested against both the Sun Cloud Storage Service and the Amazon Simple Storage Service (S3) using the "Another S3 Bash Interface" tool published by "nescafe5" at:
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1081
Thanks to "nescafe5" for posting this great tool. It is expected that this software can be easily adapted to other CLI back-end software components, however. To use with the Sun Cloud, this script must be modified to access object.storage.network.com in place of s3.amazonaws.com.
Note: By default, the "Another S3 Bash Interface" tool is configured to use HTTP not HTTPS. It is strongly recommended that HTTPS be used wherever possible. To enable HTTPS, change the curl command entry from http to https. Also, one of the curl SSL certificate verification steps must also be performed.
Note: By default, the s3-crypto.sh and csb programs will attempt to call the "Another S3 Bash Interface" command as s3. If this functionality was saved under a different name, simply use the S3C_CLI_CMD_NAME environment variable, setting its value to be the name of the program to be executed. For example:
$ export S3C_CLI_CMD_NAME="s3-suncloud"
DOWNLOAD
Currently, this software is accessible from a Mercurial source code repository and a (tar) bundle.
CSB USAGE
To create a new storage bucket:
csb put bucket
To remove an empty storage bucket:
csb rm bucket
To display a listing of storage buckets:
csb buckets
To display the contents of a specified bucket:
csb ls bucket
To put a file into a specified bucket:
csb put bucket local_file [remote_file]
To get a file from a specified bucket:
csb get bucket remote_file [local_file]
To remove a file from a specified bucket:
csb rm bucket remote_file
To remove all files from a specified bucket:
csb rmrf bucket
CSB EXAMPLES
To display a listing of storage buckets:
$ ./csb buckets
test-bucket-a
test-bucket-b
test-bucket-c
To display the contents of a bucket:
$ ./csb ls test-bucket-a
test-file-1
test-file-2
test-file-3
To compress, encrypt and put a file (split if necessary) into a bucket:
$ ./csb put test-bucket-a /export/myfile myfile
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
To get (reassembling if necessary), decrypt and decompress a file from a bucket:
$ ./csb get test-bucket-a myfile ./new_myfile
enter aes-256-cbc decryption password:
To remove a file from a specified bucket:
$ ./csb rm test-bucket-a myfile
To remove all files from a specified bucket:
$ ./csb rmrf test-bucket-a
S3-CRYPTO.SH USAGE
To generate a new encryption/decryption key:
./s3-crypto.ksh -m genkey -k key_file [-s key_size]
To display a listing of storage buckets:
./s3-crypto.ksh -m buckets
To display the contents of a specified bucket:
./s3-crypto.ksh -m ls -b bucket
To put a file into a specified bucket:
./s3-crypto.ksh -m put -b bucket -l local_file -r remote_file
To compress and put a file into a specified bucket:
./s3-crypto.ksh -C -m put -b bucket -l local_file -r remote_file
To encrypt and put a file into a specified bucket:
./s3-crypto.ksh -c [ [-a enc_alg] [-p crypto_provider] [-k key_file |-K key_label] ]
-m put -b bucket -l local_file -r remote_file
To get a file from a specified bucket:
./s3-crypto.ksh -m get -b bucket -r remote_file [-l local_file]
To get and decompress a file from a specified bucket:
./s3-crypto.ksh -C -m get -b bucket -r remote_file [-l local_file]
To get and decrypt a file from a specified bucket:
./s3-crypto.ksh -c [ [-a enc_alg] [-p crypto_provider] [-k key_file | -K key_label] ] -m get -b bucket
-r remote_file [-l local_file]
To remove a file from a specified bucket:
./s3-crypto.ksh -m rm -b bucket -r remote_file
To remove all files from a specified bucket:
./s3-crypto.ksh -m rmrf -b bucket
Specific options:
-a. The name of the encryption algorithm to be used. The default is aes (Solaris encryption) and aes-256-cbc for OpenSSL. Valid names are those defined by the encrypt(1) and openssl(5) commands.
-b. The name of the bucket on the Storage Cloud.
-c. Enable encryption (put) / decryption (get).
-C. Enable compression (put) / decompression (get).
-h. Display this message.
-k. The name of the key file used only for cryptographic operations when the -c option is selected. If this parameter is not specified, then the program will prompt the user for a passphrase to be used for encrypt/decrypt operations.
-K. (Solaris cryptographic provider only). Specify the key label of a symmetric token key in a PKCS#11 token.
-l. The name (path) to the local file.
-L. The maximum file size limit (in Kbytes) used to determine if an input file should be split into chunks for upload.
-m. The mode of operation. Values include:
* buckets. list (your) Storage Cloud buckets
* genkey. generate a key (file) for crypto operations
* get. get a file from the Storage Cloud
* ls. list the contents of a Storage Cloud bucket
* put. put a file into the Storage Cloud
* rm. remove a file from a Storage Cloud bucket
* rmrf. remove all files from a Storage Cloud bucket
-p. The cryptographic services provider, currently either "openssl" (default) or "solaris".
-r. The name of the remote file.
-s. The size of the key file to be generated (in bytes). This parameter is only used with the genkey command mode.
-S. Split the file into chunks if its size is greater than 4 GB (default) or the size specified by the "-L" (size limit) option.
S3-CRYPTO.SH EXAMPLES
These examples are based upon the use of this tool with the "Another S3 Bash Interface" tool discussed above.
To generate a encryption/decryption key
$ ./s3-crypto.ksh -m genkey -k ./my_key -s 32
To display a listing of storage buckets:
$ ./s3-crypto.ksh -m buckets
test-bucket-a
test-bucket-b
test-bucket-c
To display the contents of a bucket:
$ ./s3-crypto.ksh -m ls -b test-bucket-a
test-file-1
test-file-2
test-file-3
To create a new bucket:
$ ./s3-crypto.ksh -m put -b new-bucket
To compress a file before storing it in a bucket:
$ ./s3-crypto.ksh -C -m put -b test-bucket-a \
-l ./myfile -r cloudfile
To encrypt a file before storing it in a bucket (OpenSSL w/key generated from passphrase):
$ ./s3-crypto.ksh -c -m put -b test-bucket-a \
-l ./myfile -r cloudfile
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
To encrypt a file before storing it in a bucket (OpenSSL w/user-supplied key file):
$ ./s3-crypto.ksh -c -m put -b test-bucket-a \
-k ./my_key -l ./myfile -r cloudfile
To encrypt a file before storing it in a bucket (Solaris w/key generated from passphrase):
$ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
-l ./myfile -r cloudfile
Enter passphrase:
Re-enter passphrase:
To encrypt a file before storing it in a bucket (Solaris w/user-supplied key file):
$ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
-k ./my_key -l ./myfile -r cloudfile
To encrypt a file before storing it in a bucket (Solaris w/user-supplied key label):
$ ./s3-crypto.ksh -c -p solaris -m put -b test-bucket-a \
-K my_key_label -l ./myfile -r cloudfile
Enter PIN for Sun Software PKCS#11 softtoken :
To split a file before storing it in a bucket:
$ ./s3-crypto.ksh -S -m put -b test-bucket-a \
-l ./myfile -r cloudfile
To compress, encrypt and split a file before storing it in a bucket (OpenSSL w/key generated from passphrase):
$ ./s3-crypto.ksh -C -c -S -m put -b test-bucket-a \
-l ./myfile -r cloudfile
To reassemble a file after retrieving it from a bucket:
$ ./s3-crypto.ksh -S -m get -b test-bucket-a \
-l ./new_file -r cloudfile
To decrypt a file after retreiving it from a bucket (OpenSSL w/key generated from passphrase):
$ ./s3-crypto.ksh -c -m get -b test-bucket-a \
-l ./new_file -r cloudfile
enter aes-256-cbc decryption password:
To decrypt a file after retreiving it from a bucket (OpenSSL w/user-supplied key file):
$ ./s3-crypto.ksh -c -m get -b test-bucket-a \
-k ./my_key -l ./new_file -r cloudfile
To decrypt a file after retreiving it from a bucket (Solaris w/key generated from passphrase):
$ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
-l ./new_file -r cloudfile
Enter passphrase:
To decrypt a file after retreiving it from a bucket (Solaris w/user-supplied key file):
$ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
-k ./my_key -l ./new_file -r cloudfile
To decrypt a file after retreiving it from a bucket (Solaris w/user-supplied key label):
$ ./s3-crypto.ksh -c -p solaris -m get -b test-bucket-a \
-K my_key_label -l ./new_file -r cloudfile
Enter PIN for Sun Software PKCS#11 softtoken :
To decompress a file after retrieving it from a bucket:
$ ./s3-crypto.ksh -C -m get -b test-bucket-a \
-l ./new_file -r cloudfile
To reassemble, decrypt and decompress a file after retrieving it from a bucket (OpenSSL w/key generated from passphrase):
$ ./s3-crypto.ksh -C -c -S -m get -b test-bucket-a \
-l ./new_file -r cloudfile
To remove a file from a specified bucket:
$ ./s3-crypto.ksh -m rm -b test-bucket-a -r cloudfile
To remove all files from a specified bucket:
$ ./s3-crypto.ksh -m rmrf -b test-bucket-a
To remove a bucket:
$ ./s3-crypto.ksh -m rm -b test-bucket-a
