Digital Ocean Object Storage | Spaces
Digital Ocean has started a new services called ‘Object Storage’ which provides users with a means to store their data.
In the Control Panel, there is a new tab for “Spaces”. This enables users to set up and manage data stored on the service.
Early access members are entitled to use Spaces for free till October 31, 2017. There is a limit of 1 TB of stored data.
The pricing of Spaces after that date is not known at present.
At present the Object Storage is only available at the NYC3 datacenter of Digital Ocean.
How To Configure s3cmd 2.x To Manage DigitalOcean Spaces
This guide will show how to install, configure, and manage DigitalOcean Spaces with the command-line tool s3cmd.
Prerequisites
To use this s3cmd, the following are required:
• A working installation of s3cmd version 2.0.0+. Note that package managers may be out of date. To get this version, visit the s3cmd Download page.
• An access key pair for your Spaces. To generate these, visit the API page in the DigitalOcean Control Panel. To learn more about access keys, see the Introduction to DigitalOcean Spaces access key section.
When you have completed the prerequisites, you’re ready to follow along.
Verify the Version
We’ll begin by confirming our version of s3cmd: with the following command:
s3cmd --version
Output:
s3cmd version 2.0.0+
By verifying that we’re running the same version, we can expect that the output of our commands will match the diretions that follow.
Choose the Configuration File
By default, s3cmd stores its configuration file, .s3cfg, in the home directory of the user who runs the configuration command. This is a plain text file of key/value pairs which can be edited directly once it has been created. You can choose between setting up DigitalOcean as the default configuration or creating a different configuration file:
Option 1: Make DigitalOcean the default connection
If DigitalOcean is the main or only provider you’ll connect to with s3cmd, configure it in the default ~/.s3cfg file with the following command:
s3cmd --configure
By choosing this option, you won’t have to specify the configuration file each time you run a command.
Option 2: Create an explicit configuration file for DigitalOcean
If you’re already using s3cmd with another service, you may want to create an alternate configuration file, which you can do by adding the -c flag and supplying a filename. The configuration file will be created in the directory where you issue the command, so specify the path if you want it created elsewhere.
In this example, we’ll place a config file named nyc3in our home directory:
s3cmd --configure -c ~/nyc3
Important: To use this configuration file, it must be explicitly provided at the end of each command by appending -c ~/nyc3.
Configure s3cmd
Whether you use the default configuration file or specify your own, when you issue the configuration command, it will launch an interactive multi-step configuration script. In this section, we’ll go through each of the steps in the configuration process. The places where you should substitute your specific information are surrounded by angle brackets < ... >.
Enter Access Keys
The script begins by asking for an Access Key and Secret Key. If you don’t already have keys, you can generate a set for s3cmd by visiting the Control Panel’s API page.
We’ll enter our keys, then accept US for the Default Region since the region information isn’t relevant to DigitalOcean.
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []:
Secret Key []:
Default Region [US]:
Note: If you prefer, you can use the environment variables AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_ID to store a set of keys.
Set the Endpoint
Next, we’ll enter the early access DigitalOcean endpoint, nyc3.digitaloceanspaces.com
Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com
Since Spaces supports DNS-based buckets, at the next prompt, we’ll supply the bucket value in the required format:
%(bucket)s.nyc3.digitaloceanspaces.com
Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars
can be used if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: %(bucket)s.nyc3.digitaloceanspaces.com
Optional: Set an encryption password
Next, we’re asked to supply an encryption password. Unlike HTTPS, which protects file only while in transit, GPG encryption prevents others from reading files while they are stored on DigitalOcean as well as in transit. Setting a password now won’t cause objects to be automatically encrypted; that’s done by adding a -e flag to the put command when you copy the object to a Space.
We’ll enter a password so it will be available in the event we want to use encryption.:
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
Connect via HTTPS
Next, we’re prompted to connect via HTTPS, which protects data from being read while it is in transit. DigitalOcean Spaces do not support unencrypted transfer, so we’ll press ENTER to accept the default, YES:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
Optional: Set a Proxy Server
If your network requires you to use an HTTP Proxy server, enter its IP address or domain name without the protocol, e.g. 203.0.113.1 or proxy.example.com Since we aren’t using an HTTP Proxy server, we’ll leave this question blank and press ENTER:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:
Confirm and test settings
After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them:
New settings:
Access Key: EXAMPLES7UQOTHDTF3GK4
Secret Key: b8e1ec97b97bff326955375c5example
Default Region: US
S3 Endpoint: nyc3.digitaloceanspaces.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.n
yc3.digitaloceanspaces.com
Encryption password: secure_password
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n]
Save the settings
When the test completes successfully, enter Y to save the settings:
Please wait, attempting to list all buckets…
Success. Your access key and secret key worked fine 🙂
Now verifying that encryption works…
Success. Encryption and decryption worked fine 🙂
Save settings? [y/N]
If the test fails or you choose N you’ll have the opportunity to retry the configuration. Once you save the configuration, you’ll receive confirmation of its location:
Configuration saved to ‘/home/sammy/nyc3’
The test step confirms that we can connect to our DigitalOcean Spaces.
Conclusion
In this tutorial we’ve configured and tested the s3cmd settings.
• For a guide to some of the most common s3cmd tasks, see the How to Manage DigitalOcean Spaces with s3cmd.
• For a complete listing of options and commands, reference the s3cmd usage guide or access the help file from the command line with s3cmd –help.
How to Manage DigitalOcean Spaces with s3cmd
A Cheat Sheet
Introduction
s3cmd is a popular cross-platform command-line tool for managing S3 and S3-compatible object stores. This guide provides a quick reference to commands that are useful for managing DigitalOcean Spaces.
How to Use this Guide:
This guide is in cheat sheet format with self-contained command-line snippets
Jump to any section that is relevant to the task you are trying to complete
Important: If you followed How To Configure s3cmd 2.x To Manage DigitalOcean Spaces and chose to create a separate profile, remember to append -c ~/nyc3 or whatever you named your specific profile to the end of each command.
Create
Create one or more Spaces: mb
Use the command mb, short for “make bucket”, to create a new space:
s3cmd mb s3://spacename s3://secondspace
Create Directories
Object stores use a flat namespace and don’t have directories. What look like directories in the path of an object are actually part of the object’s file name. Graphical interfaces mimic the more familiar experience of directories for their users by creating an empty file for each element that doesn’t already exist.
From the command line, there’s no need to create empty files. Instead, type the path where you want the file to be as part of the put command.
List
List Spaces
Get a list of all the existing Spaces with ls:
s3cmd ls
List Contents of one or more specific Spaces
List all the contents of a one or more specific Spaces, recursively:
s3cmd ls s3://spacename s3://secondspace
List Contents of all Spaces List all the content in every Space:
s3cmd la --recursive
Copy
Copy Objects into a Space: put
Use the put command to copy files from your local machine to a Space. In all of these commands, you must include the trailing slash.
Put one file
When you include the trailing slash, as in the example below, the original file name will be appended. If you omit the slash, then the file will be copied to the space with the new name, path.
s3cmd put file.txt s3://spacename/path/
Put multiple files
This example shows multiple specific files getting transferred to a Space:
s3cmd put file1.txt file2.txt path/to/file3.txt s3://spacename/path/
Put all the files in your current working directory
Using the * with the put command will copy everything in the current working directory, recursively, into your Space:
s3cmd put * s3://spacename/path/ --recursive
Put a file under a new name
You can change the name of a file at the same time you put it in a Space by typing the new name at the end of the path as follows:
s3cmd put file.txt s3://spacename/newname.txt
Copy Files to your Local Machine: get
The command get copies files from a Space to your local machine.
Get one file
s3cmd get s3://spacename/path/to/file.txt
Get all the files in directory
To get multiple files, the s3 address must end with a trailing slash, and the command requires the –recursive flag.
s3cmd get s3://spacename/path/ --recursive
Get a file and save it under a new name
Like the put command, the command get command allows you to give the file a different name.
s3cmd get s3://spacename/file.txt newfilename.txt
Set Permissions
Currently the setacl commands are supported by DigitalOcean Spaces.
Set directory listings on a Space
Note that s3cmd only provides output when the command you issue changes tha access. For example, when you change the ACL from private to public, you’ll see output like s3://spacename/: ACL set to Public. If the ACL is already public, s3cmd will return silently to the command prompt.
Enable directory listings
s3cmd setacl s3://spacename/ --acl-public
Disable directory listings
s3cmd setacl s3://spacename/ --acl-private
Set permissions on file
Using the setacl command, files can be made private so that only someone connecting with a valid key pair will be able to read the file, or public so that anyone can read the file with either an S3 compatible client or via HTTPS.
Note that s3cmd only provides output when the command you issue changes tha access. For example, when you change the ACL from private to public, you’ll see output like s3://spacename/test.txt: ACL set to Public [1 of 1]. If the ACL is already public, s3cmd will return silently to the command prompt.
Make a file public
s3cmd setacl s3://spacename/file.txt --acl-public
Make all the files at a path public, recursively Use the –recursive flag to make multiple files public recursively:
s3cmd setacl s3://spacename/path/to/files/ --acl-public --recursive
Make a file private
s3cmd setacl s3://spacename/file.txt --acl-private
Make all the files at a path private, recursively Use the –recursive flag to make multiple files private recursively:
s3cmd setacl s3://spacename/path/to/files/ --acl-private --recursive
Delete
The s3cmd commands del and rm are identical and can be used interchangeably.
Delete a Space
Use rb short for “remove bucket” to delete an empty Space. If you wish to remove the Space and all its contents, append –recursive. This will permanently remove all the files.
s3cmd rb s3://spacename
Delete a file
s3cmd rm s3://spacename/name/of/file
Delete all files in a Space
Use rm or del with both the –recursive and –force flags to remove all the files in a Space but not the Space itself.
s3cmd rm s3://spacename/ --recursive --force
Encrypt a file
Adding the -e or –encrypt flag when you put a file in a Space with s3cmd will encrypt the file to protect if from being read on the server or in transit. If you download the file using s3cmd and the same configuration file, s3cmd will automatically use the password to decrypt it. Other users would need to use gpg -d file.txt to decrypt it and would have to enter the password you supplied. s3cmd allows you to use only one password, so it’s suitable for you and for other users with full administrative access. You can learn more about gpg’s symmetric encryption in The GNU Privacy Handbook:
s3cmd put s3://path/to/file.txt -e
Conclusion
This guide covered some of the common commands used to manage Spaces with s3cmd version 2.0.0+. There are many other flags and commands that can be used in combination. For a comprehensive guide to what’s available, see the s3cmd usage guide.
Read the official guide at Digital Ocean
