<div dir="ltr" style="text-align: left;">Fortunately, We have several tools at our disposal to move data from local disks to Amazon S3</p>
<div style="clear: both; text-align: center;"><a href="http://www.thecloudxperts.co.uk/wp-content/uploads/2017/05/amazon-s3_preview.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="420" data-original-width="960" height="140" src="http://www.thecloudxperts.co.uk/wp-content/uploads/2017/05/amazon-s3_preview-300x131.png" width="320" /></a></div>
<p><b>1. Using the Jets3t Java Library</b><br />JetS3t is an open-source Java toolkit for developers to create powerful yet simple applications to interact with Amazon S3 or Amazon CloudFront. JetS3t provides low-level APIs but also comes with tools that let you work with Amazon S3.</p>
<p>One of the tools provided in the JetS3t toolkit is an application called Synchronize. Synchronize is a command-line application for synchronizing directories on your computer with an Amazon S3 bucket. It is ideal for performing backups or synchronizing files between different computers.<br />One of the benefits of Synchronize is configuration flexibility. Synchronize can be configured to open as many upload threads as possible.</p>
<p><b>To set up Synchronize</b><br />1. Download JetS3Tt from the following URL: http://jets3t.s3.amazonaws.com/downloads.html.<br />2. Unzip jets3t.<br />3. Create a synchronize.properties file and add the following parameters, replacing the values for accesskey and secretkey with your AWS access key identifiers:<br />accesskey=xxx<br />secretkey=yyy<br />upload.transformed-files-batch-size=100<br />httpclient.max-connections=100<br />storage-service.admin-max-thread-count=100<br />storage-service.max-thread-count=10<br />threaded-service.max-thread-count=15<br />4. Run Synchronize using the following command line example:<br />bin/synchronize.sh -k UP somes3bucket/data /data/ &#8211;properties synchronize.properties</p>
<p><b>2. GNU Parallel ;</b><br />GNU parallel is a shell tool that lets you use one or more computers to execute jobs in parallel. GNU parallel runs jobs, which can be a single command or a small script to run for each of the lines in the input. Using GNU parallel, you can parallelize the process of uploading multiple files by opening multiple threads simultaneously. In general, you should open as many parallel upload threads as possible to use most of the available bandwidth. The following is an example of how you can use GNU parallel: 1. Create a list of files that you need to upload to Amazon S3 with their current full path 2. Run GNU parallel with any Amazon S3 upload/download tool and with as many thread as possible using the following command line example: ls | parallel -j0 -N2 s3cmd put {1} s3://somes3bucket/dir1/ The previous example copies the content of the current directly (ls) and runs GNU parallel with two parallel threads (-N2) to Amazon S3 by running the s3cmd command.</p>
<p><b>3. ;Direct-to-S3</b><br />Aspera Direct-to-S3 offers UDP-based file transfer protocol that would transfer large amount of data with fast speed directly to Amazon S3. If you have a large amount of data stored in your local data center and would like to move your data to Amazon S3 for later processing on AWS (Amazon EMR for example), Aspera Direct-To-S3 can help move your data to Amazon S3 faster compared to other protocols such as HTTP, FTP, SSH, or any TCP-based protocol. ;http://cloud.asperasoft.com/big-data-cloud/.</p>
<p><b>4. ;Using AWS Import/Export</b><br />AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage devices for transport.<br /><b>To use AWS Import/Export</b><br />1. Prepare a portable storage device from the list of supported devices. For more information, see Selecting Your Storage Device, http://aws.amazon.com/importexport/#supported_devices.<br />2. Submit a Create Job request to AWS that includes your Amazon S3 bucket, Amazon Elastic Block Store (EBS), or Amazon Glacier region, AWS access key ID, and return shipping address. You will receive back a unique identifier for the job, a digital signature for authenticating your device, and an AWS address to which to ship your storage device.<br />3. Securely identify and authenticate your device. For Amazon S3, place the signature file on the root directory of your device. For Amazon EBS or Amazon Glacier, tape the signature barcode to the exterior of the device.<br />4. Ship your device along with its interface connectors, and power supply to AWS.</div>

