Mount S3 Bucket on CentOS/RHEL and Ubuntu using S3FS

Sharing is caring!

<div dir&equals;"ltr" style&equals;"text-align&colon; left&semi;">&NewLine;<h3 itemprop&equals;"name" style&equals;"background-color&colon; white&semi; color&colon; &num;666666&semi; font-family&colon; &quot&semi;Trebuchet MS&quot&semi;&comma; Trebuchet&comma; Verdana&comma; sans-serif&semi; font-size&colon; 22px&semi; font-stretch&colon; normal&semi; font-weight&colon; normal&semi; line-height&colon; normal&semi; margin&colon; 0&period;75em 0px 0px&semi; position&colon; relative&semi;">Mount S3 Bucket on CentOS&sol;RHEL and Ubuntu using S3FS<&sol;h3>&NewLine;<div style&equals;"background-color&colon; white&semi; color&colon; &num;666666&semi; font-family&colon; &quot&semi;Trebuchet MS&quot&semi;&comma; Trebuchet&comma; Verdana&comma; sans-serif&semi; font-size&colon; 10&period;8px&semi; line-height&colon; 1&period;6&semi; margin&colon; 0px 0px 1&period;5em&semi;">&NewLine;<div><&sol;div>&NewLine;<&sol;div>&NewLine;<div itemprop&equals;"articleBody" style&equals;"background-color&colon; white&semi; color&colon; &num;666666&semi; font-family&colon; &quot&semi;Trebuchet MS&quot&semi;&comma; Trebuchet&comma; Verdana&comma; sans-serif&semi; font-size&colon; 13&period;2px&semi; line-height&colon; 1&period;4&semi; position&colon; relative&semi; width&colon; 776px&semi;">&NewLine;<div dir&equals;"ltr">&NewLine;<div style&equals;"border&colon; 0px&semi; color&colon; &num;444444&semi; font-family&colon; Ubuntu&comma; Helvetica&comma; Arial&comma; sans-serif&semi; font-size&colon; 14px&semi; line-height&colon; 1&period;71429&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">S3FS is FUSE &lpar;File System in User Space&rpar; based solution to mount an Amazon S3 buckets&comma; We can use system commands with this drive just like as another Hard Disk in system&period; On s3fs mounted files systems we can simply use cp&comma; mv and ls the basic Unix commands similar to run on locally attached disks&period;<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">This article will help you to install S3FS and Fuse by compiling from source&comma; and also help you to mount S3 bucket on your CentOS&sol;RHEL and Ubuntu systems&period;<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><strong style&equals;"border&colon; 0px&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Step 1&colon; Remove Existing Packages<&sol;strong><&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">First check if you have any existing s3fs or fuse package installed on your system&period; If installed it already remove it to avoid any file conflicts&period;<&sol;div>&NewLine;<blockquote style&equals;"background&colon; rgb&lpar;250&comma; 250&comma; 250&rpar;&semi; border-bottom-width&colon; 0px&semi; border-left&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-radius&colon; 14px&semi; border-right&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-top-width&colon; 0px&semi; font-style&colon; italic&semi; margin&colon; 0px 0px 24px&semi; padding&colon; 24px&semi; quotes&colon; none&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">CentOS&sol;RHEL Users&colon;<br &sol;>&num; yum remove fuse fuse-s3fs<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Ubuntu Users&colon;<br &sol;>&dollar; sudo apt-get remove fuse<&sol;div>&NewLine;<&sol;blockquote>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><strong style&equals;"border&colon; 0px&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Step 2&colon; Install Required Packages<&sol;strong><&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">After removing above packages&period; First we will install all dependencies for fuse and s3cmd&period; Install the required packages to system use following command&period;<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">CentOS&sol;RHEL Users&colon;<br &sol;>&num; yum install gcc libstdc&plus;&plus;-devel gcc-c&plus;&plus; curl-devel libxml2-devel openssl-devel mailcap<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Ubuntu Users&colon;<br &sol;>&dollar; sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support<br &sol;>Step 3&colon; Download and Compile Latest Fuse<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Download and compile latest version of fuse source code&period; For this article we are using fuse version 2&period;9&period;3&period; Following set of command will compile fuse and add fuse module in kernel&period;<&sol;div>&NewLine;<blockquote style&equals;"background&colon; rgb&lpar;250&comma; 250&comma; 250&rpar;&semi; border-bottom-width&colon; 0px&semi; border-left&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-radius&colon; 14px&semi; border-right&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-top-width&colon; 0px&semi; font-style&colon; italic&semi; margin&colon; 0px 0px 24px&semi; padding&colon; 24px&semi; quotes&colon; none&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&num; cd &sol;usr&sol;src&sol;<br &sol;>&num; wget http&colon;&sol;&sol;downloads&period;sourceforge&period;net&sol;project&sol;fuse&sol;fuse-2&period;X&sol;2&period;9&period;3&sol;fuse-2&period;9&period;3&period;tar&period;gz<br &sol;>&num; tar xzf fuse-2&period;9&period;3&period;tar&period;gz<br &sol;>&num; cd fuse-2&period;9&period;3<br &sol;>&num; &period;&sol;configure –prefix&equals;&sol;usr&sol;local<br &sol;>&num; make &amp&semi;&amp&semi; make install<br &sol;>&num; export PKG&lowbar;CONFIG&lowbar;PATH&equals;&sol;usr&sol;local&sol;lib&sol;pkgconfig<br &sol;>&num; ldconfig<br &sol;>&num; modprobe fuse<&sol;div>&NewLine;<&sol;blockquote>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><strong style&equals;"border&colon; 0px&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Step 4&colon; Download and Compile Latest S3FS<&sol;strong><&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Download and compile latest version of s3fs source code&period; For this article we are using s3fs version 1&period;74&period; After downloading extract the archive and compile source code in system&period;<&sol;div>&NewLine;<blockquote style&equals;"background&colon; rgb&lpar;250&comma; 250&comma; 250&rpar;&semi; border-bottom-width&colon; 0px&semi; border-left&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-radius&colon; 14px&semi; border-right&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-top-width&colon; 0px&semi; font-style&colon; italic&semi; margin&colon; 0px 0px 24px&semi; padding&colon; 24px&semi; quotes&colon; none&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&num; cd &sol;usr&sol;src&sol;<br &sol;>&num; wget https&colon;&sol;&sol;s3fs&period;googlecode&period;com&sol;files&sol;s3fs-1&period;74&period;tar&period;gz<br &sol;>&num; tar xzf s3fs-1&period;74&period;tar&period;gz<br &sol;>&num; cd s3fs-1&period;74<br &sol;>&num; &period;&sol;configure –prefix&equals;&sol;usr&sol;local<br &sol;>&num; make &amp&semi;&amp&semi; make install<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><&sol;div>&NewLine;<&sol;blockquote>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><strong style&equals;"border&colon; 0px&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Step 5&colon; Setup Access Key<&sol;strong><&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Also In order to configure s3fs we would required Access Key and Secret Key of your S3 Amazon account&period; Get these security keys from Here&period;<&sol;div>&NewLine;<blockquote style&equals;"background&colon; rgb&lpar;250&comma; 250&comma; 250&rpar;&semi; border-bottom-width&colon; 0px&semi; border-left&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-radius&colon; 14px&semi; border-right&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-top-width&colon; 0px&semi; font-style&colon; italic&semi; margin&colon; 0px 0px 24px&semi; padding&colon; 24px&semi; quotes&colon; none&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&num; echo AWS&lowbar;ACCESS&lowbar;KEY&lowbar;ID&colon;AWS&lowbar;SECRET&lowbar;ACCESS&lowbar;KEY &gt&semi; ~&sol;&period;passwd-s3fs<br &sol;>&num; chmod 600 ~&sol;&period;passwd-s3fs<br &sol;>Note&colon; Change AWS&lowbar;ACCESS&lowbar;KEY&lowbar;ID and AWS&lowbar;SECRET&lowbar;ACCESS&lowbar;KEY with your actual key values&period;<&sol;div>&NewLine;<&sol;blockquote>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;"><strong style&equals;"border&colon; 0px&semi; margin&colon; 0px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Step 6&colon; Mount S3 Bucket<&sol;strong><&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">Finally mount your s3 bucket using following set of commands&period; For this example&comma; we are using s3 bucket name as mydbbackup and mount point as &sol;s3mnt&period;<&sol;div>&NewLine;<blockquote style&equals;"background&colon; rgb&lpar;250&comma; 250&comma; 250&rpar;&semi; border-bottom-width&colon; 0px&semi; border-left&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-radius&colon; 14px&semi; border-right&colon; 4px solid rgb&lpar;214&comma; 214&comma; 214&rpar;&semi; border-top-width&colon; 0px&semi; font-style&colon; italic&semi; margin&colon; 0px 0px 24px&semi; padding&colon; 24px&semi; quotes&colon; none&semi; vertical-align&colon; baseline&semi;">&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; margin-bottom&colon; 20px&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&num; mkdir &sol;tmp&sol;cache<br &sol;>&num; mkdir &sol;s3mnt<br &sol;>&num; chmod 777 &sol;tmp&sol;cache &sol;s3mnt<&sol;div>&NewLine;<div style&equals;"border&colon; 0px&semi; line-height&colon; 1&period;71429&semi; padding&colon; 0px&semi; vertical-align&colon; baseline&semi;">&num; s3fs -o use&lowbar;cache&equals;&sol;tmp&sol;cache mydbbackup &sol;s3mnt<&sol;div>&NewLine;<&sol;blockquote>&NewLine;<&sol;div>&NewLine;<&sol;div>&NewLine;<&sol;div>&NewLine;<&sol;div>&NewLine;

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.