How to copy a file to or from Amazon S3 and your EC2 instance By Raj Gupta

Option 1:- By using GET or wget

 The wget utility is an HTTP and FTP client that allows you to download public objects from Amazon S3. It is installed by default in Amazon Linux and most other distributions, and available for download on Windows. To download an Amazon S3 object, use the following command, substituting the URL of the object to download.

[root@ip-172-31-33-189 ~]# wget https://s3.amazonaws.com/raj05212019/pic1.png

[root@ip-172-31-33-189 ~]# ls

pic1.png

So one file pic1.png downloaded from s3 bucket to our EC2 server.

This method requires that the object you request is public; if the object is not public, you receive an "ERROR 403: Forbidden" message. If you receive this error, open the Amazon S3 console and change the permissions of the object to public

Option 2:-  By using AWS Command Line

For this we required a system in which AWS CLI tool already install.

Use the following command to copy an object from Amazon S3 to your instance by using AWS CLI

[root@ip-172-31-33-189 ~]# aws s3 cp s3://raj05212019/pic1.png pic2.png

download: s3://raj05212019/pic1.png to ./pic2.png

[root@ip-172-31-33-189 ~]# ls

pic1.png  pic2.png

Use the following command to copy an object from your instance back into Amazon S3. 

[root@ip-172-31-33-189 ~]# aws s3 cp pic2.png s3://raj05212019/pic2.png

upload: ./pic2.png to s3://raj05212019/pic2.png

Use the following command to download an entire Amazon S3 bucket to a local directory on your EC2 server.

[root@ip-172-31-33-189 rahul]# aws s3 sync s3://raj05212019 /root/rahul

download: s3://raj05212019/ramu.txt to ./ramu.txt

download: s3://raj05212019/pic1.png to ./pic1.png

download: s3://raj05212019/pic2.png to ./pic2.png

[root@ip-172-31-33-189 rahul]# ls

pic1.png  pic2.png  ramu.txt

The aws s3 sync command can synchronize an entire Amazon S3 bucket to a local directory location. This can be helpful for downloading a data set and keeping the local copy up-to-date with the remote set. If you have the proper permissions on the Amazon S3 bucket, 

you can push your local directory back up to the cloud when you are finished by reversing the source and destination locations in the command.

[root@ip-172-31-33-189 rahul]# aws s3 sync /root/rahul s3://raj05212019 

How to make all Objects in AWS S3 bucket public by default by Raj Gupta

We can use the AWS Policy Generator to generate a bucket policy for our bucket.
Select the option as per below then click on "Add Statement"

The above example allows (Effect: Allow) anyone (Principal: *) to access (Action: s3:GetObject) any object in the bucket (Resource: arn:aws:s3:::<bucket-name>/*).


Then select "Generate Policy"

Then We will get policy like below
this policy will allow anyone to read every object in our S3 bucket(raj03282019) (just replace <bucket-name> with the name of your bucket):

{
  "Id": "Policy1553770384193",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1553769495927",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::raj03282019/*",
      "Principal": "*"
    }
  ]
}


Now go to your AWS S3 console, At the bucket level(raj03282019), click on Permissions, then Select Bucket Policy. Paste the above generated code into the editor and hit save.

Now all your items in the bucket(raj03282019 in my case) will be public by default.

How to download an entire S3 bucket to your local system by Raj Gupta

We've basically many options to do that, but the best one is using AWS CLI

• Step 1

Download and install AWS CLI in your local machine as per operating system

In my case local operating system is Linux

[root@ip-172-31-88-124 ~]# sudo pip install --upgrade awscli

If you are using widow system then follow the below link

Install the AWS CLI using the MSI Installer (Windows)

• Step 2

Configure AWS CLI

[root@ip-172-31-88-124 ~]# aws configure

AWS Access Key ID [None]: AKIAJCJBWU2IWDKGCAMA    ----Enter your Access Key

AWS Secret Access Key [None]: sp6F/OOod2stGLmWi22hPTWDZ1dYLzWq/S2mMliu  ---Enter your Secret Access Key

Default region name [None]:   -----Press enter for default value 

Default output format [None]:  ----------Press enter for default value 

• Step 3

In my S3 bucket(raj03272019) 2 files(pic1.png  pic2.png) are there and all files are going to download in  my local system at same time.

• Step 4

Sync s3 bucket with following command

[root@ip-172-31-88-124 ~]# ls     

[root@ip-172-31-88-124 ~]#  aws s3 sync s3://raj03272019 /root

download: s3://raj03272019/pic1.png to ./pic1.png

download: s3://raj03272019/pic2.png to ./pic2.png

[root@ip-172-31-88-124 ~]# ls

pic1.png  pic2.png           ------both pic are downloaded to local system

s3://raj03272019 >> your s3 bucket that you want to download

/root >> path in your local system where you want to download all the files in my case I have downloaded all file under root but you can download any where you want.

How to Create Presigned URL for S3 by Raj Gupta

Create a role in which give EC2 full access to S3 then assign that role to EC2 during creation of EC2

[root@ip-172-31-82-249 ~]# sudo pip install --upgrade awscli ---this command is used to install awscli

[root@ip-172-31-82-249 ~]# aws s3 ls –this command is used to check all bucket

[root@ip-172-31-82-249 ~]# aws s3 mb s3://raj1987  -- this command is used to create a bucket

[root@ip-172-31-82-249 ~]# echo "Raj" > raj.txt   --this command is used to create a file and in this file copy the output of echo command

[root@ip-172-31-82-249 ~]# aws s3 cp raj.txt s3://raj1987 –this command to copy file from ec2 to s3

[root@ip-172-31-82-249 ~]# aws s3 ls s3://raj1987  --this command to check file copy or not

[root@ip-172-31-82-249 ~]# aws s3 presign s3://raj1987/raj.txt --expires-in 300

This command will give below presign URL to access private object for 300s

 

https://raj1987.s3.amazonaws.com/raj.txt?AWSAccessKeyId=ASIAWSYHFCM35HQSWCTB&Exp                                                                                        ires=1551863679&x-amz-security-token=FQoGZXIvYXdzEHIaDAN4uVeuOZupqTsfkyK3Aw3myN3                                                                                        8nhQGgiH4ooe8NGR7ro0ehNE6ciF5nljp2OBDD7TLDRO08yVdKflNUwYVQadWTVSjHb%2BKWpvLJFHLn                                                                                        IGXeWAsiq2xGmCh%2Fhw%2FO2e8IaTOP6w8%2BShhk6IM2em69YEYzAR8nAWA5%2BntOaSWFnwgzocjw                                                                                        VKOOPUGxVtFVRMWqN2nMJ9vD78fgFEy5Tb5jnHJSNqvr8HkRGdLRLbbi2UFFrdE23DlSJIcWecbEmVJr                                                                                        ClYn02b1SQ%2FNISqRU4W0F%2BAzF3wzGrIExRLaUgDbaP4ui%2FDOSE3Wxti9JlE2Zmjja8Nt07ezHt                                                                                        deAHFNiA13WidzMnv1zjx5RyBXqoEUKm86Kq9CYmYkiYPO1D%2B%2BldZ1ek18J0ucZwj%2Bm9NCJ5x4                                                                                        FtDzIE96D8Fg6kBJE%2FFCCejrrrR9euEZMPreipDTK%2B6c4WN6zC8Loaek0RRbBrB6cOmtqRS95VYs                                                                                        hlmYVGGvCHGd8SNG7NTbAuZTp%2FILiq%2BO1MLOOsPfJJGNC88tXcKYKHTG%2BhKj%2FmA6ec5y5jZv                                                                                        C5yp6o2b2mLbcz2N27ArPJ3hgiqsAhn%2FQHsZLsRlecbyXuRs5rgP%2Fkov5j%2B4wU%3D&Signatur                                                                                        e=L1oQoh%2Bs5OjCBfegEEU5fn48i8g%3D