Graphical User Interface (GUI) was introduced to reduce the use of command inputs. Why we still need to use Command Line Interface (CLI)? There are many reasons. One of them is that CLI can achieve certain outcomes that GUI may not be able to (easily) achieve, e.g. bulk operations. We will take a look at how to use AWS CLI to do bulk operations in this article.
Format
A typical format of an AWS CLI command is like below:
aws <service name> <action> <parameter>
- The aws prefix is to distinguish AWS commands from the native OS commands, e.g. PowerShell commands
- The <service name> part is to tell which service the command should act upon
- The <action> part is to tell what the command should be doing
- The <parameter> part is about how the command should be conducted
For example, an S3 bucket listing command is like aws s3 ls s3://mybucket --recursive.
Command Helper
We will see the actual bulk operation commands examples in the later section. But when there are no example to follow, questions like “what parameter should I use” will arise. There are a few ways to address it:
- Use the auto prompt option, e.g.
- aws --cli-auto-prompt
- aws <service name> --cli-auto-prompt
- aws <service name> <action> --cli-auto-prompt
- Use help, e.g.
- aws help
- aws <service name> help
- aws <service name> <action> help
- Use AWS CLI Command Reference
Output
AWS CLI supports various output formats: json, yaml, yaml-stream, text and table.
To specify the output of a particular command, you can add --output at the end of your command, e.g. aws ec2 describe-regions --output text. If you want to set a format as the default format, you can put output=<format name> in your config file. For details of output configuration, see AWS CLI Commands – Output Format.
Bulk Operation Commands
Note that in the command examples below, the parts highlighted in red need to be replaced with the actual value.
| Service | Action | Command Example |
|---|---|---|
| S3 | Bulk and parallel copy files from one bucket to another | aws s3 cp s3://bucket_name_1/ s3://bucket_name_2 --recursive --exclude "*" --include "keyword1*" --include "keyword2*" and execute another command as aws s3 cp s3://bucket_name_1/ s3://bucket_name_2 --recursive --exclude "*" --include "keyword3*" --include "keyword4*" |
| S3 | Bulk upload local files (in the current folder) to an S3 bucket | aws s3 sync . s3://bucket_name |
| S3 | Bulk download S3 bucket files to the current local folder | aws s3 sync s3://bucket_name . |
| S3 | Bulk rename files within a bucket | aws s3api list-objects --bucket bucket_name --prefix "identifying_keyword" --delimiter "|" --output text| ForEach-Object { $_.split("`t")[2] } | Select-String -Pattern tobe_replaced_keyword | ForEach-Object -Process {$outputFile = $_ -replace 'tobe_replaced_keyword', 'replacing_keyword'; aws s3 mv s3://bucket_name/$_ s3://bucket_name/$outputFile} if you are using SSO, --profile profile_name need to be added after each aws command |
| S3 | List the size of a bucket and number of items stored in it | aws s3api list-objects --bucket bucket_name --output json --query "[sum(Contents[].Size), length(Contents[])]" or aws s3 ls --summarize --human-readable --recursive s3://bucket_name |
| EC2 | Bulk add multiple tags to multiple ec2 instances | aws ec2 create-tags --resources instance_id_1 instance_id_2 --tags 'Key=\"key1_name\", Value=\"key1_value\"' 'Key=\"key2_name\", Value=\"key2_value\"' |
| EC2 | Bulk start ec2 instances | aws ec2 start-instances --instance-ids instance_id_1 instance_id_2 |
| EC2 | Bulk stop ec2 instances | aws ec2 stop-instances --instance-ids instance_id_1 instance_id_2 |
| EC2 | Display all ec2 images’ name filtered by owner and platform | aws ec2 describe-images --owners amazon --filters "Name=platform, Values=windows" --query "Images[*].[Name]" --output text |
| EC2 | Display all ec2 instances that have been stopped and reason for the stop | aws ec2 describe-instances --filters Name=instance-state-name,Values=stopped --region ap-southeast-2 --query "Reservations[].Instances[].[StateReason.Message]" |
| Cloud Formation | Specify the resource(s) to retain when stack is in the DELETE_FAILED state | aws cloudformation delete-stack --stack-name stack_name --retain-resources "resource_logic_id_1" "resource_logic_id_2" |
| IAM | List users by ARN | aws iam list-users --query "Users[].[Arn]" |
Reference
AWS CLI Command Line Interface
