Loops!
Execute it again and again and again....
Loops allow us to take a series of commands and keep re-running them until a particular situation is reached. They are useful for automating repetitive tasks.
For Loop
A ‘for loop’ is a bash programming language statement which allows code to be repeatedly executed. A for loop is classified as an iteration statement i.e. it is the repetition of a process within a bash script. For example, you can run Linux command or task 5 times or read and process list of files using a for loop. A for loop can be used at a shell prompt or within a shell script itself.
for var in <list>
do
<commands>
done
The for loop will take each item in the list (in order, one after the other), assign that item as the value of the variable var, execute the commands between do and done then go back to the top, grab the next item in the list and repeat over. The list is defined as a series of strings, separated by spaces.
For loops iterate for as many arguments given:
Example:
The contents of $Variable is printed three times.
#!/bin/bash
for Variable in {1..3}
do
echo "$Variable"
done
Or write it the "traditional for loop" way:
for ((a=1; a <= 3; a++))
do
echo $a
done
They can also be used to act on files.
This will run the command 'cat' on file1 and file2
for Variable in file1 file2
do
cat "$Variable"
done
or the output from a command
This will cat the output from ls.
for Output in $(ls)
do
cat "$Output"
done
While loop
The bash while loop is a control flow statement that allows code or commands to be executed repeatedly based on a given condition. For example, run echo command 5 times or read text file line by line or evaluate the options passed on the command line for a script.
Syntax:
The syntax is as follows:
while [ condition ]
do
command1
command2
command3
done
command1 to command3 will be executed repeatedly till condition is true. The argument for a while loop can be any Boolean expression. Infinite loops occur when the conditional never evaluates to false. For example, following while loop will print welcome 5 times on screen:
#!/bin/bash
a=1
while [ $a -le 5 ]
do
echo "Number $a"
x=$(( $x + 1 ))
done
Printing list of host IP addresses from hosts file.
Create a file named hosts
$ vi hosts
192.168.1.10
192.168.1.11
192.168.1.12
192.168.1.13
#!/bin/bash
1. for i in `cat hosts`;do
2. echo "Printing list of hosts."
3. echo $i
4. done
Explanation
Line 1. For command substitution we are using backticks `` . It’s different from single quote ‘’ `cat hosts` will return the content of hosts file line by line, which will be stored in variable “i”.
Real time use cases
Bash script to install Apache, MYSQL and PHP for Ubuntu OS.
Colouring your script.
Check below mentioned link for information on colouring your echo output. http://misc.flogisoft.com/bash/tip_colors_and_formatting
The ANSI/VT100 terminals and terminal emulators are not just able to display black and white text ; they can display colours and formatted texts thanks to escape sequences. Those sequences are composed of the Escape character (often represented by ”^[” or ”<Esc>”) followed by some other characters: ”<Esc>[FormatCodem”.
In Bash, the <Esc> character can be obtained with the following syntaxes:
\e
\033
\x1B
#!/bin/bash
#COLORS
# Reset
Color_Off='\033[0m'# Text Reset
# Regular Colors
Red='\033[0;31m'# Red
Green='\033[0;32m'# Green
Yellow='\033[0;33m'# Yellow
Purple='\033[0;35m'# Purple
Cyan='\033[0;36m'# Cyan
# Update packages and Upgrade system
echo -e "$Cyan \n Updating System.. $Color_Off"
sudo apt-get update -y &&sudo apt-get upgrade -y
## Install AMP
echo -e "$Cyan \n Installing Apache2 $Color_Off"
sudo apt-get install apache2 apache2-doc apache2-mpm- prefork apache2-utils libexpat1 ssl- cert -y
echo -e "$Cyan \n Installing PHP &Requirements $Color_Off"
sudo apt-get install libapache2-mod- php5 php5 php5-common php5-curl php5-dev php5-gd
php5-idn php-pear php5-imagick php5-mcrypt php5-mysql php5-ps php5-pspell php5-recode
php5-xsl -y
echo -e "$Cyan \n Installing MySQL $Color_Off"
sudo apt-get install mysql-server mysql-client libmysqlclient15.dev -y
echo -e "$Cyan \n Installing phpMyAdmin $Color_Off"
sudo apt-get install phpmyadmin -y
echo -e "$Cyan \n Verifying installs$Color_Off"
sudo apt-get install apache2 libapache2-mod- php5 php5 mysql-server php-pear php5-mysql
mysql-client mysql-server php5-mysql php5-gd -y
## TWEAKS and Settings
# Permissions echo -e "$Cyan \n Permissions for /var/www $Color_Off"
sudo chown -R www-data:www- data /var/www
echo -e "$Green \n Permissions have been set $Color_Off"
# Enabling Mod Rewrite, required for WordPress permalinks and .htaccess files
echo -e "$Cyan \n Enabling Modules $Color_Off"
sudo a2enmod rewrite
sudo php5enmod mcrypt
# Restart Apache
echo -e "$Cyan \n Restarting Apache $Color_Off"
sudo service apache2 restart
Backup scripts
Being working with systems you may need to take backup of files, directories, log files etc. Below mention scenario shows you how you can automate the backup procedures. In this example, we will create a file and mention the name of log files that needs to be backup up with tar command. Before taking backup, our script will also tell us if the log file exists or not. It will skip the backup procedure if the file does not exists. After all there is no point running backup command if the log file does not exist..
Create directory for storing log files
$ mkdir -p /tmp/scripts/logs
$ mkdir -p /tmp/scripts/logs
$ cd /tmp/scripts/logs
$ touch ansible.log apache.log mysql.log nagios.log
You can choose to put some content in the log files, touch will just create empty files.
$ cd /tmp/scripts
Create a file where you place the name of the files that you want to backup.
$ vi backup_files.txt
apache.log
mysql.log
nagios.log
ansible.log
chef.log
There is one extra filename chef.log which is not present in our logs directory /tmp/scripts/logs.
We will see how we will handle it in our script
$ vi backup.sh
#!/bin/bash
LOG_DIR='/tmp/scripts/logs'
BACKUP_DIR='/tmp/scripts/logs_backup'
mkdir -p $BACKUP_DIR
for i in `cat backup_files.txt`; do
if [ -f $LOG_DIR/$i ];
then
echo "Copying $i to logs_backup directory."
cp $LOG_DIR/$i $BACKUP_DIR
else
echo "$i log file does exist, skipping."
fi
done
echo
echo
echo "Zipping log files"
tar -czvf logs_backup.tgz logs_backup
echo
echo
echo "Backup completed successfully."
Mysql Database Backup Script
mysqldump command is used to take the db dump for mysql. In the script, we’re taking the dbdump and sending it to a target directory in a zipped format. We are also removing 8 days old dbbackup file by using find command. This process is called a purging or purging old backup/log files.
#!/bin/sh
TIME_NOW="$(date +'%d_%m_%Y_%H_%M_%S')"
BACKUPFILE="db_backup_$TIME_NOW".gz
BACKUP_DIR="/opt/db_backup_dir"
PATHOFBACKUPFILE="$BACKUP_DIR/$BACKUPFILE"
LOG_FILE="$BACKUP_DIR/"backup_log_"$(date +'%Y_%m')".txt
echo "mysqldump started at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
mysqldump -- user=dbuser-- password=dbpass -- default-character- set=utf8 mydatabase |
gzip >"$PATHOFBACKUPFILE"
echo "mysqldump finished at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
chown myuser "$PATHOFBACKUPFILE"
chown myuser "$LOG_FILE"
echo "file permission changed">>"$LOG_FILE"
find "$BACKUP_DIR"-name db_backup_* -mtime +8 -exec rm {} \;
echo "old files deleted">>"$LOG_FILE"
echo "operation finished at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
echo "*****************">>"$LOG_FILE"
exit 0
Running command on remote servers/nodes
Sometimes we need to run a command or set of commands on multiple nodes/server. We use ssh to login to these nodes and run that command individually and manually on all the nodes/servers. But it’s a very time consuming and mundane work if you have to do it many times. We will write a bash script to do that.
For this exercise, we will choose to run “yum install httpd” on three nodes. Assumptions:
Three centos VMs
VMs have internet connection to download and install software.
All VMs have same username to connect.
We will create a file named “hosts-dev” and add ip address of all three nodes in that.
$ vi hosts-dev
192.168.2.5
192.168.2.6
192.168.2.7
We will write a script which will read the ip address from the hosts-dev file, do ssh to all of them one by one and run yum install httpd command over ssh.
$ vi install.sh
#!/bin/bash
for hosts in `cat hosts-dev`
do
ssh vagrant@$hosts sudo yum install httpd -y
done
Let’s break it down
Line 2: for loop will run over the content on hosts-dev file one by one, which are the ip addresses of the VMs. Notice we have used backticks `` and not single quotes ‘’ to read the hosts-dev file ( `cat hosts-dev` ).
Line 4: we are running sudo yum install httpd -y command over ssh. $hosts variable will hold the ip address and will establish ssh connection with vagrant user( ssh vagrant@$hosts).
This loop will run until we exhaust all the entries in the hosts-dev file, if you have lets say 50 nodes you can add ip address of all the 50 nodes in this file. Every time it logs into the vm’s/sever/nodes it will ask you a password, it will be painful if you have lot of servers that you manage to enter password manually. In the next section, we will deal with this issue by doing ssh key exchange.
The above script will install httpd package on all the three nodes but will also ask you password everytime it does a ssh login. To avoid this we can do key based login which is discussed in next chapter.
Execute it again and again and again....
Loops allow us to take a series of commands and keep re-running them until a particular situation is reached. They are useful for automating repetitive tasks.
For Loop
A ‘for loop’ is a bash programming language statement which allows code to be repeatedly executed. A for loop is classified as an iteration statement i.e. it is the repetition of a process within a bash script. For example, you can run Linux command or task 5 times or read and process list of files using a for loop. A for loop can be used at a shell prompt or within a shell script itself.
for var in <list>
do
<commands>
done
The for loop will take each item in the list (in order, one after the other), assign that item as the value of the variable var, execute the commands between do and done then go back to the top, grab the next item in the list and repeat over. The list is defined as a series of strings, separated by spaces.
For loops iterate for as many arguments given:
Example:
The contents of $Variable is printed three times.
#!/bin/bash
for Variable in {1..3}
do
echo "$Variable"
done
Or write it the "traditional for loop" way:
for ((a=1; a <= 3; a++))
do
echo $a
done
They can also be used to act on files.
This will run the command 'cat' on file1 and file2
for Variable in file1 file2
do
cat "$Variable"
done
or the output from a command
This will cat the output from ls.
for Output in $(ls)
do
cat "$Output"
done
While loop
The bash while loop is a control flow statement that allows code or commands to be executed repeatedly based on a given condition. For example, run echo command 5 times or read text file line by line or evaluate the options passed on the command line for a script.
Syntax:
The syntax is as follows:
while [ condition ]
do
command1
command2
command3
done
command1 to command3 will be executed repeatedly till condition is true. The argument for a while loop can be any Boolean expression. Infinite loops occur when the conditional never evaluates to false. For example, following while loop will print welcome 5 times on screen:
#!/bin/bash
a=1
while [ $a -le 5 ]
do
echo "Number $a"
x=$(( $x + 1 ))
done
Printing list of host IP addresses from hosts file.
Create a file named hosts
$ vi hosts
192.168.1.10
192.168.1.11
192.168.1.12
192.168.1.13
#!/bin/bash
1. for i in `cat hosts`;do
2. echo "Printing list of hosts."
3. echo $i
4. done
Explanation
Line 1. For command substitution we are using backticks `` . It’s different from single quote ‘’ `cat hosts` will return the content of hosts file line by line, which will be stored in variable “i”.
Real time use cases
Bash script to install Apache, MYSQL and PHP for Ubuntu OS.
Colouring your script.
Check below mentioned link for information on colouring your echo output. http://misc.flogisoft.com/bash/tip_colors_and_formatting
The ANSI/VT100 terminals and terminal emulators are not just able to display black and white text ; they can display colours and formatted texts thanks to escape sequences. Those sequences are composed of the Escape character (often represented by ”^[” or ”<Esc>”) followed by some other characters: ”<Esc>[FormatCodem”.
In Bash, the <Esc> character can be obtained with the following syntaxes:
\e
\033
\x1B
#!/bin/bash
#COLORS
# Reset
Color_Off='\033[0m'# Text Reset
# Regular Colors
Red='\033[0;31m'# Red
Green='\033[0;32m'# Green
Yellow='\033[0;33m'# Yellow
Purple='\033[0;35m'# Purple
Cyan='\033[0;36m'# Cyan
# Update packages and Upgrade system
echo -e "$Cyan \n Updating System.. $Color_Off"
sudo apt-get update -y &&sudo apt-get upgrade -y
## Install AMP
echo -e "$Cyan \n Installing Apache2 $Color_Off"
sudo apt-get install apache2 apache2-doc apache2-mpm- prefork apache2-utils libexpat1 ssl- cert -y
echo -e "$Cyan \n Installing PHP &Requirements $Color_Off"
sudo apt-get install libapache2-mod- php5 php5 php5-common php5-curl php5-dev php5-gd
php5-idn php-pear php5-imagick php5-mcrypt php5-mysql php5-ps php5-pspell php5-recode
php5-xsl -y
echo -e "$Cyan \n Installing MySQL $Color_Off"
sudo apt-get install mysql-server mysql-client libmysqlclient15.dev -y
echo -e "$Cyan \n Installing phpMyAdmin $Color_Off"
sudo apt-get install phpmyadmin -y
echo -e "$Cyan \n Verifying installs$Color_Off"
sudo apt-get install apache2 libapache2-mod- php5 php5 mysql-server php-pear php5-mysql
mysql-client mysql-server php5-mysql php5-gd -y
## TWEAKS and Settings
# Permissions echo -e "$Cyan \n Permissions for /var/www $Color_Off"
sudo chown -R www-data:www- data /var/www
echo -e "$Green \n Permissions have been set $Color_Off"
# Enabling Mod Rewrite, required for WordPress permalinks and .htaccess files
echo -e "$Cyan \n Enabling Modules $Color_Off"
sudo a2enmod rewrite
sudo php5enmod mcrypt
# Restart Apache
echo -e "$Cyan \n Restarting Apache $Color_Off"
sudo service apache2 restart
Backup scripts
Being working with systems you may need to take backup of files, directories, log files etc. Below mention scenario shows you how you can automate the backup procedures. In this example, we will create a file and mention the name of log files that needs to be backup up with tar command. Before taking backup, our script will also tell us if the log file exists or not. It will skip the backup procedure if the file does not exists. After all there is no point running backup command if the log file does not exist..
Create directory for storing log files
$ mkdir -p /tmp/scripts/logs
$ mkdir -p /tmp/scripts/logs
$ cd /tmp/scripts/logs
$ touch ansible.log apache.log mysql.log nagios.log
You can choose to put some content in the log files, touch will just create empty files.
$ cd /tmp/scripts
Create a file where you place the name of the files that you want to backup.
$ vi backup_files.txt
apache.log
mysql.log
nagios.log
ansible.log
chef.log
There is one extra filename chef.log which is not present in our logs directory /tmp/scripts/logs.
We will see how we will handle it in our script
$ vi backup.sh
#!/bin/bash
LOG_DIR='/tmp/scripts/logs'
BACKUP_DIR='/tmp/scripts/logs_backup'
mkdir -p $BACKUP_DIR
for i in `cat backup_files.txt`; do
if [ -f $LOG_DIR/$i ];
then
echo "Copying $i to logs_backup directory."
cp $LOG_DIR/$i $BACKUP_DIR
else
echo "$i log file does exist, skipping."
fi
done
echo
echo
echo "Zipping log files"
tar -czvf logs_backup.tgz logs_backup
echo
echo
echo "Backup completed successfully."
Mysql Database Backup Script
mysqldump command is used to take the db dump for mysql. In the script, we’re taking the dbdump and sending it to a target directory in a zipped format. We are also removing 8 days old dbbackup file by using find command. This process is called a purging or purging old backup/log files.
#!/bin/sh
TIME_NOW="$(date +'%d_%m_%Y_%H_%M_%S')"
BACKUPFILE="db_backup_$TIME_NOW".gz
BACKUP_DIR="/opt/db_backup_dir"
PATHOFBACKUPFILE="$BACKUP_DIR/$BACKUPFILE"
LOG_FILE="$BACKUP_DIR/"backup_log_"$(date +'%Y_%m')".txt
echo "mysqldump started at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
mysqldump -- user=dbuser-- password=dbpass -- default-character- set=utf8 mydatabase |
gzip >"$PATHOFBACKUPFILE"
echo "mysqldump finished at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
chown myuser "$PATHOFBACKUPFILE"
chown myuser "$LOG_FILE"
echo "file permission changed">>"$LOG_FILE"
find "$BACKUP_DIR"-name db_backup_* -mtime +8 -exec rm {} \;
echo "old files deleted">>"$LOG_FILE"
echo "operation finished at $(date +'%d-%m- %Y %H:%M:%S')">>"$LOG_FILE"
echo "*****************">>"$LOG_FILE"
exit 0
Running command on remote servers/nodes
Sometimes we need to run a command or set of commands on multiple nodes/server. We use ssh to login to these nodes and run that command individually and manually on all the nodes/servers. But it’s a very time consuming and mundane work if you have to do it many times. We will write a bash script to do that.
For this exercise, we will choose to run “yum install httpd” on three nodes. Assumptions:
Three centos VMs
VMs have internet connection to download and install software.
All VMs have same username to connect.
We will create a file named “hosts-dev” and add ip address of all three nodes in that.
$ vi hosts-dev
192.168.2.5
192.168.2.6
192.168.2.7
We will write a script which will read the ip address from the hosts-dev file, do ssh to all of them one by one and run yum install httpd command over ssh.
$ vi install.sh
#!/bin/bash
for hosts in `cat hosts-dev`
do
ssh vagrant@$hosts sudo yum install httpd -y
done
Let’s break it down
Line 2: for loop will run over the content on hosts-dev file one by one, which are the ip addresses of the VMs. Notice we have used backticks `` and not single quotes ‘’ to read the hosts-dev file ( `cat hosts-dev` ).
Line 4: we are running sudo yum install httpd -y command over ssh. $hosts variable will hold the ip address and will establish ssh connection with vagrant user( ssh vagrant@$hosts).
This loop will run until we exhaust all the entries in the hosts-dev file, if you have lets say 50 nodes you can add ip address of all the 50 nodes in this file. Every time it logs into the vm’s/sever/nodes it will ask you a password, it will be painful if you have lot of servers that you manage to enter password manually. In the next section, we will deal with this issue by doing ssh key exchange.
The above script will install httpd package on all the three nodes but will also ask you password everytime it does a ssh login. To avoid this we can do key based login which is discussed in next chapter.
Comments
Post a Comment