Wednesday, October 28, 2015

Linux scripting, date functions and renaming

This is something I always shy away from as I view it somewhat like a black art.

Sample regex in bash including case insensitive, and parsing based on a variable

  host=myhostname
# use the ${var,,} syntax to convert to lowercase
  regex="^myProd.*"
  KerberosU=NonProdUser
  [[ ${host,,} =~ $regex ]] && KerberosU=ProdUser

This is a script I wrote to bulk rename a number of files (and perl rename was not available), on newYears day

There are some nice tricks in here

#!/bin/bash
# Usage
# ksh newYearsTasks.sh |DayOfWeek|     Note DayOfWeek is optional and only to be used for testing purposes
# e.g.  ksh newYearsTasks.sh    This is normal behaviour. The script will default to days day of week
# ksh newYearsTasks.sh Mon . This will override actual DayOfWeek to be Monday for testing purposes
# ksh newYearsTasks.sh . If a paramater other than Mon is set, then the DayOfWeek is set NOT to be Monday. This is for forcing testing of this behaviour on Mondays
#  Requirement is to copy filesfrom a number of src dirs to dest dirs, and rename them somewhat along the way
#
#
#################
#Testing Note
#################
# In order to test this we need create files in the archive directories that would be present on new years day.
# Pay attention to the year component. The script calculates the previous year to generate the rename command, so 
#If you are testing this in 2015 make sure you rename the old files to have 2014 as the year component. Like wise if you are testing in 2016, set the old files to have 2015 as the year component.
# The script copyies files based on their age. If the script is run on a Monday it will copy files that are less than 3 days old
# If the script is run on any other day it will try to copy files that are less than 1 day old (based on modifation date)
# Both scenarios shoudl be tested, and for simplicity sake it is recommended to test Mondays behaviour first.
# 1/ Test Mondays
# To test Mondays behaviour you will need to use touch -d command to set some archive files 3 days old. testDate format is yyyyMMdd, 
#     e.g. if running tests Nov 2. We want to set the files to be 3 days old on touch -d 20151030 * in each of the archive directories to be tested
# To manually invoke the script you can call "ksh newYearsTasks.sh Mon"", which will force the script to behave as if it is Monday regardless of day of week.
# 2/ Test other weekday 
#     Use touch *  in each of the archive directories to be tested to update to be less than one day old
# Run this to force testing with NotMonday behaviour ... "ksh newYearsTasks.sh Tue"


# First Check if server is active
LOG_FILE="/$HOME/log/ops/dailyscriptlogs.log"
# Set variables
CurrentDateTime=`date`
DayOfWeek=$(date +%a)
typeset -i Year
Year=$(date +%Y)
OldYear=$(($Year-1))
function Rename {
echo "$CurrentDateTime : newYearsTasks.sh : Copying files dropped in the last $1 day(s) from archive to filedrop " >> $LOG_FILE
        #find files modified in $1 time, and copy them to dest
find . -mtime -$1 -type f  -exec cp {} ../dest \;
cd ../filedrop
echo "$CurrentDateTime : newYearsTasks.sh : Renaming file date from ${OldYear}$2 to ${Year}0101, and from ${2}${OldYear} to 0101${Year} " >> $LOG_FILE
rename ${OldYear}$2 ${Year}0101 *.csv
rename ${2}${OldYear} 0101${Year} *.csv
}

# Can pass in parameter to overrider todays day of week for testing. Set it to Mon for Monday testing, or anyting else for other day testing
if [[ $# -eq 1 ]]; then
if [[ $1 = "Mon" ]]; then
echo "$CurrentDateTime : newYearsTasks.sh : Overriding Day Of week to $1"  >> $LOG_FILE
DayOfWeek=$1
else
echo "$CurrentDateTime : newYearsTasks.sh : Overriding Day Of week to Tue"  >> $LOG_FILE
DayOfWeek="Tue" 
fi
fi

# Script will run on both active and inactive server, to ensure both Prod and DR have correct files

echo "$CurrentDateTime : newYearsTasks.sh : Executing New Years Day script today " >> $LOG_FILE
#
# Three Arrays representing 1/ srcDirs to copy files from
# 2/ findStr used by the find command to select filename for renaming
# 3/ renameREgex, used to rename the files selected by the find
srcDirs[0]=/$HOME/dir1 
findStr[0]="*ext-[0-9][0-9]*.csv"  #findStr represents the variable used in the find
#rename regex causes original value ('p'), and renamed value to be output. Result is piped to mv
renameRegex[0]="p;s/ext-[0-9]*.csv/ext.csv/" # renameRegex represents the sed regex usedTo rename
srcDirs[1]=/$HOME/dir2 
findStr[1]="glcext_[0-9]*-[0-9]*.csv"
renameRegex[1]="p;s/-[0-9]*.csv/.csv/"
set -A loopIdx 0 1 # loop index values
# loop through srcDirs
for i in ${loopIdx[@]};
do
dir=${srcDirs[$i]}
echo "$CurrentDateTime : newYearsTasks.sh : Checking Dir=$dir. Not expecting file delivery from here today, so copying files from previous day" >> $LOG_FILE
cd  $dir
if [[ $DayOfWeek = "Mon" ]]; then
Rename 3 1229
else   # Tues - Sat
Rename 1 1231
fi
#echo "find . -name \""${findStr[$i]}\"" -print | sed \"${renameRegex[$i]}\" | xargs -n2 mv"
find . -name "${findStr[$i]}" -print | sed "${renameRegex[$i]}" | xargs -n2 mv
done
if [[ $? -eq 0 ]]; then
echo "$CurrentDateTime : newYearsTasks.sh : Finished without error"  >> $LOG_FILE
else
echo "$CurrentDateTime : newYearsTasks.sh : No files copied. Please check if process flow"  >> $LOG_FILE
fi