Monday, June 24, 2019

SaltStack

Some notes

Get formulas from https://github.com/saltstack-formulas/ e.g. for HAProxy
 https://github.com/saltstack-formulas/haproxy-formula

Steps below is condensed (and includes more explicit commands than on this page https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html)

To use curl 
curl -LOk https://github.com/saltstack-formulas/haproxy-formula.git
Unzip Add to salt Master file_roots sudo vi /etc/salt/master
file_roots:
  base:
    - /srv/salt
    - /srv/formulas/apache-formula
Restart Salt Master
sudo pkill salt-master 
sudo salt-master -d 

Run state e.g. for haproxy
sudo salt '*' state.apply haproxy.install

Salt States

Backup folder
backup_folder:
  file.copy:
    - name: {{ folder-name }}.bak.{{ None|strftime("%Y-%m-%d_%H_%M") }}
    - source: {{folder-name}}
    - user: {{ user}}
    - group: {{group}}


Set variable to latest filename
{%- set fileName = salt['file.find']('/var/publish/',type='f', name='PackageToPublish-1.*.tar.gz')  | last -%}

Include Another state
Say we have a service stop state in a file called service/stop.sls
stop_service:
  service.dead:
    - nameservice

If we are in the same folder and want to include it, we include it usings its filename (with any directories in front)
However in the require step we just include the id name (can also have idnetifies like service: pkg:  etc)

include:
  - service.stop

upgrade_archive_unpacked:
  archive.extracted:
    - name: {{ pillar['root_dir'] }}/{{ pillar['service']['upgrade'] }}
    - source:  {{ pillar['service']['source'] }}
    - source_hash: {{ pillar['service']['source_hash'] }}
    - user: {{ pillar['user'] }}
    - group: {{ pillar['group'] }}
    - overwriteTrue
    - enforce_ownership_on: {{ pillar['root_dir'] }}
    - enforce_toplevelFalse
    - options"--strip-components=1"
    - require
      - stop_service


Rollback to backup
{%- set rollbackFolderTuple = salt['file.find']('/pathToSearch/', type='d', name='dirname.bak.*', print='mtime,name')| sort  | last -%}
{%- set rollbackFolder = rollbackFolderTuple[1] %}
rollback_folder:
  file.rename:
    - name: {{ rollbackFolder }}
    - source: {{folder-name}}
    - user: {{ user}}
    - group: {{group}}

Tuesday, April 09, 2019

Postgres

Cheat sheets

https://gist.github.com/Kartones/dd3ff5ec5ea238d4c546

https://gist.github.com/apolloclark/ea5466d5929e63043dcf

Number of active connections by DB and IP
select count(*),datname, client_addr from pg_stat_activity group by datname, client_addr;

Note can also use ps to show number of active processes
 ps -ef |grep -i postgres

To just show all connections to a particular DB
select substring(query,0,90),state,query_start,pid from pg_stat_activity where datname='DBNAME' order by query_start;

Locked queries
Can use something like this to show locks

SELECT blocked_locks.pid     AS blocked_pid,
         blocking_locks.pid     AS blocking_pid,
         blocking_activity.state AS blocking_state,
         blocking_activity.query_start AS blocking_query_start,
         substring(blocked_activity.query,0,60)    AS blocked_statement,
         substring(blocking_activity.query,0,60)   AS current_statement_in_blocking_process,
         blocked_activity.datname AS db
    FROM  pg_catalog.pg_locks         blocked_locks
     JOIN pg_catalog.pg_stat_activity blocked_activity  ON blocked_activity.pid = blocked_locks.pid
     JOIN pg_catalog.pg_locks         blocking_locks
         ON blocking_locks.locktype = blocked_locks.locktype
         AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
         AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
         AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
         AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
         AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
         AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
         AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
         AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
         AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
         AND blocking_locks.pid != blocked_locks.pid
     JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
    WHERE NOT blocked_locks.GRANTED ORDER BY blocking_activity.query_start;

Permissions and pg_hba

Permissions are controlled by the pg_hba file

To find out where this is run

show hba_file;

Normally somewhere like /var/lib/pgsql/10/data/pg_hba.conf

By default you will not be able to psql -U postgres unless you are the postgres user (in linux)  (You will get fatal Peer authentication failed.. See  https://gist.github.com/AtulKsol/4470d377b448e56468baef85af7fd614).

I have seen this setup to allow all local users get acess


# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# Default is host    all             all             127.0.0.1/32            ident


Data

Postgres stores in files in data folder.
The default is something like this
/var/lib/postgresql/9.5/main
Also seen /var/lib/pgsql/9.6/data/,  Default dir is /usr/local/pgsql/data
Run this to find out actual location

SHOW data_directory

New Setup

After installing

e.g. for postgres 10.

# Init DB (using default data folder)
sudo service postgresql-10 initdb
sudo service postgresql-10 start
exit
sudo su - postgres
cp /var/lib/pgsql/10/data/pg_hba.conf /var/lib/pgsql/10/data/pg_hba.conf.orig
# See below for allowing local users to login as postgres user
vi /var/lib/pgsql/10/data/pg_hba.conf
sudo service postgresql-10 reload
exit
sudo su -
psql -U postgres
    > CREATE ROLE NOSUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;
    > ALTER USER WITH PASSWORD '';
    > \q

Backup/ Restore

SQL Dump

The idea behind this dump method is to generate a file with SQL commands that, when fed back to the server, will recreate the database in the same state as it was at the time of the dump. PostgreSQL provides the utility program pg_dump for this purpose. The basic usage of this command is:

pg_dump dbname > dumpfile

Can be more specific. e.g. -n to backup individucal schema, -t for individual table
pg_dump -Fc %DATABASE% -f %DUMP_FILE_PATH%

As you see, pg_dump writes its result to the standard output. We will see below how this can be useful. While the above command creates a text file, pg_dump can create files in other formats that allow for parallelism and more fine-grained control of object restoration.

Restore

Non-text file dumps are restored using the pg_restore utility. Text files can use psql
psql dbname < dumpfile

where dumpfile is the file output by the pg_dump command. The database dbname will not be created by this command, so you must create it yourself from template0 before executing psql (e.g., with createdb -T template0 dbname)

By default, the psql script will continue to execute after an SQL error is encountered. You might wish to run psql with the ON_ERROR_STOP variable set to alter that behavior and have psql exit with an exit status of 3 if an SQL error occurs:
psql --set ON_ERROR_STOP=on dbname < dumpfile
Either way, you will only have a partially restored database. 
pg_dump dumps only a single database at a time, and it does not dump information about roles or tablespaces (because those are cluster-wide rather than per-database). To support convenient dumping of the entire contents of a database cluster, the pg_dumpall program is provided. pg_dumpall backs up each database in a given cluster, and also preserves cluster-wide data such as role and tablespace definitions. The basic usage of this command is:
pg_dumpall > dumpfile
The resulting dump can be restored with psql:

Barman

Barman is the postgres Backup and ARchive Manager. See http://docs.pgbarman.org/release/2.12/
It will backup the databases configured on a DB server.

Otions.. Streaming (prefered) vs rsync

This is a simple script we used to keep a certain number of fodler (backups). Note it will only delete 1 folder ( the oldest) at a time. So if you have way more folders, you may need to manually delete them first.

 #!/bin/bash

dir="<barmanDir>/Local/base/"
min_dirs=3 // If there are more dirs than this we will delete the oldest

[[ $(find "$dir" -maxdepth 1 -type d | wc -l) -ge $min_dirs ]] &&
IFS= read -r -d $'\0' line < <(find "$dir" -maxdepth 1 -printf '%T@ %p\0' 2>/dev/null | sort -z -n)
file="${line#* }"
ls -lLd "$file"
rm -rf "$file"

Starting stopping

service postgresql-9.6 initdb
chkconfig postgresql-9.6 on
service postgresql-9.6 start

Monday, November 05, 2018

Windows Active Directory Groups/ Roles

To list a users Active Directory groups run this

net user /domain

Problem with this is that it is limited to 21 characters.

Here is a Windows Powershell command to do the same (less memorable though)

(New-Object System.DirectoryServices.DirectorySearcher("(&(objectCategory=User)(samAccountName=$($env:username)))")).FindOne().GetDirectoryEntry().memberOf
or
([ADSISEARCHER]"samaccountname=$($env:USERNAME)").Findone().Properties.memberof

Tuesday, January 10, 2017

Unit testing Spring caching with grails

Grails unit tests do no autowire by default (certainly in version 2.2.3) , so to enable caching in a unit test we had to jump through a few hoops.

The easiest thing in the end was to manually create an xml to load the bean in question. (Once we created the bean in the xml, then the cachable annotations were recognized)
This worked in terms of loading the bean with the caching functionality built in, but then we began to run into class cast exceptions, because of the way that spring implements the caching (using proxys). See http://spring.io/blog/2012/05/23/transactions-caching-and-aop-understanding-proxy-usage-in-spring

The easiest solution we found to this, was to create an interface for the service in question. Then the proxying was able to cast the dynamically generated proxyClass to the interface.

Test xml (in test/unit)

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:cache="http://www.springframework.org/schema/cache"
       xmlns:p="http://www.springframework.org/schema/p"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/cache http://www.springframework.org/schema/cache/spring-cache.xsd">

   

   
   
          class="org.springframework.cache.ehcache.EhCacheCacheManager" p:cache-manager-ref="ehcache"/>

   
   
          class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean" p:config-location="TestEhCache.xml"/>




EhCache.xml (in grails-app/conf)


        xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
        xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">


   
                  maxElementsInMemory='100'
                  overflowToDisk='false' />

   
           maxElementsInMemory="100"
           eternal="false"
           timeToIdleSeconds="3600"
           timeToLiveSeconds="0"
           overflowToDisk="false"
           memoryStoreEvictionPolicy="LFU"/>




Interface
import org.springframework.cache.annotation.Cacheable
interface MyServiceIF  {

    // calling stored procedure to determine the as_of_date
    @Cacheable("priorToDate")
    public Date priorToDate(String yyyymmdd);

}

Class
class MyService implements MyServiceIF  {

    static transactional = false

    public Date priorToDate(String yyyymmdd) {
        return evaluate(yyyymmdd, -1);
    }
}

Spock Test

Note also that if you are declaring a method cacheable, with multiple parameters, then you may want to define a keyGenerator, (or ignore the params)
 public void validateCache() {
        given:
        CacheManager cacheManager = ctx.getBean("cacheManager")
        String result;

        when:
        Cache dateCache = cacheManager.getCache(testName);
        String result1FromCache  = dateCache.get(dateToTest);   // Verify that the cache is empty
        Object resultFromSds
        Object result2FromSds
        Object result2FromCache
        if(testName =="futureBusinessDate" || testName == "pastBusinessDate"){
            resultFromSds = dateToString(daoService."$testName"(dateToTest,1 ))
            Object key = new DefaultKeyGenerator().generate(daoService, DalSdsDateIF.class.getMethod(testName, String.class, int.class), dateToTest, 1)   //compund params, so must generate key
            result2FromCache = dateToString(dateCache.get(key).get());
            result2FromSds = dateToString(daoService."$testName"(dateToTest,1 ))
        } else  {
            resultFromSds = dateToString(daoService."$testName"(dateToTest) )
            result2FromCache = dateToString(dateCache.get(dateToTest).get());
            result2FromSds = dateToString(daoService."$testName"(dateToTest) ) // expect this come from cache, so will not call log again
        }

        then:
        dateCache!=null
        result1FromCache==null    //verify cache is empty
        resultFromSds==expectedResult
        result2FromCache==expectedResult
        result2FromSds==expectedResult
        count ==expectedCallsToLog    // count number of calls  to log.info.. Expect one per call, except for isBusinessDate

        where:
        testName            |  dateToTest   | expectedResult | expectedCallsToLog
        "priorToDate"       | "2016-09-06"  | "2016-09-02"   | 1
        "nextToDate"        | "2016-09-02"  | "2016-09-06"   | 1
        "futureBusinessDate"| "2016-09-02"  | "2016-09-06"   | 1
        "pastBusinessDate"  | "2016-09-06"  | "2016-09-02"   | 1

    }

e.g. in the test
Object key = new DefaultKeyGenerator().generate(daoService, DalSdsDateIF.class.getMethod(testName, String.class, int.class), dateToTest, 1)   //compund params, so must generate key


If you have parameters in the method call that you don't want influsencing the cahce (e.g. ignroe them you can do this, or this)
e.g. to ignore params you can do this

 @Cacheable(value="myCache", key="#root.methodName")// Force key name to be fixed no matter what params passed in
 public Map getValues(List warnings){


Friday, December 30, 2016

Simple script to send email once job finishes

If you have a unix process running then this script can be used to send an email when it is done.
First find th epid of the process

(while kill -0 ; do sleep 1; done) && (echo 'Process Finished now' | mail -s 'job done' Email@recipient.com

Thursday, October 13, 2016

Oracl sql notes

I'm about as far from a Sql expert as its possible to get, so these notes are probably qiute basic

  • Insert into a table only if row does not exists

http://stackoverflow.com/questions/529098/removing-duplicate-rows-from-table-in-oracle
INSERT INTO table
SELECT 'jonny', NULL
  FROM dual -- Not Oracle? No need for dual, drop that line
 WHERE NOT EXISTS (SELECT NULL -- canonical way, but you can select
                               -- anything as EXISTS only checks existence
                     FROM table
                    WHERE name = 'jonny'
                  )

Note its possible to have multiple conditions, e.g.g if table3 also has expected data
INSERT INTO table
    SELECT HIBERNATE_SEQUENCE.NEXTVAL, 1, (select id from table2 where account='052BAFJJ8'), (select id from table3 where name='ABCDE'), 1, 0 FROM dual
     WHERE NOT EXISTS (SELECT id FROM table WHERE account_id = (select id from table2 where account='052BAFJJ8'))
  and EXISTS (SELECT id FROM table3 where name='ABCDE');
Note, there may be race conditions with this approach. In our case we were running it in liquibase scripts, and we didn't have multiple servers running in parallel so this wasn't an issue
  • Delete duplicates
After all my inserts, I ended up with some unexpected duplicates. so I had to delete them. 
I was able to find te duplicate rows easily enough. I searched for all rows with a the same name having a count >1 to find duplicates.
However to delete them we had foreign key relationships, that meant that we could only delte the newly created rows. But these weren't easily identifiable. We decided on the following approach.
We ran the following expression twice. Once with max(rowid) and once with min(rowid), since the table had foreign key dependencies it couldn't be reliably be delted, so this way I managed to get all the duplicates. note if you have many duplicates this may prove more problematic.
exec dbms_errlog.create_error_log(dml_table_name => 'table3' ,err_log_table_name => 'table3_ERRORS') DELETE FROM table3 where rowid in (select min(rowid) from table3 group by name having count(*)>1) log errors into dcu_fund_ERRORS('Is referenced') reject limit 999999999;

Thursday, January 28, 2016

Security conscious coding

OWASP maintain a top 10 list of  security vulnerabilites in systems ( https://www.owasp.org/index.php/Top_10_2013-Top_10 )


They have also now introduced a Developer centric top ten list for proactive controls .
Full document is here.  https://www.owasp.org/images/5/57/OWASP_Proactive_Controls_2.pdf

1. Verify for Security Early and Often
2. Parameterize Queries
3. Encode Data
4. Validate All Inputs
5. Implement Identity and Authentication Controls
6. Implement Appropriate Access Controls
7. Protect Data
8. Implement Logging and Intrusion Detection
9. Leverage Security Frameworks and Libraries
10. Error and Exception Handling

Thursday, January 14, 2016

Hibernate n+1, and Error: a different object with the same identifier value was already associated with the session

I ran into this issue today. It is somewhat related to the causes behind the LazyInstantiationError, in that it is hibernate Sessions getting into a twist.

I had a class structure where we had a Process object, that contains many ProcessEvents. Also, the processEvents could be nested, so optionally they could refer to a parent ProcessEvent.

In grails

class Process {

  static hasMany = [processEvents: ProcessEvent]
  public enum ProcessStatus {
    QUEUED, PROCESSING, SUCCESS, WARN, FAILED
  }
  public enum ProcessSeverity {
    CRITICAL, ERROR, WARN
  }

  //Persisted members
  String name
  Date initiated
  Date complete
  Float progress //Progress percentage
  ProcessStatus status
  String userId
  Map context     //Map to pass arbitrary data
  Date dateCreated
  Date lastUpdated
  ProcessSeverity severity // To determine how to log the error

  static transients = ["context"]

  static constraints = {
    name(blank: false, nullable: false)
    initiated(blank: false, nullable: false)
    complete(blank: true, nullable: true)
    progress(blank: false, nullable: false, max: 100F)
    userId(blank: false, nullable: false, maxSize: 20)
    severity(nullable: true)
    dateCreated(editable: false, required: true)
    lastUpdated(editable: false, required: true)
  }

  static mapping = {
    processEvents sort: 'id'
    processEvents fetch: 'join'
    sort initiated:  'desc'
    processEvents cascade: "all-delete-orphan"
  }

A few things of note here, is in the mapping, we are specifiying fetch join, for the processEvents. This means that instead of loading the processEvents individually (n+1 loads), we bulk load all in advance. Be careful with this if you have large tables, as this can quickly mount up.

Note also we set the processEvents to cascade all deletes, so that all child events get deleted when the parent process is deleted. Note this may not be needed since we have a belongsTo in the processEvent below

class ProcessEvent {
static belongsTo = [process: Process, parent: ProcessEvent]

  public enum EventLevel {DEBUG, INFO, WARN, ERROR}

  String message;
  EventLevel eventLevel
  Date dateCreated
  Date lastUpdated
  Date timestamp
  Boolean hasChildEvents = false // This is for performance increase, instead of calling DB.
  
  static constraints = {
    parent(nullable:  true)
    message(maxSize:3000)
    dateCreated(editable: false, required:true)
    lastUpdated(editable: false, required:true)
    timestamp(editable: false, required:true)
    hasChildEvents(required:false, nullable: true)
  }

  /** table mappings */
  static mapping = {
    parent index: 'processEvent_idx'
    process index:  'process_idx'
sort id:"asc"
  }


  void setMessage(String d){
        message = d?.length() > 3000 ? d.substring(0,3000) : d
    }

}

In the processEvents, we have 2 belongs 2 relations, denoting that all processEvents are a child of a single process, and (optionally) a single parent processEvent.

We began to see the Hibernate Error a different object with the same identifier value was already associated with the session once we added the belongsTo processEvent clause.

The problem was in our add method
Originally we had it coded this way. This will add a new ProcessEvent, to an existing Process object, and an exsitng processEvent parent Event. (We have another method where we do not sepcify a ProcessEvent parent, but that was not causing any problems)

public ProcessEvent addProcessEvent(Long argProcessId, String argMessage, EventLevel argEventLevel, ProcessEvent parent)
{
        if (parent != null) {
            parent = parent.refresh()
            parent.hasChildEvents = true
            parent.save(flush: true)
        }
Process pd = Process.findById(argProcessId)
        ProcessEvent pe = new ProcessEvent(
            message: argMessage,
            eventLevel: argEventLevel,
            timestamp: new Date(),
            parent: parent)
        pd.addToProcessEvents(pe)

      saveProcess(pd)  // saves top level Process, flushes, and logs errors
       pe

}

When we got to the pd.addToProcessEvents (which basically does a save on the Process parent object), it would fail and throw the Hibernate exception.

With some help from Stackoverflow. It mentioned that we had multiple java objects referring to the same row. 
The problem was that we were had a java reference to the parent processEvent object (parent). However we were also loading (findById) the Process object, which was loading a 2nd java reference to the same processEvent object. When we then saved it, there were 2 java references to parent, which was not correct.

The correct version was to load the top level Process object first, and then use the parent ProcessEvent from there. See below

public ProcessEvent addProcessEvent(Long argProcessId, String argMessage, EventLevel argEventLevel, ProcessEvent p)
{
        Process pd = Process.findById(argProcessId)
        Iterator i = pd.processEvents.toArray().iterator()
        ProcessEvent parent=null
        while(i.hasNext()) {
            ProcessEvent next = i.next()
            if(next.id==p.id){
                parent = next
                continue
            }
        }
        if (parent != null) {
            parent.hasChildEvents = true
        }
        ProcessEvent pe = new ProcessEvent(
            message: argMessage,
            eventLevel: argEventLevel,
            timestamp: new Date(),
            parent: parent)
        pd.addToProcessEvents(pe)

      saveProcess(pd)
       pe

}

Worth mentioning also are some other pages with good information
Gorm gotchas part 1, part2, and part 3

Wednesday, October 28, 2015

Linux scripting, date functions and renaming

This is something I always shy away from as I view it somewhat like a black art.

Sample regex in bash including case insensitive, and parsing based on a variable

  host=myhostname
# use the ${var,,} syntax to convert to lowercase
  regex="^myProd.*"
  KerberosU=NonProdUser
  [[ ${host,,} =~ $regex ]] && KerberosU=ProdUser

This is a script I wrote to bulk rename a number of files (and perl rename was not available), on newYears day

There are some nice tricks in here

#!/bin/bash
# Usage
# ksh newYearsTasks.sh |DayOfWeek|     Note DayOfWeek is optional and only to be used for testing purposes
# e.g.  ksh newYearsTasks.sh    This is normal behaviour. The script will default to days day of week
# ksh newYearsTasks.sh Mon . This will override actual DayOfWeek to be Monday for testing purposes
# ksh newYearsTasks.sh . If a paramater other than Mon is set, then the DayOfWeek is set NOT to be Monday. This is for forcing testing of this behaviour on Mondays
#  Requirement is to copy filesfrom a number of src dirs to dest dirs, and rename them somewhat along the way
#
#
#################
#Testing Note
#################
# In order to test this we need create files in the archive directories that would be present on new years day.
# Pay attention to the year component. The script calculates the previous year to generate the rename command, so 
#If you are testing this in 2015 make sure you rename the old files to have 2014 as the year component. Like wise if you are testing in 2016, set the old files to have 2015 as the year component.
# The script copyies files based on their age. If the script is run on a Monday it will copy files that are less than 3 days old
# If the script is run on any other day it will try to copy files that are less than 1 day old (based on modifation date)
# Both scenarios shoudl be tested, and for simplicity sake it is recommended to test Mondays behaviour first.
# 1/ Test Mondays
# To test Mondays behaviour you will need to use touch -d command to set some archive files 3 days old. testDate format is yyyyMMdd, 
#     e.g. if running tests Nov 2. We want to set the files to be 3 days old on touch -d 20151030 * in each of the archive directories to be tested
# To manually invoke the script you can call "ksh newYearsTasks.sh Mon"", which will force the script to behave as if it is Monday regardless of day of week.
# 2/ Test other weekday 
#     Use touch *  in each of the archive directories to be tested to update to be less than one day old
# Run this to force testing with NotMonday behaviour ... "ksh newYearsTasks.sh Tue"


# First Check if server is active
LOG_FILE="/$HOME/log/ops/dailyscriptlogs.log"
# Set variables
CurrentDateTime=`date`
DayOfWeek=$(date +%a)
typeset -i Year
Year=$(date +%Y)
OldYear=$(($Year-1))
function Rename {
echo "$CurrentDateTime : newYearsTasks.sh : Copying files dropped in the last $1 day(s) from archive to filedrop " >> $LOG_FILE
        #find files modified in $1 time, and copy them to dest
find . -mtime -$1 -type f  -exec cp {} ../dest \;
cd ../filedrop
echo "$CurrentDateTime : newYearsTasks.sh : Renaming file date from ${OldYear}$2 to ${Year}0101, and from ${2}${OldYear} to 0101${Year} " >> $LOG_FILE
rename ${OldYear}$2 ${Year}0101 *.csv
rename ${2}${OldYear} 0101${Year} *.csv
}

# Can pass in parameter to overrider todays day of week for testing. Set it to Mon for Monday testing, or anyting else for other day testing
if [[ $# -eq 1 ]]; then
if [[ $1 = "Mon" ]]; then
echo "$CurrentDateTime : newYearsTasks.sh : Overriding Day Of week to $1"  >> $LOG_FILE
DayOfWeek=$1
else
echo "$CurrentDateTime : newYearsTasks.sh : Overriding Day Of week to Tue"  >> $LOG_FILE
DayOfWeek="Tue" 
fi
fi

# Script will run on both active and inactive server, to ensure both Prod and DR have correct files

echo "$CurrentDateTime : newYearsTasks.sh : Executing New Years Day script today " >> $LOG_FILE
#
# Three Arrays representing 1/ srcDirs to copy files from
# 2/ findStr used by the find command to select filename for renaming
# 3/ renameREgex, used to rename the files selected by the find
srcDirs[0]=/$HOME/dir1 
findStr[0]="*ext-[0-9][0-9]*.csv"  #findStr represents the variable used in the find
#rename regex causes original value ('p'), and renamed value to be output. Result is piped to mv
renameRegex[0]="p;s/ext-[0-9]*.csv/ext.csv/" # renameRegex represents the sed regex usedTo rename
srcDirs[1]=/$HOME/dir2 
findStr[1]="glcext_[0-9]*-[0-9]*.csv"
renameRegex[1]="p;s/-[0-9]*.csv/.csv/"
set -A loopIdx 0 1 # loop index values
# loop through srcDirs
for i in ${loopIdx[@]};
do
dir=${srcDirs[$i]}
echo "$CurrentDateTime : newYearsTasks.sh : Checking Dir=$dir. Not expecting file delivery from here today, so copying files from previous day" >> $LOG_FILE
cd  $dir
if [[ $DayOfWeek = "Mon" ]]; then
Rename 3 1229
else   # Tues - Sat
Rename 1 1231
fi
#echo "find . -name \""${findStr[$i]}\"" -print | sed \"${renameRegex[$i]}\" | xargs -n2 mv"
find . -name "${findStr[$i]}" -print | sed "${renameRegex[$i]}" | xargs -n2 mv
done
if [[ $? -eq 0 ]]; then
echo "$CurrentDateTime : newYearsTasks.sh : Finished without error"  >> $LOG_FILE
else
echo "$CurrentDateTime : newYearsTasks.sh : No files copied. Please check if process flow"  >> $LOG_FILE
fi