Tuesday, August 14, 2012

Grails and MongoDb

These are a few notes I took while setting up a project using grails 2.1.0 and mongodb 2.0.6.

Firstly there is a grails plugin for mongoDB. The notes for its use are here

So you can run grails install-plugin mongoDB (or update the BuildConfig.groovy as is the prefered method for grails guru's).

If you want Mongo to be your main DB, then you need to uninstall the hibernate plugin. This shoudl be done via the BuildConfig.groovy. (Comment out the //runtime ":hibernate:$grailsVersion" line)

Next you also need to remove the database-migration plugin, since this is uses the hibernate plugin (//runtime ":database-migration:1.1")

My mongoDB was running on port 27017 which is different from the post specified in the grails mongoDb tutorial (27107), so this must be updated the Datasource.groovy

After a bit of searching I found this sample MongoDb data file . (See http://stackoverflow.com/questions/5723896/is-there-a-sample-mongodb-database-along-the-lines-of-world-for-mysql)

To create a grails domain to fit in with the zips.bson data I did the following. Note we want the id field to be mapped to the mongo _id field, and since it represents the zip code we need to mark the generator as assigned.

class Zip {
    String city
    List loc
    Integer pop
    String state
    String id
   
   
    static mapping = {
        id generator: 'assigned',
        column: '_id'
    }

    static constraints = {
    }
}




Thursday, June 21, 2012

Grails/ Hibernate clean schema

Sometimes you may want to clean up the entire DB. Grails does this for you if you specify create-drop as the datasource dbCreate property, but sometimes (e.g. if you are deploying to a test of prod server, you can't set this to create-drop), you might just want to be able to clean down the database.



One handy script to clear the DB is to run grails schema-export export. This will run the hibernate schema-export ant task, setting the flag to clear the DB. (It will recreate it again after, using the new schema).

If you want to permanently delete the schema you can modify the schema-export and create your own, e.g.

...
target(cleanDb: 'Clean Db using Run Hibernate SchemaExport') {
    depends(packageApp)

    configureFromArgs()

    def file = new File(filename)
    ant.mkdir(dir: file.parentFile)

    configClasspath()
    loadApp()

    populateProperties()

    def configuration = classLoader.loadClass(configClassName).newInstance()
    configuration.setGrailsApplication(grailsApp)
    configuration.setProperties(props)
    def hibernateCfgXml = eventsClassLoader.getResource('hibernate.cfg.xml')
    if (hibernateCfgXml) {
        configuration.configure(hibernateCfgXml)
    }

    def schemaExport = classLoader.loadClass('org.hibernate.tool.hbm2ddl.SchemaExport')
        .newInstance(configuration)
        .setHaltOnError(true)
        .setOutputFile(file.path)
        .setDelimiter(';')

    def action = "Cleaning Db "
    println "${action} in environment '${grailsEnv}' using properties ${props}"


    // 1st drop, warning exceptions
    //schemaExport.execute(stdout, true, true, false)
    schemaExport.drop(true, true)


    if (!schemaExport.exceptions.empty) {
        schemaExport.exceptions[0].printStackTrace()
    }
}




Eager fetching grails and Hibernate

This is taken from grails docs http://grails.org/doc/latest/guide/GORM.html#fetching  
 Basically for single associations use fetch: 'join' (using lazy:false means that there will be 2 sql statements per association.). However for one to many use lazy:false

Configuring Eager Fetching

An alternative approach that avoids the N+1 queries is to use eager fetching, which can be specified as follows:
class Airport {
    String name
    static hasMany = [flights: Flight]
    static mapping = {
        flights lazy: false
    }
}
In this case the flights association will be loaded at the same time as its Airport instance, although a second query will be executed to fetch the collection. You can also use fetch: 'join' instead of lazy: false , in which case GORM will only execute a single query to get the airports and their flights. This works well for single-ended associations, but you need to be careful with one-to-manys. Queries will work as you'd expect right up to the moment you add a limit to the number of results you want. At that point, you will likely end up with fewer results than you were expecting. The reason for this is quite technical but ultimately the problem arises from GORM using a left outer join.
So, the recommendation is currently to use fetch: 'join' for single-ended associations and lazy: false for one-to-manys.
Be careful how and where you use eager loading because you could load your entire database into memory with too many eager associations. You can find more information on the mapping options in the section on the ORM DSL.

Tuesday, June 12, 2012

Maven

Some quick maven tips.

To run some code from a maven project
See this link http://www.vineetmanohar.com/2009/11/3-ways-to-run-java-main-from-maven/

mvn exec:java -Dexec.mainClass="com.vineetmanohar.module.Main" -Dexec.args="arg0 arg1 arg2"

Article explains 2 other ways from mvn pom.xml 

To run unit tests from a maven project
See this link http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html

mvn -Dtest=TestSquare,TestCi*le test
 
(Note can have wildcards in class names)
 
 
Can also specify methods (with wildcards)
mvn -Dtest=TestCircle#test* test
 
To not run tests
I used to use 
mvn -Dmaven.tests.skip install 
 
back when I used maven 2.x. 
However with maven 3, you can skip the entire test phase by this switch
 
mvn -DskipTests=true isntall
 
 

Thursday, June 07, 2012

SSH and Putty

Here's a little time saver for SSH and Putty.

1/ Putty (See http://www.howtoforge.com/ssh_key_based_logins_putty)
Use PuTTYgen to generate a new private/ public keypair
  1. Create a new putty connection
  2. Enter a hostname and a "Saved Sessions" name
    1. Under Connection->Data->Username enter 'username'
    2. Under Connection->Data->SSH->Auth->Private File key for Authentication enter the path of the Private Key
  3. Save the connection
  4. Click Open - You should be logged in without being prompted for user/pass combo.




2/SSH Key Setup for internode SSH
This allows you to ssh between machines without re-logging in
  1. Start a Putty session to each node you wish to reach
  2. Run "ssh-keygen -t rsa" on each one
  3. Append the contents of ~/.ssh/id_rsa.pub to the ~/.ssh/authorized_keys on each of the other machines
You should be able to ssh between each machine (any combo) now without a password.

Friday, May 25, 2012

Bouncy Castle PGP

I was testing some PGP methods from the Bouncy Castle API. When running their ClearSignedFileProcessor I got the following exception 
...
 java.lang.SecurityException: JCE cannot authenticate the provider BC
...


However steps 1 and 2 are not necessary
1. Find java.security in /path_to_your_jvm/jre/lib/security
2. Add security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider

if you instead explicitly add the Bouncy castle provider, (like is done in the example)
Security.addProvider(new BouncyCastleProvider());

Step 3 proved to be necessary. However, when I ran the program as part of a grails project... Strangely it wasnt' necessary. Not sure why that is

Friday, May 11, 2012

Grails org.hibernate.LazyInitializationException: could not initialize proxy - no Session

I was getting this exception which working on a series of domain objects at once. The problem is a well known one, and there are various posted solutions but none of them seemed to apply to me.

Stackoverflow.. They walk about using attach to manually attach to session..
Another Stackoverflow. This time talking about stale DB connections. Not applicable to me, but interesting nonetheless. 
Grails mailing list. Talks about wrapping DB calls in a withTransaction. This could be a runner.. I haven't tried it yet though.

 A quick explanation of my code. The code is calculating reconciliation values. Nothing complicated. The problem sems to be when I look up a currency value in a domain class embedded within the main loop domain object. I have loaded this domain object eagerly, so in theory all the component pieces should also get loaded, but this does not fix the problem.

The commented line  below caused the problems. Explicitly loading each component class seems to fix the problem. This is obviously slow , but speed is not important in this app.

One compounding problem was that this worked on my local environment, but failed in a test environment, so it proved difficult to recreate.

 bank.each(){
        def dcmAccounts = DcmAccount.findAllByDcm(it, [fetch:[fund:"eager", clearingHouse:"eager"]]) // This line fixes the problem
        Long subPercent =percent
        if(dcmAccounts.size()>0){
            subPercent = percent / dcmAccounts.size()
        }
       
        dcmAccounts.eachWithIndex {DcmAccount dcmAccount, cIndex ->
          //def exchangeRate =
ExchangeRate.findByCurrency(dcmAccount.fund.curerncy)?.rate  // This line caused the problem. The fund object is lazily instantiated so when I go to access it, the session has expired. But eagerly loading the funds at the start fixed it
          def a=dcmAccount.fundId
          def b = Fund.get(a).currencyId
          def c = Currency.get(b)
          BigDecimal exchangeRate = ExchangeRate.findByCurrency(c)?.rate
          FundSegregation fundSegregation = new FundSegregation(fundNumber:dcmAccount.fund.fundNumber,fund:dcmAccount.fund, dcmAccount:dcmAccount, limit: dcmAccount.excessCollateralLimit)
          fundSegregation.buildDcmCollateral()
          fundSegregation.buildPledgedCollateral()

Monday, April 02, 2012

JNDI with grails and TcServer

Just something to be aware of.

In order to reference JNDI resources from a grails (and probably java spring) app on TC Server there are 2 steps to perform.

1/ Add the jndi name to the tcserver conf/server.xml file. This will declare the jndi name and the resource that it represents. In your head you may assume that this is the only step you need (I know I did)...

This is a configuration object in the conf dir of your tcServer directory
e.g. (Note also have variables e.g. my.url defined in catalina.properties here)

<Resource accessToUnderlyingConnectionAllowed="false"
defaultAutoCommit="true" defaultReadOnly="false"
driverClassName="com.sybase.jdbc3.jdbc.SybDriver"
factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
initialSize="0" logAbandoned="false" maxActive="8"
maxIdle="8" maxOpenPreparedStatements="0" maxWait="-1"
minEvictableIdleTimeMillis="1800000" minIdle="0"
name="jdbc/myOtherDS" numTestsPerEvictionRun="3"
password="${my.password}" poolPreparedStatements="false"
removeAbandoned="false" removeAbandonedTimeout="300"
testOnBorrow="true" testOnReturn="false"
testWhileIdle="false" timeBetweenEvictionRunsMillis="-1"
type="javax.sql.DataSource" url="${myOther.url}"
username="${myOther.username}" validationQuery="select @@servername" />
<Resource accessToUnderlyingConnectionAllowed="false"
defaultAutoCommit="true" defaultReadOnly="false"
driverClassName="com.ibm.db2.jcc.DB2Driver"
factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
initialSize="0" logAbandoned="false" maxActive="8" maxIdle="8"
maxOpenPreparedStatements="0" maxWait="-1"
minEvictableIdleTimeMillis="1800000" minIdle="0"
name="jdbc/myDS" numTestsPerEvictionRun="3"
password="${my.password}" poolPreparedStatements="false"
removeAbandoned="false" removeAbandonedTimeout="300"
testOnBorrow="true" testOnReturn="false" testWhileIdle="false"
timeBetweenEvictionRunsMillis="-1"
type="javax.sql.DataSource" url="${my.url}"
username="${my.username}" validationQuery="SELECT 1 from sysibm.sysdummy1" />


2/ You also need to add the jndi name to META_INF/context.xml
This simply creates a mapping from the jndi name to a Java object (Normally a dataSource).

the context.xml is normally part of the source code for the application so will be checked into your source control repository.. You will need a new build if you add or change the data in here. This isn't a big deal really since you will need to add code to deal with the new or updated jndi object at any rate.

Sample

<Context path="myAppUrl" debug="1" reloadable="true" docBase="myAppDocBase" antiResourceLocking="true">
<ResourceLink global="jdbc/myDS" name="jdbc/myDS" type="javax.sql.Datasource"/>
<ResourceLink global="jdbc/myOtherDS" name="jdbc/myOtherDS" type="javax.sql.Datasource"/>
</Context>

Wednesday, March 21, 2012

Grails dynamic methods

One of the great features of Grails is the dynamic methods that allow you to create complex lookup methods dynamically.

Here's a couple of points though


Account.findByLabelInList(labels)


The InList appender allows you to search by items in a list

Likewise for Criteria.


CreateCriteria
e.g.
static findMyAccount(def name, def country, def accountType, def labels) {
(DcmAccount)DcmAccount.createCriteria().get {
eq("name", name
eq("country", country)
eq("accountType", accountType)
'in'("labels", labels)
}
}

This is a method on the domain class that I created to lookup accounts based on 4 criteria. Notice the in criteria. It has to be enclosed in quotes since it is a Grails reserved word. It allows you to search for items contained in a list.

Monday, March 12, 2012

FoxClocks

Fox Clocks is a handy little addon for firefox. It can display multiple clocks around the world so you can see the local time in any chosen city.

I thought there was a problem with the daylights saving time bevahiour of FoxClocks.

I'm using version 2.9.35. It was reporting my local time correctly, but 2 days ago the US had their daylights savings time adjustment, while in Europe we are not having ours for 2 more weeks.

My time was been reported corectly, but my US based FoxClocks were not. They were still showing the old pre-Daylight Savings time.

The problem is because that FoxClocks uses an internal database of when daylights saving events occur. I simply had to update that database using Tools/ Options/ Time Zone Data/ Check Now.
When this is done it displayed the correct US times.

Monday, March 05, 2012

Grails Class.forName (loading classes dynamically)

This is a quickie for now.. I will have to come back and expand this entry when I understand more of why it happens.

simple problem. Wanted to load classes dynamically based on a domain instance name.

Simple (in theory)

String name="Domain"
String className = "com.package.test."+name+"Service"
Object o =Class.forName(className).newInstance()

But it fails with a ClassNotFoundException. (and of course the class does exist).

After a lot of tested I came to the following answer
Object o = Thread.currentThread().contextClassLoader.loadClass(className).newInstance()
or
Object o = SomeOtherService.getClassLoader().loadClass(className).newInstance()

Why I don't know. One tends to work in unit tests and not in application code, and the opposite for the other, so often I include both

Here's some links that may help explain.
http://groovy.dzone.com/news/class-loading-fun-groovy

Based on this I wrote a simple test to check the different class loaders

def testB(){
    parseCL(this.getClass().getClassLoader()) //This ClassLoader does NOT work
        
    parseCL(AnotherService.getClassLoader()) // This class Loader does work
}
    
def parseCL(ClassLoader cl){
    println "********************"
    println cl
    println "********************"
    println "Parent ="+cl.getParent()
    cl = cl.getParent()
    def indent = "--"
    while(cl){
        println indent+cl
        cl = cl.getParent()
        println indent+"Parent ="+cl
        indent = indent +"--"
    }
}


Note the top level classloader is different

********************
java.net.URLClassLoader@247510e7 // DOES NOT LOAD CLASS DYNAMICALLY
********************
Parent =java.net.URLClassLoader@77479ef9
--java.net.URLClassLoader@77479ef9
--Parent =org.codehaus.groovy.grails.cli.support.GrailsRootLoader@15ded0fd
----org.codehaus.groovy.grails.cli.support.GrailsRootLoader@15ded0fd
----Parent =sun.misc.Launcher$AppClassLoader@35a16869
------sun.misc.Launcher$AppClassLoader@35a16869
------Parent =sun.misc.Launcher$ExtClassLoader@77cde100
--------sun.misc.Launcher$ExtClassLoader@77cde100
--------Parent =null
********************
java.net.URLClassLoader@77479ef9 // LOADS CLASS DYNAMICALLY
********************
Parent =org.codehaus.groovy.grails.cli.support.GrailsRootLoader@15ded0fd
--org.codehaus.groovy.grails.cli.support.GrailsRootLoader@15ded0fd
--Parent =sun.misc.Launcher$AppClassLoader@35a16869
----sun.misc.Launcher$AppClassLoader@35a16869
----Parent =sun.misc.Launcher$ExtClassLoader@77cde100
------sun.misc.Launcher$ExtClassLoader@77cde100
--------Parent =null

Friday, February 24, 2012

Windows batch files

Some notes on windows batch files e.g. runnign them in background etc. (start /b cmd ..

for example I use this alot in Grails to start a hsql db instance in the background and the asociated db manager

start /B cmd /c "%JAVA_HOME%\bin\java -cp hsqldb-1.8.0.10.jar org.hsqldb.Server -port 9010 -database.0 myDB -dbname.0 myDB"

%JAVA_HOME%\bin\java -classpath hsqldb-1.8.0.10.jar org.hsqldb.util.DatabaseManager --url jdbc:hsqldb:hsql://localhost:9010/myDB


All listed here (antak's answer) http://stackoverflow.com/questions/649634/how-do-i-run-a-bat-file-in-the-background-from-another-bat-file

(note Call is only needed if you are calling another batch file.. If you have an inline command remove the call)

Built in variables
There are a few build in variables that are useful when writing .bat files

(do a
cmd /?
to get some help on the matter)

%~dp0 lists the current path (d represents the directory, p represents the path)
this is useful if you call a batch file from a different directory.

e.g. cd %~dp0

This however can prove problematic in batch files if you want to append information after the path (e.g. I was trying to append a jar filename e.g.
%~dp0/filename 
does not work. (However you can assign it to a variable

set DIR=%~dp0
set FILE=%DIR%/filename


In those cases %CD% is usefule

e.g.

groovyconsole -cp %CD%\jarfile.jar


This will start the groovy console with jarfile in the path. Useful for tinkering with a new api

Monday, January 16, 2012

Java regex's

Just a quick note on java.util.Pattern. e.g. extract numbers from a known pattern

We know that our data format will be various letters, the number we want to extract. One or more letters. How do we extract the numbers?

Easy, using a regex Pattern, and the Matcher class.

e.g. (in groovy)
import java.util.regex.*
def testString ="abc4567!TheEnd"
def pattern = Pattern.compile("\\D*(\\d*).*") // N.B. Double quotes to escape \ pattern == \D*(\d*).*.. i.e. not a number/ number/ anything
Matcher m = pattern.matcher(testString)
if(m.matches()){
def i=0
while(i<=m.groupCount()) println m.group(i++) }else println "Doesn't match"


Returns our number... 4567

Note groups are returned based 1/ the whole matching string followed by a group for each cycle of parentheses (1 in this case).

e.g. if the above regex is changed to
"\\D*(\\d(\\d(\\d*))).*"

Then the results would be as follows. Note how as the group index increases, we traverse deeper into the parentheses of the regex
group(0) == abc4567!TheEnd
group(1) == 4567
group(2) == 567
group(3) == 67