Mercurial behind a proxy server

I found some web resources that confused things when I was trying to figure this out so I thought I'd add a page.

Mercurial (hg) stores its config information in an ini file. Type
hg help config
to see the details.

Based on this its possible to have a config file per repo with differetn setting (e.g. email addresses).

The format of the config file is explained at
http://www.selenic.com/mercurial/hgrc.5.html

So to set a proxy server I used the follwoing ini file

[ui]
username = My Name <my.name@myemail.com>
verbose = True
[http_proxy]
host=http://<proxyhost>:<proxyport>
no=192.168.*
user=<optionalusername>
passwd=<optionalpassword>

ScreenScraping with Groovy

I'm attempting to write a screen scraper for various websites, so I will document my findings here.

A quick search on the internet finds links to examples like the following

http://groovysnippets.blogspot.com/2008/11/groovy-screen-scraping-example.html

#!/usr/bin/env groovy
// Depends on tagsoup library:
//      http://ccil.org/~cowan/XML/tagsoup/
def slurper = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser())
def url = new URL("http://fcd.mcw.edu/?module=faculty&func=view&id=1674")
url.withReader { reader ->
       html = slurper.parse(reader)
       //we should now have a parsed file
       def value = html.body.div.div.div[2].ul.li
       value.list().each { f ->
               println "\nPub : " << f.toString()[0..80] << "..."
       }
}

This is super simple and works really well.

My problem was I needed to also post data to the websites (e.g. to login, enter data etc).

For that you need to incorporate HttpClient. For Groovy there is a HttpBuilder library that wraps HttpClient libraries with Groovy syntax. It also allows you to use GPath expressions to quickly identify locations in the response page.

I needed to go through a proxy and this proved the first hurdle. After much messing this code worked.

(Note I added an if statement to get the current IP address of the machine, so this script would work in work (where we use a proxy), and at home (where I don't have a proxy)

#N.B. Also that httpBuilder can be got using grapes. However I had some problems getting this to work from behind a proxy. Check out this page for some tips

http://groovy.codehaus.org/modules/http-builder/download.html

Also worth of note is that when using grab (grapes) all files are pulled down to $HOME/.groovy/grapes 

So you could also manually download the latest version of the HttpBuilder and manualyl install it

Eventually this worked for me (this was after grape resolve failed.. Not sure why.. Obviously it worked once it was installed)

>grape install org.codehaus.groovy.modules.http-builder http-builder 0.6
 

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.2' )
import groovyx.net.http.*
import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.* 

def http = new HTTPBuilder( 'http://www.twitter.com' ) 
def ip=java.net.InetAddress.getLocalHost().getHostAddress() 
println "IP = $ip" if(ip.startsWith("10.5.") || ip.startsWith("10.2.")){
 def proxy = "10.5.0.250"  def proxyPort = 80  //Required for HttpClient
 http.setProxy(proxy, proxyPort, "http")  //http.setProxy(proxy, proxyPort, "https")
 http.auth.basic( proxy, proxyPort, System.properties["user.name"], System.getenv("user.password") ) 
} 
http.get( path: '/', query:[id:'httpbuilder'] ) { resp, xml ->     
 println resp.status  
 println xml     
 xml.status.each {  // iterate over each XML 'status' element in the response:         
  println it.created_at.text()         
  println "  " + it.text.text()     
 } 
}

Like the simple example above (which used the TagSoup Sax Parser), the HttpBuilder parser includes an Xml Parser that can handle HTML

HTML response data will also be parsed automatically, by using NekoHTML which corrects the XML stream before it is passed to the XmlSlurper. The resulting behavior is that you can parse HTML as if it was well-formed XML.

Web effects

I will use this post to pull together some neat effects I use. In general I will try to use jquery as the javacscript library.

Overlay

http://www.sohtanaka.com/web-design/fancy-thumbnail-hover-effect-w-jquery/

Strange error from Sql Server

ErrorMessage:

While trying to extract DDL and Data sql scripts using the Database publishing Wizard

Could not load file or assembly 'Microsoft.SqlServer.BatchParser, Version=9.0.242.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. An attempt was made to load a program with an incorrect format.

Solution:

This is due to a problem with Microsoft.SqlServer.BatchParser.dll on Windows 64-bit.

Download and install

X64 Package (SQLServer2005_XMO_x64.msi) - 14675 KB

This installs the 64 bit smo objects.

ErrorMessage:

com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near '@P0'.

Solution

This was a simple one.

This was my JDBC call

PreparedStatement ps = dbConnection.prepareStatement("insert into master_key (key_value) values ?");

ps.setString(1, key);

int id = ps.executeUpdate();

Problem was the missing '()' in the Sql

PreparedStatement ps = dbConnection.prepareStatement("insert into master_key (key_value) values (?)");

Setting up drupal

• Install drupal/ Db etc. Don't use command line for loading DB
• install modules Nice Menu/ Views/ tinyMCE/ Image Assist
• set up categories
• Add content , this creates menu
• Adjust setting/ Content Types to have correct defaults for adding content
• Configure tinyMCE, and imageAssist. Follow install.txt in tinyMce modules dir
• Create Admin role. administrator/ access control
• Add tinyMCE permissions to different roles

Weblogic slow to start (and slow cryptography) on linux (unix),

This is a doozie.

Weblogic takes many minutes to start on unix.

Solution provided by http://www.itonguard.com/20090313/weblogic-starts-slow/

Note this would also apply for any java process that uses the random number generator such as cryptography.

Turns out Weblogic uses random number generator during start up. Because of the bug in java it reads ‘randomness’ from /dev/random. /dev/random is very good random numbers generators but it is extremely slow. It takes sometimes 10 minutes or more to generate one number. /dev/urandom is not that good, but it is instant.
Java somehow maps /dev/urandom file to /dev/random. That’s why default settings in $JAVA_HOME/jre/lib/security/java.security are useless.

Possible solutions:
1) Add “-Djava.security.egd=file:/dev/./urandom” (/dev/urandom does not work) to java parameters.

Worse but working solution is:
2) mv /dev/random /dev/random.ORIG ; ln /dev/urandom /dev/random

3) Best solution is to change $JAVA_HOME/jre/lib/security/java.security
Replace securerandom.source with

securerandom.source=file:/dev/./urandom

This problem does not happen under windows because it uses different implementation of /dev/random.

It takes seconds to start weblogic server now.

Using AES from Java

The following exception

java.security.InvalidKeyException: Illegal key size or default parameters

at javax.crypto.Cipher.a(DashoA13*..)

at javax.crypto.Cipher.a(DashoA13*..)

at javax.crypto.Cipher.a(DashoA13*..)

at javax.crypto.Cipher.init(DashoA13*..)

at javax.crypto.Cipher.init(DashoA13*..)

...

Searching java forums reveals that I need to download the Unrestricted Policy Files from JDK Distribution Center.

After much searching I found it at

http://java.sun.com/javase/downloads/index.jsp

Under Additional Resources/ Other Downloads.

Simply copy the jars included in it into %JRE_HOME%/lib/security overwriting what is there

On my system that means doing it in multiple locations (eclipse JRE/ Oracle JRE etc.) to make

Keytool keystore Cannot store non-PrivateKeys

Don't believe the error, or a lot of websites....

I have a requirement to encrypt passwords in a file. I figured I'd use the java security tools to store the encryption key.

Step 1/ Generate the secret key (Using AES as encryption algorithm)

.\keytool -genseckey -alias aestest -keyalg AES -keysize 192

This results in error

keytool error: java.security.KeyStoreException: Cannot store non-PrivateKeys

After much searching, and some incorrect pointers (Keytool cannot generate symmetric keys, keytool cannot generate AES keys etc.), I found the solution. The problem is the default Keystore type is JKS. This cannot store symmetric keys. However if you change the keystore type to JCEKS then it works.

e.g.

.\keytool -genseckey -alias aestest -keyalg AES -keysize 192 -storetype JCEKS

One side effect of this is that you then need to specify the storeType in every command there-after.

e.g.

keytool -list

keytool error: java.io.IOException: Invalid keystore format

keytool -list -storetype JCEKS

Your keystore contains 3 entries

aestest, 14-Dec-2009, SecretKeyEntry,

aestest2, 14-Dec-2009, SecretKeyEntry,

test, 14-Dec-2009, SecretKeyEntry,

Xquery to mask CreditCard number

declare function local:maskString($input as xs:string, $numClearDigits as xs:int)

as xs:string{

let $len := fn:string-length($input)

let $maskLen := $len - $numClearDigits

let $clearText := fn:substring($input, $maskLen+1)

let $mask := fn:string-join (

(for $i in (1 to $maskLen) return "X", $clearText)

,'')

return

$mask

};

Detect Java method name from code

http://www.rgagnon.com/javadetails/java-0420.html

JDK1.4

public class MyTest {

public static void main(String args[]) {

new MyTest().doit();

}

public void doit() {

System.out.println

(new Exception().getStackTrace()[0].getMethodName());

}

}

The output

doit

JDK1.5

While the above snippet is not bad, it is expensive since we need to create an Exception.

With JDK1.5, a new technique is available.

public class Test {

public static void main(String args[]) {

trace(Thread.currentThread().getStackTrace());

new Test().doit();

trace(Thread.currentThread().getStackTrace());

}

public void doit() {

trace(Thread.currentThread().getStackTrace());

doitagain();

}

public void doitagain() {

trace(Thread.currentThread().getStackTrace());

}

public static void trace(StackTraceElement e[]) {

boolean doNext = false;

for (StackTraceElement s : e) {

if (doNext) {

System.out.println(s.getMethodName());

return;

}

doNext = s.getMethodName().equals("getStackTrace");

}

}

}

Output

main

doit

doitagain

main

Mvn Mojo architype type

Quick note.

This maven page (on your first mojo) has a typo.

When creating a default mojo project using the architype the instruction says run

mvn archetype:create -DgroupId=quinn -DartifactId=maven-loop-task -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-mojo

This fails. In fact you need to include the version attribute like so

mvn archetype:create -DgroupId=quinn -DartifactId=maven-loop-task -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-mojo -DarchetypeVersion=1.0

USB drive won't go

I was been particularly stupid one day, trying to repartition my backup USB drive without first backing it up.

Yes I know I was stupid ,and frankly I probably deserved to lose everything, but I was lucky. The drive woulldn't appear on my laptop, and yet after running all the windows diagnostics , I could find no problems.

Running the following prog solved the problem

diskmgmt.msc

Select a drive letter and away you go.

Grails cheatsheet

Getting Started

grails set-proxy (if necessary to set proxy)

grails create-app AppName

cd AppName

grails create-domain-class Domain

grails generate-all Domain

Install Acegi

grails install-plugin acegi

grails create-auth-domains

grails generate-manager

grails generate-registration

Tools

grails schema-export (Export a DDL sql script for the domain classes)

grails generate-hbm-xml (Generate hibernate mapping file (From grails 1.3)

grails install-templates (Install base scaffolding files. Can then update them before generating scaffolding.

e.g. I update list default size to 50 (from 10)), Increase number of columns displayed in list tables (max by defauilt is 6 columns), changing if (i < 6){ { allows you to increase this (change in 2 places)

DB-Plugin

grails install-plugin db-stuff

grails create-db

grails export-data

grails export-data-diff

grails export-ddl

grails load-data

Maven integration

http://www.grails.org/Maven+Integration

Build a new grails project with maven.

mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate -DarchetypeGroupId=org.grails -DarchetypeArtifactId=grails-maven-archetype -DarchetypeVersion=1.2.0 -DgroupId=example -DartifactId=my-app

mvn initialize

Misc

1. GORM Non numeric ID field

See my question, and someone elses helpful answer on the grails mailing list for how to create a non-numeric ID field, and rename it to something other than ID. Its a bit sneaky but works perfectly. http://grails.1312388.n4.nabble.com/Non-numeric-ID-tt2073698.html#none

1. Dependency Injection.

Ran into a stupid problem with Dependency injection. I was trying to inject the DataSource into a Service (and into a Java library). It appeared the Data source was not getting injected into grails Service class. Dependency injection was not working!!

As usual it was working fine, and the problem was mine...

Dependency Injection (DI) is configured by default to occur by name. If you create a property in your Domain/ Controller/ Service class with the same name as another Service or property you want injected, then it will automatically get injected. See this article http://www.grails.org/Services

To inject the DataSource into your Service, All you need to do is add the line

def dataSource

to your Service. Thats all.. nothing else is needed.

I also had to add the following to initialize the java libraries dataSource, so I also needed

class BenefitService implements InitializingBean{

void afterPropertiesSet() {

println "Initializing BenefitService "+dataSource

BenefitWrapper.setDataSource(dataSource)

}

This all would work. My problem was I (helpfully) added an accessor method for the dataSource

def getDataSource(){return dataSource)

This simple method modifies the groovy class dataSource property, effectively making the setter of the dataSource unavailable, thus effectively blocking dependency injection.

Moral: Groovy is designed to help reduce the number of code lines you write. Don't add unneccessary lines or there may be unexpected side-effects.

Shutdown hook in Groovy... Inner Classes in Groovy

Groovy doesn't support Inner classes directly. I was in a hurry to implement a shutdown hook in Java so I came accross the groovy workaround
http://groovy.codehaus.org/Groovy+Alternatives+to+Inner+Classes

I had to tweak the script slightly to get it to work for the shutdown hook (see below).

To implement the shutdown hook in Java a Thread instance is passed to the addShutdownHook() method of the Runtime instance.

In Java we could create an Inner class that extends Thread, or that implements Runnable.

In Groovy we must create a Map. The Map contains a list of key/ values. These correspond to method names (key), and method implementations (closure). For this case the closure will represent the Thread that we pass to the Runtime.addShutdownHook() method. Therefore we are implmenting the run() method, so the key in the map must be run().

The ProxyGenerator is then used to dynamically add the interface to the close. Finally since Runtime.addShutdownHook() expects a Thread instance (not simply a class that implements Runnable) ,we must add the Thread.class to the call to ProxyGenerator.instantiateAggregate().



def shutdownClosureMap = [run: {
outputFile.close();
println "Shutting down";
}
]
def interfaces = [Runnable]
def shutdownListener = ProxyGenerator.instantiateAggregate(shutdownClosureMap, interfaces, Thread.class)

Runtime.getRuntime().addShutdownHook((Thread)shutdownListener);

Groovy GString/ STring

A few Gotchas that I've been getting stung with in Groovy.

GStrings and Strings

Be careful with Groovy Strings and Sql.

Make sure you know the rules of when Strings become GStrings, and therefore can support dynamic variable replacements.

GString docs can be found
http://docs.codehaus.org/display/GROOVY/Strings+and+GString

These rules are listed here: (Note this doc is aproposal for changnig GString)
http://docs.codehaus.org/display/GroovyJSR/Groovy+String+Handling

For example in the code below, if you simply use "def" to declare your variable, you will not see explicitly which types get converted to GStrings and which get converted to plain Strings.

By been explicit in your definitions however you can see that sql6 will not cast by default to a GString. This is becausei t is a multiline java.lang.String (see rules in link above). You will get this exception.

org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object "..." with class 'java.lang.String' to class 'groovy.lang.GString'

To fix it.


Date startDate = new Date() -1;
Date endDate = new Date();
def Format = "yyyy-MM-dd"
def OFormat = "YYYY-MM-DD"
String startString = startDate.format(Format);
String endString = endDate.format(Format);
println "Running between $startString and $endString";
def sql = []
def sql1 ="""
select * from prov_auditing_master m where EVENTTIMECOMPLETED > to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql2 =""> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql3 =""> to_date($startString,$OFormat)"+
" and EVENTTIMECOMPLETED < sql4 =""> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql5 =" "> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql6 = ""> to_date($startString,$OFormat)"+
" and EVENTTIMECOMPLETED < db =" int" i="1">
println i+": $sq"
i++
}

To fix

GString sql7 = "$sql6" ;
sql <<>

DB/ Database notes (sql tips)

JDBC Format 

Sybase  (jconn3.jar)
jdbc:sybase:Tds:server:1050


Oracle
Use Service Name (format /)

jdbc:oracle:thin:@server:1521/fmwa

user SID (format :)

jdbc:oracle:thin:@bggwrsoa12:1521:fmwa

Limit Rows

// mysql
select col from tbl limit 20;


//Sybase / SqlServer

select top 10 * from people

 //Also
SET ROWCOUNT 10 // use SET ROWCOUNT 0 to turn off

// Oracle
select col from tbl where rownum <=20;

Using rank()

This query will return the top 10 rows..
select col1,col2 from (select rank() over (order by col1) r , col1,col2 from TABLE) where r<11 For bottom 10 rows use select col1,col2 from (select rank() over (order by col1 DESC) r , col1,col2 from TABLE) where r<11

Select unique

//Sybase / SqlServer
select distinct col from table

Oracle

MySql

Outer Join

// Oracle left outer Join
select * from Customer, Order where Order.customerId (+) = Customer.id
//right join
select * from Customer, Order where Order.customerId = Customer.id (+)

Backup

Mysql

>mysqldump --host vc2c09mmk0326.fmr.com --port=3306 -u datasift2 -p --set-gtid-purged=OFF datasift2  > datasift.sql

• This will create a sql with all the tables. Note the --set-gtid-purged=OFF is required on newer versions of the client. If you are running on vc2c09mmk0326 it is probably not required.
• Note the sql will be over 100MB in size

Oracle drop all tables

BEGIN
   FOR cur_rec IN (SELECT object_name, object_type
                     FROM user_objects
                    WHERE object_type IN
                             ('TABLE',
                              'VIEW',
                              'PACKAGE',
                              'PROCEDURE',
                              'FUNCTION',
                              'SEQUENCE'
                             ))
   LOOP
      BEGIN
         IF cur_rec.object_type = 'TABLE'
         THEN
            EXECUTE IMMEDIATE    'DROP '
                              || cur_rec.object_type
                              || ' "'
                              || cur_rec.object_name
                              || '" CASCADE CONSTRAINTS';
         ELSE
            EXECUTE IMMEDIATE    'DROP '
                              || cur_rec.object_type
                              || ' "'
                              || cur_rec.object_name
                              || '"';
         END IF;
      EXCEPTION
         WHEN OTHERS
         THEN
            DBMS_OUTPUT.put_line (   'FAILED: DROP '
                                  || cur_rec.object_type
                                  || ' "'
                                  || cur_rec.object_name
                                  || '"'
                                 );
      END;
   END LOOP;
END;
/

// HSQL
from Customer left join Order where Order.customerId = Customer.id

Could only get this working by explicitly linking the join in the hbm file, and marking it as optional e.g.

Need some more testing to see if this can be done without updating the hbm file

<class name="AdjustmentCode" table="adjustment_code">
<id name="id" column="adjustment_code" type="string">
<generator class="native">
<property name="description" column="description" type="string">
<property name="adjustmentCode" column="adjustment_code" type="string" insert="false" update="false">
<join table="adjustment_message">optional="true">
<key column="adjustment_code">
</join>
<class>
Insert if row not Exists

// MySql
if not exists (select * from url where url = ...)
insert into url...

//Oracle (where not exists)
insert into <table> (<row1>, <row2>)
select <value1>, <value2> from dual
where not exists (
select * from <table> where <col1> = <value1>)

//SQL Server
if not exists (select * from url where url = ...)
insert into url...
Date Operations

GetDate
//MySql
DATE()  see (http://dev.mysql.com/doc/refman/5.0/en/date-and-time-functions.html)

//Oracle
SYSDATE

// Sql Server
getDate()


To Date
//Oracle
to_date('2009-01-01','YYYY-MM-DD')
to_date('2009-05-19 18:32','YYYY-MM-DD HH24:MI')

http://www.dba-oracle.com/f_to_date.htm

Detecting Duplicates (group by criteria)
Say you want to detect duplicates (instances of more than one value) in a column. This is how to do it

select dupCol, count(dupCol) from table group by dupCol having count(dupCol)>1

Listing Constrainsts
Oracle
SELECT * FROM USER_CONSTRAINTS WHERE TABLE_NAME = 'myTable';

Online XPath Tester

After looking for a quick online XPath Tester I decided to write my own.

This is an online xpath tool written in Javascript

Online XPath Tester

The following articles were helful in getting it running

Javascript Xpath tutorial

Javascript to convert  Xml to String/ String to Xml Node  

This article here is the Official Javascript guide to XPATH

These articles links to the Reference guides for XML. The  API for the XPath objects and the API for the w3c Xml objects

Not used here but useful nonetheless is a Javascript Reflection API. This article shows how to do it. I modified the code slightly to get it working

This article described

How to create static pages within blogger

Weblogic Jms error.. "Peer Gone"

"It is likely that the remote side declared peer gone on this JVM"

This error had been causing me grief for a long time.

The problem was fixed by updating my etc hosts file on the client machine... (/windows/system32/drivers/etc/hosts on windows).

Some of our weblogic servers display the server name including domain name e.g. server.domain.com and some only display the server name (e.g. servername). By adding an entry for the target weblogic server in the etc/hosts the problem was fixed.

e.g.
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
127.0.0.1 localhost
192.168.1.10 servername servername.domain.com

Bea Articles on Oracle

When Oracle acquired BEa they began moving all the bea and weblogic sites into the oracle.com domain.

Problem of course is that a lot of good articles are referenced on the web to their old original Bea urls.

This page lists where the equivilent sections exist on the oracle site

Overview of migration -> http://wiki.oracle.com/page/Dev2Dev+%2F+Arch2Arch+Status+and+Migration+Update?t=anon

Dev2dev and arch2arch -> http://www.oracle.com/technology/pub/articles/dev2arch/index.html

Weblogic JMS rollback

I was getting the following error in a Weblogic 9.2 deployment.

####<10-apr-2009> <info> <ejb> <vmmiddleware11> <mwalsb11> <[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1239355884374> <bea-010213> <message-driven xid="BEA1-23F28F44E9B311BE176F(110581804),Status="Rolled" reason="weblogic.transaction.internal.AppSetRollbackOnlyException],numRepliesOwedMe="0,numRepliesOwedOthers="0,seconds" begin="0,seconds" left="60,XAServerResourceInfo[WLStore_MWALSBDomain_JDBCStore_MWALSB11]="(ServerResourceInfo[WLStore_MWALSBDomain_JDBCStore_MWALSB11]="(state="rolledback,assigned="MWALSB11),xar="WLStore_MWALSBDomain_JDBCStore_MWALSB1146900074,re-Registered" state="rolledback),OwnerTransactionManager="ServerTM[ServerCoordinatorDescriptor="(CoordinatorURL="MWALSB11+vmmiddleware11:7011+MWALSBDomain+t3+," xaresources="{WLStore_MWALSBDomain_JDBCStore_MWALSB11," nonxaresources="{})],CoordinatorURL="MWALSB11+vmmiddleware11:7011+MWALSBDomain+t3+).">

First steps are to get more informative error messages by enabling debugging for JTA

This is done by adding the following switches to weblogic on startup

-Dweblogic.debug.DebugJDBCJTA=true
-Dweblogic.log.StdoutSeverity="Debug"

This article give a list of other possible switches available on Weblogic
http://www.questtech.co.uk/index.php?option=com_content&view=article&id=163&Itemid=40