#!/usr/bin/env groovy
// Depends on tagsoup library:
// http://ccil.org/~cowan/XML/tagsoup/
def slurper = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser())
def url = new URL("http://fcd.mcw.edu/?module=faculty&func=view&id=1674")
url.withReader { reader ->
html = slurper.parse(reader)
//we should now have a parsed file
def value = html.body.div.div.div[2].ul.li
value.list().each { f ->
println "\nPub : " << f.toString()[0..80] << "..."
}
}
This is super simple and works really well.
My problem was I needed to also post data to the websites (e.g. to login, enter data etc).
For that you need to incorporate HttpClient. For Groovy there is a HttpBuilder library that wraps HttpClient libraries with Groovy syntax. It also allows you to use GPath expressions to quickly identify locations in the response page.
I needed to go through a proxy and this proved the first hurdle. After much messing this code worked.
(Note I added an if statement to get the current IP address of the machine, so this script would work in work (where we use a proxy), and at home (where I don't have a proxy)
#N.B. Also that httpBuilder can be got using grapes. However I had some problems getting this to work from behind a proxy. Check out this page for some tips
http://groovy.codehaus.org/modules/http-builder/download.html
Also worth of note is that when using grab (grapes) all files are pulled down to $HOME/.groovy/grapes
So you could also manually download the latest version of the HttpBuilder and manualyl install it
Eventually this worked for me (this was after grape resolve failed.. Not sure why.. Obviously it worked once it was installed)
>grape install org.codehaus.groovy.modules.http-builder http-builder 0.6
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.2' ) import groovyx.net.http.* import static groovyx.net.http.ContentType.* import static groovyx.net.http.Method.*
def http = new HTTPBuilder( 'http://www.twitter.com' )
def ip=java.net.InetAddress.getLocalHost().getHostAddress()
println "IP = $ip" if(ip.startsWith("10.5.") || ip.startsWith("10.2.")){
def proxy = "10.5.0.250" def proxyPort = 80 //Required for HttpClient
http.setProxy(proxy, proxyPort, "http") //http.setProxy(proxy, proxyPort, "https")
http.auth.basic( proxy, proxyPort, System.properties["user.name"], System.getenv("user.password") )
}
http.get( path: '/', query:[id:'httpbuilder'] ) { resp, xml ->
println resp.status
println xml
xml.status.each { // iterate over each XML 'status' element in the response:
println it.created_at.text()
println " " + it.text.text()
}
}
Turns out Weblogic uses random number generator during start up. Because of the bug in java it reads ‘randomness’ from /dev/random. /dev/random is very good random numbers generators but it is extremely slow. It takes sometimes 10 minutes or more to generate one number. /dev/urandom is not that good, but it is instant.
Java somehow maps /dev/urandom file to /dev/random. That’s why default settings in $JAVA_HOME/jre/lib/security/java.security are useless.
Possible solutions:
1) Add “-Djava.security.egd=file:/dev/./urandom” (/dev/urandom does not work) to java parameters.
Worse but working solution is:
2) mv /dev/random /dev/random.ORIG ; ln /dev/urandom /dev/random
3) Best solution is to change $JAVA_HOME/jre/lib/security/java.security
Replace securerandom.source with
securerandom.source=file:/dev/./urandom
This problem does not happen under windows because it uses different implementation of /dev/random.
It takes seconds to start weblogic server now.
Date startDate = new Date() -1;
Date endDate = new Date();
def Format = "yyyy-MM-dd"
def OFormat = "YYYY-MM-DD"
String startString = startDate.format(Format);
String endString = endDate.format(Format);
println "Running between $startString and $endString";
def sql = []
def sql1 ="""
select * from prov_auditing_master m where EVENTTIMECOMPLETED > to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql2 =""> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql3 =""> to_date($startString,$OFormat)"+
" and EVENTTIMECOMPLETED < sql4 =""> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql5 =" "> to_date($startString,$OFormat) and EVENTTIMECOMPLETED < sql6 = ""> to_date($startString,$OFormat)"+
" and EVENTTIMECOMPLETED < db =" int" i="1">
println i+": $sq"
i++
}
To fix
GString sql7 = "$sql6" ;
sql <<>select top 10 * from peopleSET ROWCOUNT 10 // use SET ROWCOUNT 0 to turn offBEGIN
FOR cur_rec IN (SELECT object_name, object_type
FROM user_objects
WHERE object_type IN
('TABLE',
'VIEW',
'PACKAGE',
'PROCEDURE',
'FUNCTION',
'SEQUENCE'
))
LOOP
BEGIN
IF cur_rec.object_type = 'TABLE'
THEN
EXECUTE IMMEDIATE 'DROP '
|| cur_rec.object_type
|| ' "'
|| cur_rec.object_name
|| '" CASCADE CONSTRAINTS';
ELSE
EXECUTE IMMEDIATE 'DROP '
|| cur_rec.object_type
|| ' "'
|| cur_rec.object_name
|| '"';
END IF;
EXCEPTION
WHEN OTHERS
THEN
DBMS_OUTPUT.put_line ( 'FAILED: DROP '
|| cur_rec.object_type
|| ' "'
|| cur_rec.object_name
|| '"'
);
END;
END LOOP;
END;
/
// MySql
if not exists (select * from url where url = ...)
insert into url...
//Oracle (where not exists)
insert into <table> (<row1>, <row2>)
select <value1>, <value2> from dual
where not exists (
select * from <table> where <col1> = <value1>)
//SQL Server
if not exists (select * from url where url = ...)
insert into url...
Date Operations
GetDate
//MySql
DATE() see (http://dev.mysql.com/doc/refman/5.0/en/date-and-time-functions.html)
//Oracle
SYSDATE
// Sql Server
getDate()
To Date
//Oracle
to_date('2009-01-01','YYYY-MM-DD')
to_date('2009-05-19 18:32','YYYY-MM-DD HH24:MI')
http://www.dba-oracle.com/f_to_date.htm
Detecting Duplicates (group by criteria)
Say you want to detect duplicates (instances of more than one value) in a column. This is how to do it
select dupCol, count(dupCol) from table group by dupCol having count(dupCol)>1
Listing Constrainsts
Oracle
SELECT * FROM USER_CONSTRAINTS WHERE TABLE_NAME = 'myTable';