Follow me

Thursday, September 13, 2012

GRAILS DATABASE AND CONFIG PROPERTIES EXTERNALIZATION

In Grails' DataSource descriptor file located at grails-app/conf/DataSource.groovy.
This file contains the dataSource definition which includes the database settings like
driverClassName ,username ,password ,url  etc.....

Since Grails' DataSource definition is "environment aware", so you can do it like:

local {
                              dataSource {
                                             dbCreate = "update" // one of 'create', 'create-drop','update'
                                             url = "jdbc:mysql://localhost/dbname"                   
                                             username = "username"
                                             password = "password"
                              }
               }
production {
                              dataSource {
                                             dbCreate = "update" // one of 'create', 'create-drop','update'
                                             url = "jdbc:mysql://hostname/dbname"                  
                                             username = "username"
                                             password = "password"
                              }
               }

We can externalize these settings by creating the separate property file to set above parameter.
Below are the steps I follow to externalize the database parameter in grails application.

Step 1)
Set one System property which is path to your externalize property file or dir
                          JAVA_OPTS="$JAVA_OPTS -DextConfigDir=/path/to/ext/prop"
Step 2)
                       In Config.groovy add the below chunk of code. This will search for config files that get
                       merged into the main config.
                              def extConfig = System.properties.getProperty('extConfigDir')
                              grails.config.locations = ["file:${userHome}/.grails/${appName}-config.groovy"]
                              if (extConfig) {
                                             grails.config.locations << "file:${extConfig}/Config.properties"
                              }
                 
Step 3)
                    Creating the externalize property file.
                    Remove the datasource properties from DataSource.groovy and set this properties in
                    Config.properties mentioned below
                       Ex: - In DataSource.groovy
                              local {
                                             dataSource {
                                                            url = "jdbc:mysql://localhost/dbname"                   
                                                            username = "username"
                                                            password = "password"
                                             }
                              }
                              The Exteralize Config.properties file will be
                              dataSource.url=jdbc:mysql://localhost/dbname
                              dataSource.username=username
                              dataSource.password=password

 In this way we can externalize the data source as well as Config.groovy properties

Monday, September 10, 2012

Grails Log4j Externalization


Problem statement  

 As we know Grails application used its common configuration mechanism to
 provide the settings for log4j so for this we just need to add the log4j setting in
grails-pp/conf/Config.groovy but when this application goes live as a part of war we can’t set the logging level until we deploy the new war by making change is log4j settings.So externalization allows you to change the logs with just a server restart.

Below are the steps I perform to externalize the grails log4j settings.

1) Crete the log4j.xml by referring the log4j  block  in Config.groovy

   below is the xml representation of grails log4j block

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false" threshold="null">
    <appender name="stacktraceLog" class="org.apache.log4j.FileAppender">
        <param name="file" value="changeme/logs/stackTrace.log"/>
        <param name="name" value="stackTraceLog"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="conversionPattern" value="%d [%t] %-5p %c{2} %x - %m%n"/>
        </layout>
    </appender>
 <appender name="FILE" class="org.apache.log4j.DailyRollingFileAppender">
    <param name="File" value="changeme/logs/catalina.out" />
   <param name="Threshold" value="INFO"/>
    <param name="Append" value="true"/>
 <param name="DatePattern" value="'.'yyyy-MM-dd"/>
    <layout class="org.apache.log4j.PatternLayout">
      <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS Z} [Thread: %t  ] level=%-5p %m%n"/>
    </layout>
  </appender>
  <appender name="stdout" class="org.apache.log4j.ConsoleAppender">
        <param name="name" value="stdout"/>
 <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="conversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS Z} level=%-5p class=%c %X{uniqueId} %X{hostname} %X{requestURI} %X{clientIP} %X{userId} %X{realmId} %X{sessionId} %X{locale} %X{callingHost} %X{uniqueIdCallingHost} %X{asyncUserId} %X{isAsync} %X{taskId} %m%n"/>
        </layout>
    </appender>
     <!-- FOR STACKTRACE -->
    <logger name="StackTrace" additivity="false">
        <appender-ref ref="stacktraceLog"/>
    </logger>
    <!-- FROM GRAILS-->
    <logger name="org.codehaus.groovy.grails.web.servlet">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.web.pages">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.web.sitemesh">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.web.mapping.filter">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.web.mapping">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.commons">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.plugins">
        <level value="ERROR"/>
    </logger>
    <logger name="org.codehaus.groovy.grails.orm.hibernate">
        <level value="ERROR"/>
    </logger>
    <logger name="org.springframework">
        <level value="ERROR"/>
    </logger>
    <logger name="org.hibernate">
        <level value="INFO"/>
    </logger>
    <logger name="org">
        <level value="ERROR"/>
    </logger>
    <logger name="org.mortbay.log">
        <level value="WARN"/>
    </logger>
    <logger name="com.opensymphony">
        <level value="WARN"/>
    </logger>
   
    <!-- FROM GRAILS END -->
   <category additivity="true" name="net.sf">
      <priority value="INFO"/>
   </category>
    <category additivity="false" name="org.apache">
         <priority value="INFO"/>
 <appender-ref ref="FILE"/>
   </category>
    <root>
       <level value="DEBUG"/>
      <appender-ref ref="stdout"/>
       <appender-ref ref="APPLOG"/>
 <appender-ref ref="FILE"/>
    </root>
</log4j:configuration>


2) Remove the log4j block from the Config.groovy i.e.       log4j = {  }

3)  Add below log4jConfigurer bean in groovy class
     /grails-app/conf/spring/resources.groovy and set the path of       externalize log4j.xml

      log4jConfigurer(org.springframework.beans.factory.config.
       MethodInvokingFactoryBean){
        targetClass = "org.springframework.util.Log4jConfigurer"
        targetMethod = "initLogging"
        arguments = "path/to/external/log4j/log4j.xml"
      }


This will allow you to configure grails log externally !!!

How To Add Log4J For Logging In Tomcat [For grails application]

Purpose

The purpose of this document is to explain you the steps to enable the log4j logging in  apache  tomcat as you may know that tomcat used system logging API i.e. java.util.logging for its internal logging.

In one of my project I came across such scenario where I need to implement the log4j logger for tomcat logs as well as application [grails ] log.

So  below are the steps we perform to set up log4j logging on tomcat 7 for grails 1.3.7 application.

1) Move existing  tomcat conf/logging.properties to some other name

        mv logging.properties logging.properties-back

2) Download Tomcat juli adapters library to  tomcat/lib/

     wget http://mirror.sdunix.com/apache/tomcat/tomcat-7/v7.0.28/bin/extras/tomcat-juli-adapters.jar

3)  Download Tomcat juli library to tomcat/lib/

wget http://mirror.sdunix.com/apache/tomcat/tomcat-7/v7.0.28/bin/extras/tomcat-juli.jar

4)  To make log4j work with grails app using tomcat-juli.jar we need
org\apache\juli\logging\UserDataHelper.java  which is not available in extra/ tomcat-juli.jar [but available in default tomcat jar]  so we make a patch called tomcat-log4j-patch.jar. contain UserDataHelper.class [NOT UPLOADED] . Copy this jar in tomcat lib folder

5) Copy  log4j.jar in tomcat lib folder

6) Open the catalina.sh and find juli LogManager config section. In else part add the log4j.xml path

 #Set juli LogManager config file if it is present and an override has not been issued
 if [ -z "$LOGGING_CONFIG" ]; thenif [ -r "$CATALINA_BASE"/conf/logging.properties ]; then
 LOGGING_CONFIG="-Djava.util.logging.config.file=$CATALINA_BASE/conf/logging.properties"
 else
 LOGGING_CONFIG="-Dlog4j.configuration=file://$CATALINA_BASE/lib/log4j.xml" fi

7) Restart the server

Saturday, March 24, 2012

Mongo Monitor Script: Health Check

Description


The Script will search for any working server listed in the mongo.properties. It executes the rs.status() command and check the health of all the server in replication


Step to configure mongo.properties


In mongo.properties we need to set the below properties


<!--[if !supportLists]-->· <!--[endif]-->USERNAME [mongo database user's username]


<!--[if !supportLists]-->· <!--[endif]-->PASSWORD [Password]


<!--[if !supportLists]-->· <!--[endif]-->MONGO_INSTALL_BIN_DIR [Mongo installed bin directory]


<!--[if !supportLists]-->· <!--[endif]-->HOSTNAME [List of all servers (get it by execute rs.status() command on mongo client) separated with ;]


<!--[if !supportLists]-->· <!--[endif]-->DEFAULT_HOST_PORT [Default server setting for health check]


USERNAME=username


PASSWORD=password


MONGO_INSTALL_BIN_DIR=/usr/local/mongo/bin/


DEFAULT_HOST_PORT=hostname:27017


HOSTNAME="host1:27017;host2:27017;host3:27017"


Step to configure mongo-health-chk.sh


Open the mongo-health-chk.sh and correct the path for the mongo.properties


For e.g. . /usr/local/mongo.properties


Setting the CRON job


To execute the script as cron job, run below command on shell prompt


#crontab -e


This will open the cron job file. Make an entry of below cron expression [update the script location path]. This will run the script in every 5 min. you need to pass the log file name/path as parameter while running this script


*/5 0 * * * /path/to/script/mongo-health-chk.sh mongo_health.log




Below is the script







#!/bin/bash


. /home/mongo/mongo.properties


exec >> $1


#################################################################


# CHECK THE HEALTH OF ALL SERVER IN REPLICATION #


# version:- 1.0 #


# Author:- Chandrashekhar Dehankar #


# THIS SCRIPT WILL SEARCH OUT FOR ANY WORKING SERVER LISTED #


# IN mongo.properties. EXECUTES THE rs.status() COMMAND AND #


# */5 * * * * /home/mongo/mongo-health-chk.sh health.log #


# CRON EXPRESSION EVERY 5MIN IS BELOW #


#################################################################


#DATABASE HOST-PORT DEFAULT SETTING


HOST_PORT=$DEFAULT_HOST_PORT


#METHOD TO CHECK SERVER HEALTH


function _do_health_check {


#FIND ANY WORKING SERVER


DATABASE_HOST_SERVER=$HOSTNAME


DATABASE_HOST_SERVER=${DATABASE_HOST_SERVER//;/$'\n'}


for host in $DATABASE_HOST_SERVER



do


#CHECK WEATHER ITS IS UP OR NOT WHAT, NO MATTERS PRIMARY OR SECONDARY


IS_SERVER_UP=$($MONGO_INSTALL_BIN_DIR/mongo $host/admin -u $USERNAME -p $PASSWORD --eval "printjson(db.isMaster().secondary)" )



#FILTER THE RESULT


IS_SERVER_UP=`echo $IS_SERVER_UP grep -o '[^ ]*$'`




if [ "$IS_SERVER_UP" == "true" ] [ "$IS_SERVER_UP" == "false" ]; then




HOST_PORT=$host


break


fi


done



count=0


CURRENT_DATE=`date +%Y%m%d_%H-%M-%S`


echo "--------HEALTH TEST START[$CURRENT_DATE] ----------"



while [ $count -ne -1 ]


do




HOST_NAME=$($MONGO_INSTALL_BIN_DIR/mongo $HOST_PORT/admin -u $USERNAME -p $PASSWORD --eval "printjson(rs.status().members[$count].name)" )


HOST_NAME=`echo $HOST_NAME grep -o '[^ ]*$'`


CHECK_HEALTH=$($MONGO_INSTALL_BIN_DIR/mongo $HOST_PORT/admin -u $USERNAME -p $PASSWORD --eval "printjson(rs.status().members[$count].health)" )


CHECK_HEALTH=`echo $CHECK_HEALTH grep -o '[^ ]*$'`




count=$(( $count + 1 ))


#CHECK ALL THE NODES IN REPLICATION


if [ "$CHECK_HEALTH" != "1" ] && [ "$CHECK_HEALTH" != "0" ]; then


echo "..........All SERVER HEALTH TEST DONE.........."


count="-1"


fi


if [ $count -ne -1 ]; then


echo "----------------------------------------------------------"


if [ "$CHECK_HEALTH" == "1" ]; then


echo $HOST_NAME IS WORKING FINE


else



echo $HOST_NAME IS DOWN


fi


echo "----------------------------------------------------------"


fi


done


}


## check the health


_do_health_check






Mongo Database Backup Script

Description

This script use for taking the mongo database backup using --oplog option, this script create The backup folder with current timestamp, also delete the backup folder older than 2 days for daily backup and delete the folder older than 30 days for weekly backup, This script will Run as part of cron job midnight everyday and once in week. This script find out the secondary server for the backup configured in mongo.properties. If no secondary found then it will use The primary for default backup. The default behavior is daily backup if no input provide.

Configure mongo.properties

In mongo.properties we need to set the below properties

· USERNAME [Backup user's username]

· PASSWORD [Password]

· DEFAULT_HOST_PORT [Default server set for backup]

· DEFAULT_BACKUP_DESTINATION [ Path to backup root folder ]

· MONGO_INSTALL_BIN_DIR [mongo installation bin directory path ]

· HOSTNAME [List of all servers(get it by execute rs.status() command on mongo client) separated with ; ]


USERNAME=username

PASSWORD=password

DEFAULT_HOST_PORT=hotsname:27017

DEFAULT_BACKUP_DESTINATION=backup/dir/name

MONGO_INSTALL_BIN_DIR=mongo/install/dir/bin

HOSTNAME="host1:27017;host2:27017;host3:27017"


Configure mongo-backup.sh

Open the mongo-backup.sh and set the path for the mongo.properties

for e.g. . /path/to/properties/mongo.properties


Setting the CRON job

To execute the script as cron job, run below command on shell prompt

#crontab -e

This will open the cron job file. For daily midnight backup make entry of below cron expression [update the script location path]

0 0 * * * /PATH/TO/SCRIPT/mongo-backup.sh daily

For weekly backup on Sunday (Sunday = 0, Monday = 1......) at midnight make entry of below cron expression [update the script location path]

0 0 * * 0 /PATH/TO/SCRIPT/mongo-backup.sh weekly


The user who is going to run this script should have the permission [ read /write] to mongo.properties file and backup folder and script.


Here is the script


# DATABASE HOST-PORT DEFAULT SETTING

HOST_PORT=$DEFAULT_HOST_PORT

#NO OF DAY FOR DELETE OLD BACKUP [DEFAULT VALUE]

NO_OF_DAY=2

CHECK_BACKUP_TYPE=$1


#METHOD TO TAKE MONGO BACKUP

function _do_backup {


# CREATING FOLDER WITH CURRENT TIME STAMP

BACKUP_CURRENT_DATE=`date +%Y%m%d_%H-%M-%S`

BACKUP_CURRENT_DATE="$(echo ${BACKUP_CURRENT_DATE} | tr ' ' '0')"

UNIQ_DIR=""


if [ "$CHECK_BACKUP_TYPE" == "daily" ] || [ "$CHECK_BACKUP_TYPE" == "" ]; then

BACKUP_FOLDER=DAILY_BACKUP/

BACKUP_DESTINATION=$DEFAULT_BACKUP_DESTINATION$BACKUP_FOLDER

UNIQ_DIR=$BACKUP_DESTINATION$BACKUP_CURRENT_DATE


elif [ "$CHECK_BACKUP_TYPE" == "weekly" ]; then

BACKUP_FOLDER=WEEKLY_BACKUP/

BACKUP_DESTINATION=$DEFAULT_BACKUP_DESTINATION$BACKUP_FOLDER

UNIQ_DIR=$BACKUP_DESTINATION$BACKUP_CURRENT_DATE


fi

mkdir -p $UNIQ_DIR

#FIND SECONDARY SERVER

DATABASE_HOST_SERVERS=$HOSTNAMES

DATABASE_HOST_SERVERS=${DATABASE_HOST_SERVERS//;/$'\n'}

for host in $DATABASE_HOST_SERVERS

do

IS_SECONDAY=$(mongo $host/admin -u $USERNAME -p $PASSWORD --eval "printjson(db.isMaster().secondary)" )

#FILTER THE RESULT

IS_SECONDAY=`echo $IS_SECONDAY | grep -o '[^ ]*$'`

if [ "$IS_SECONDAY" == "true" ]; then

HOST_PORT=$host

break

fi


done


#execute mongo dump command

#$MONGO_INSTALLATION_BIN --host $HOST_PORT -u $USERNAME -p $PASSWORD --oplog --out $UNIQ_DIR

$MONGO_INSTALL_BIN_DIR/mongodump --host $HOST_PORT -u $USERNAME -p $PASSWORD --oplog --out $UNIQ_DIR


}


# THIS METHOD WILL DELETE THE FOLDER OLDER THAN 2 DAY

function _do_delete_old_backup {


if [ "$CHECK_BACKUP_TYPE" == "daily" ] || [ "$CHECK_BACKUP_TYPE" == "" ]; then

NO_OF_DAY=2

echo "deleting daily older backup !!!!!!!!!!!!!!!!!!!!!!"

else

NO_OF_DAY=30

echo "deleting weekly older backup !!!!!!!!!!!!!!!!!!!!!!"

fi

find $BACKUP_DESTINATION -type d -mtime +$NO_OF_DAY -exec rm -rf {} \;


}

## take the backup

_do_backup

## delete the old folder

_do_delete_old_backup