Follow me

Saturday, March 24, 2012

Mongo Monitor Script: Health Check

Description


The Script will search for any working server listed in the mongo.properties. It executes the rs.status() command and check the health of all the server in replication


Step to configure mongo.properties


In mongo.properties we need to set the below properties


<!--[if !supportLists]-->· <!--[endif]-->USERNAME [mongo database user's username]


<!--[if !supportLists]-->· <!--[endif]-->PASSWORD [Password]


<!--[if !supportLists]-->· <!--[endif]-->MONGO_INSTALL_BIN_DIR [Mongo installed bin directory]


<!--[if !supportLists]-->· <!--[endif]-->HOSTNAME [List of all servers (get it by execute rs.status() command on mongo client) separated with ;]


<!--[if !supportLists]-->· <!--[endif]-->DEFAULT_HOST_PORT [Default server setting for health check]


USERNAME=username


PASSWORD=password


MONGO_INSTALL_BIN_DIR=/usr/local/mongo/bin/


DEFAULT_HOST_PORT=hostname:27017


HOSTNAME="host1:27017;host2:27017;host3:27017"


Step to configure mongo-health-chk.sh


Open the mongo-health-chk.sh and correct the path for the mongo.properties


For e.g. . /usr/local/mongo.properties


Setting the CRON job


To execute the script as cron job, run below command on shell prompt


#crontab -e


This will open the cron job file. Make an entry of below cron expression [update the script location path]. This will run the script in every 5 min. you need to pass the log file name/path as parameter while running this script


*/5 0 * * * /path/to/script/mongo-health-chk.sh mongo_health.log




Below is the script







#!/bin/bash


. /home/mongo/mongo.properties


exec >> $1


#################################################################


# CHECK THE HEALTH OF ALL SERVER IN REPLICATION #


# version:- 1.0 #


# Author:- Chandrashekhar Dehankar #


# THIS SCRIPT WILL SEARCH OUT FOR ANY WORKING SERVER LISTED #


# IN mongo.properties. EXECUTES THE rs.status() COMMAND AND #


# */5 * * * * /home/mongo/mongo-health-chk.sh health.log #


# CRON EXPRESSION EVERY 5MIN IS BELOW #


#################################################################


#DATABASE HOST-PORT DEFAULT SETTING


HOST_PORT=$DEFAULT_HOST_PORT


#METHOD TO CHECK SERVER HEALTH


function _do_health_check {


#FIND ANY WORKING SERVER


DATABASE_HOST_SERVER=$HOSTNAME


DATABASE_HOST_SERVER=${DATABASE_HOST_SERVER//;/$'\n'}


for host in $DATABASE_HOST_SERVER



do


#CHECK WEATHER ITS IS UP OR NOT WHAT, NO MATTERS PRIMARY OR SECONDARY


IS_SERVER_UP=$($MONGO_INSTALL_BIN_DIR/mongo $host/admin -u $USERNAME -p $PASSWORD --eval "printjson(db.isMaster().secondary)" )



#FILTER THE RESULT


IS_SERVER_UP=`echo $IS_SERVER_UP grep -o '[^ ]*$'`




if [ "$IS_SERVER_UP" == "true" ] [ "$IS_SERVER_UP" == "false" ]; then




HOST_PORT=$host


break


fi


done



count=0


CURRENT_DATE=`date +%Y%m%d_%H-%M-%S`


echo "--------HEALTH TEST START[$CURRENT_DATE] ----------"



while [ $count -ne -1 ]


do




HOST_NAME=$($MONGO_INSTALL_BIN_DIR/mongo $HOST_PORT/admin -u $USERNAME -p $PASSWORD --eval "printjson(rs.status().members[$count].name)" )


HOST_NAME=`echo $HOST_NAME grep -o '[^ ]*$'`


CHECK_HEALTH=$($MONGO_INSTALL_BIN_DIR/mongo $HOST_PORT/admin -u $USERNAME -p $PASSWORD --eval "printjson(rs.status().members[$count].health)" )


CHECK_HEALTH=`echo $CHECK_HEALTH grep -o '[^ ]*$'`




count=$(( $count + 1 ))


#CHECK ALL THE NODES IN REPLICATION


if [ "$CHECK_HEALTH" != "1" ] && [ "$CHECK_HEALTH" != "0" ]; then


echo "..........All SERVER HEALTH TEST DONE.........."


count="-1"


fi


if [ $count -ne -1 ]; then


echo "----------------------------------------------------------"


if [ "$CHECK_HEALTH" == "1" ]; then


echo $HOST_NAME IS WORKING FINE


else



echo $HOST_NAME IS DOWN


fi


echo "----------------------------------------------------------"


fi


done


}


## check the health


_do_health_check






Mongo Database Backup Script

Description

This script use for taking the mongo database backup using --oplog option, this script create The backup folder with current timestamp, also delete the backup folder older than 2 days for daily backup and delete the folder older than 30 days for weekly backup, This script will Run as part of cron job midnight everyday and once in week. This script find out the secondary server for the backup configured in mongo.properties. If no secondary found then it will use The primary for default backup. The default behavior is daily backup if no input provide.

Configure mongo.properties

In mongo.properties we need to set the below properties

· USERNAME [Backup user's username]

· PASSWORD [Password]

· DEFAULT_HOST_PORT [Default server set for backup]

· DEFAULT_BACKUP_DESTINATION [ Path to backup root folder ]

· MONGO_INSTALL_BIN_DIR [mongo installation bin directory path ]

· HOSTNAME [List of all servers(get it by execute rs.status() command on mongo client) separated with ; ]


USERNAME=username

PASSWORD=password

DEFAULT_HOST_PORT=hotsname:27017

DEFAULT_BACKUP_DESTINATION=backup/dir/name

MONGO_INSTALL_BIN_DIR=mongo/install/dir/bin

HOSTNAME="host1:27017;host2:27017;host3:27017"


Configure mongo-backup.sh

Open the mongo-backup.sh and set the path for the mongo.properties

for e.g. . /path/to/properties/mongo.properties


Setting the CRON job

To execute the script as cron job, run below command on shell prompt

#crontab -e

This will open the cron job file. For daily midnight backup make entry of below cron expression [update the script location path]

0 0 * * * /PATH/TO/SCRIPT/mongo-backup.sh daily

For weekly backup on Sunday (Sunday = 0, Monday = 1......) at midnight make entry of below cron expression [update the script location path]

0 0 * * 0 /PATH/TO/SCRIPT/mongo-backup.sh weekly


The user who is going to run this script should have the permission [ read /write] to mongo.properties file and backup folder and script.


Here is the script


# DATABASE HOST-PORT DEFAULT SETTING

HOST_PORT=$DEFAULT_HOST_PORT

#NO OF DAY FOR DELETE OLD BACKUP [DEFAULT VALUE]

NO_OF_DAY=2

CHECK_BACKUP_TYPE=$1


#METHOD TO TAKE MONGO BACKUP

function _do_backup {


# CREATING FOLDER WITH CURRENT TIME STAMP

BACKUP_CURRENT_DATE=`date +%Y%m%d_%H-%M-%S`

BACKUP_CURRENT_DATE="$(echo ${BACKUP_CURRENT_DATE} | tr ' ' '0')"

UNIQ_DIR=""


if [ "$CHECK_BACKUP_TYPE" == "daily" ] || [ "$CHECK_BACKUP_TYPE" == "" ]; then

BACKUP_FOLDER=DAILY_BACKUP/

BACKUP_DESTINATION=$DEFAULT_BACKUP_DESTINATION$BACKUP_FOLDER

UNIQ_DIR=$BACKUP_DESTINATION$BACKUP_CURRENT_DATE


elif [ "$CHECK_BACKUP_TYPE" == "weekly" ]; then

BACKUP_FOLDER=WEEKLY_BACKUP/

BACKUP_DESTINATION=$DEFAULT_BACKUP_DESTINATION$BACKUP_FOLDER

UNIQ_DIR=$BACKUP_DESTINATION$BACKUP_CURRENT_DATE


fi

mkdir -p $UNIQ_DIR

#FIND SECONDARY SERVER

DATABASE_HOST_SERVERS=$HOSTNAMES

DATABASE_HOST_SERVERS=${DATABASE_HOST_SERVERS//;/$'\n'}

for host in $DATABASE_HOST_SERVERS

do

IS_SECONDAY=$(mongo $host/admin -u $USERNAME -p $PASSWORD --eval "printjson(db.isMaster().secondary)" )

#FILTER THE RESULT

IS_SECONDAY=`echo $IS_SECONDAY | grep -o '[^ ]*$'`

if [ "$IS_SECONDAY" == "true" ]; then

HOST_PORT=$host

break

fi


done


#execute mongo dump command

#$MONGO_INSTALLATION_BIN --host $HOST_PORT -u $USERNAME -p $PASSWORD --oplog --out $UNIQ_DIR

$MONGO_INSTALL_BIN_DIR/mongodump --host $HOST_PORT -u $USERNAME -p $PASSWORD --oplog --out $UNIQ_DIR


}


# THIS METHOD WILL DELETE THE FOLDER OLDER THAN 2 DAY

function _do_delete_old_backup {


if [ "$CHECK_BACKUP_TYPE" == "daily" ] || [ "$CHECK_BACKUP_TYPE" == "" ]; then

NO_OF_DAY=2

echo "deleting daily older backup !!!!!!!!!!!!!!!!!!!!!!"

else

NO_OF_DAY=30

echo "deleting weekly older backup !!!!!!!!!!!!!!!!!!!!!!"

fi

find $BACKUP_DESTINATION -type d -mtime +$NO_OF_DAY -exec rm -rf {} \;


}

## take the backup

_do_backup

## delete the old folder

_do_delete_old_backup



















Friday, February 10, 2012

GPARS Concurrent Collection Processor In Grails Integration Test

Recently I came across one scenario in which we have to test one service in grails application

We want this to be done via multithreading so that multiple thread can execute the service method concurrently using grails integration test

So the very effective solution we found was GPARS

In GPars the GParsPool and GParsExecutorsPool classes give you access to low-level data parallelism techniques

This utility gives you facility of processing collections concurrently. So the size of collection will be the

No. of user for us and the collection data can be use as param pass to the service method

Below is the sample code chunk.

You need to import the below class in you class

import com.mongodb.gridfs.GridFS

also need to configure NO_OF_THREADS require for processing

suppose we have the list of size 1000 containing the user ID as string data

def userList = new ArrayList(1000);

userList.add(“1”)

Now in test Method we write the Closure which will process our collection as shown below.

GParsPool.withPool(NO_OF_THREADS){

final Closure processUser = {userId, index ->

def response = service.processUser(userId)

assertNotNull response

}

assertNotNull(userList)

userList.eachWithIndex(processUser.async())

}

This will work as concurrent collection processor for our userList with given NO_OF_THREADS . we can pass the index and its value as parameter to the service method.

Install the MYSQL along with the InnoDB plugin

Introduction

This blog represent the procedure to install the MYSQL along with the InnoDB plugin.

InnoDB can be install from SQL INSTALL command or by configuring my.cnf, here we are using my.cnf to load and configure InnoDB plugin.

NOTE: If you already have MYSQL install and just want to install the InnoDB plugin please don't follow these steps.

Download MYSQL

Download the MYSQL packages using below linux command

Note : User must have download rights to download these packages

MySQL-client-community :- The standard MySQL client programs. You probably always want to install this package.

wget http://downloads.skysql.com/archives/mysql-5.1/MySQL-client-community-5.1.54-1.rhel5.x86_64.rpm

MySQL-server-community :- The MySQL server. You need this unless you only want to connect to a MySQL server running on another machine.

wget http://downloads.skysql.com/archives/mysql-5.1/MySQL-server-community-5.1.54-1.rhel5.x86_64.rpm


Create or Modify before mysqlserver is installed /etc/my.cnf

Below is the standard MYSQL my.cnf configuration referred from QA database server

[mysqld] configuration

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

#Default to using old password format for compatibility with mysql 3.x

#clients (those using the mysqlclient10 compatibility package).

#you can/should enable old passwords by setting old_passwords=1 else comment it

old_passwords=1

#Max no of clients to connect

max_connections=750

#unique id for the server

server-id=1

#Innodb configuration

innodb_flush_method=O_DIRECT

#set these value as per memory available

innodb_buffer_pool_size=6G

innodb_flush_log_at_trx_commit=1

#set these value as per memory available

innodb_log_file_size=64M

innodb_log_files_in_group=2

ignore_builtin_innodb

#Be careful while copying.There should be no space

plugin-load=innodb=ha_innodb_plugin.so;innodb_trx=ha_innodb_plugin.so;innodb_locks=ha_innodb_plugin.so;innodb_lock_waits=ha_innodb_plugin.so;innodb_cmp=ha_innodb_plugin.so;innodb_cmp_reset=ha_innodb_plugin.so;innodb_cmpmem=ha_innodb_plugin.so;innodb_cmpmem_reset=ha_innodb_plugin.so

default-storage-engine=InnoDB

[mysql.server] configuration

[mysql.server]

user=mysql

[mysqld_safe] configuration

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

Installation

Install it using below command

rpm -ivh MySQL-client-community-5.1.54-1.rhel5.x86_64.rpm

rpm -ivh MySQL-server-community-5.1.54-1.rhel5.x86_64.rpm

[edit]

Set The Password

Once you have install all the package's please remember to set password for the MySQL root USER ! To do so, issue the below linux commands:

/usr/bin/mysqladmin -u root password 'new-password'

Note: above mysqladmin command will execute only when mysql server is up, here installation of MySQL-server-community-5.1.54-1.rhel5.x86_64.rpm package will start the server. if its not running then start the server using command

/etc/init.d/mysql start

Create User

execute below commands on Mysql

Mysql>create user 'userName'@'hostName' identified by 'password';

Mysql>GRANT SELECT, INSERT, UPDATE, DELETE, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, SHOW VIEW ON databaseName.* TO 'userName'@'hostName'

Note: hostName here is the remote(app server) machine's host name


Thursday, February 9, 2012

Xtrabackup Installation and Restore For MYSQL + INNODB

Installation

a) Using Percona Software Repositories

Run below command. This will create the Percona YUM repository

sudo rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm

Testing the Repository: This will display the repository list

sudo yum list | grep percona

Check the Xtrabackup install or not

rpm -qi xtrabackup

If it is not installed then install using below command and verify again using above command

sudo yum install xtrabackup

b) Using Generic .tar.gz binary packages

If you don’t want to install using Percona YUM repository use the below link to download ready to use Xtrabackup tool

http://www.percona.com/downloads/XtraBackup/XtraBackup-1.5/Linux/binary/

Note: - For Innodb database backup you must have innodb_file_per_table set in my.cnf.

Creating Backup

sudo ./innobackupex-1.5.1 --user=username --password=password /path/to/save/backup --databases=”databaseName”

Note: - the user should have the read/write access to the both data and target folder, also check .frm file copied or not in database backup folder

Backup Completes with following statements:

120209 01:57:22 innobackupex-1.5.1: completed OK!

Restore Backup

Prepare using binary package

Sudo ./innobackupex-1.5.1 --apply-log /path/to/save/backup/timestamp_folder

Copy Data directory

You could simply use cp to copy the files, also make sure the data directory and files are owned by the mysql user.

Stop the server and then run the below copy command

cp -R /path/to/save/backup/timestamp_folder/* /var/lib/mysql/

And run the below command

sudo chown –R mysql:mysql /var/lib/mysql

Start the server and check the database

Reference:

http://www.percona.com/doc/percona-xtrabackup/howtos/recipes_xbk_restore.html

http://agiletesting.blogspot.com/2010/09/mysql-innodb-hot-backups-and-restores.html

http://www.ovaistariq.net/590/on-hot-backups-and-restore-using-xtrabackup/