Bits and Bytes

In one of my recent projects, I needed to create a mirror server for my production server. The mirror was not to be used for any live activity, it was only
needed to serve as a development environment where the data and issues happening on production could be analyzed and debugged in isolation.
The databases used on the servers were MongoDB and MySQL.

One approach to achieve this could be to enable logging on the databases and setup a Master/Slave
relationship between the live and mirror databases. But this project could tolerate some time lag between live and mirror, so over working the databases
for this did not seem like a good move.

So I went about implementing a very basic solution using shell scripts.

Step 1: Created a shell script on the production server to take dump of both the databases and ftp them to the mirror server
Step 2: Setup a cron on production server to run this script periodically
Step 3: Created a shell script on mirror to restore the databases from the latest available dump ftped to it
Step 4: Setup a cron on mirror to run this script periodically

Here are the two scripts:

Production Script

# user local system defines
ADMINPATH=path_to_the_folder_to_be_used_for_dumps
MYSQL_USERNAME=mysql_username
MYSQL_PWD=mysql_password
MYSQL_DBNAME=mysql_database_name
MONGO_DBNAME=mongodb_database_name

# user mirror system ftp defines
HOSTNAME=mirror_ftp_server
USERNAME=mirror_ftp_username
PASSWORD=mirror_ftp_pwd
REMOTE_DUMP_PATH=remote_dump_path

DAY=`/bin/date +%Y%m%d_%H%M%S`

filename=$ADMINPATH/dumps/$DAY.sql
echo "Creating MYSQL DUMP:  $filename"
mysqldump -u$MYSQL_USERNAME -p$MYSQL_PWD $MYSQL_DBNAME > $filename

echo "Creating MONGO DUMP: mongo_$DAY.tar"
mongodump --db $MONGO_DBNAME --out $ADMINPATH/dumps/mongo_$DAY
cd $ADMINPATH/dumps/mongo_$DAY/
tar -cvzf "$ADMINPATH/dumps/mongo_$DAY.tar" $MONGO_DBNAME/

cd $ADMINPATH
rm -rf $ADMINPATH/dumps/mongo_$DAY

ftp -n $HOSTNAME << EOF
quote USER $USERNAME
quote PASS $PASSWORD

binary
cd $REMOTE_DUMP_PATH
lcd $ADMINPATH/dumps/
put $DAY.sql
put mongo_$DAY.tar
quit
EOF

Mirror Script

# user local system defines
DUMPDIR=path_to_the_folder_where_dumps_are_ftped
MYSQL_USERNAME=mysql_username
MYSQL_PWD=mysql_password
MYSQL_DBNAME=mysql_database_name
MONGO_DBNAME=mongodb_database_name

latestsqldump=`ls -t -r $DUMPDIR/*.sql | tail -n 1`
echo "Going to restore MYSQL from $latestsqldump"
mysql -u$MYSQL_USERNAME -p$MYSQL_PWD $MYSQL_DBNAME < $latestsqldump

latestmongodump=`ls -t -r $DUMPDIR/*.tar | tail -n 1`
echo "Going to restore MONGO DB from $latestmongodump"
rm -rf $DUMPDIR/tmp
mkdir $DUMPDIR/tmp

tar -C $DUMPDIR/tmp/ -xvzf $latestmongodump
mongorestore --db $MONGO_DBNAME $DUMPDIR/tmp/$MONGO_DBNAME/
rm -rf $DUMPDIR/tmp

For all the power they have, they could also be the fastest way to get you fired (or close to it)

If you have written a shell script of some potent, i.e. major server cleanup, patch application across multiple installations etc, the first thing you must do in your script is to bind it to the context. Make sure you (or any one else) don’t end up running it by mistake in a wrong context. (Forget all those generic programming lessons )

Do not perform undoable actions with the script. For e.g. if you got tons of files to replace with a newer version, get your script to make a backup archive for each run. Before replacing, put them in the archive. Let the archive deletion be done later after a manual review.

Will try to post some real scripts and goof-up details to add spice

[If you are a fiesty server programmer, you better take a print of this post and put it in your cabin]

Lost my old blog – much due to GoDaddy’s ridiculous hosting plans and my own laziness to move over the data in time. But anyhow, much has changed since the time I last updated by old blog. Gone are the days when you share your thoughts through your blog. Facebook, Twitter has taken that place very well.

So I am going to use this blog to share stuff which is not appropriate to be shared on any of the social platforms – coding stuff!