Bits and Bytes

Ignorance hurts and how. For past many days, I would often get an alert from a server that a critical process is not running any more. I would log into the server and see that the process mysteriously died without any trace of any action in the history, logs or anywhere.

One common observation I made in all these cases was that the last alive time of the killed process was always about half n hour later from the start of another periodic process (which I always found running when this main one got killed).

It was evident that the system was doing some kind of process management and killing – based on what it deemed best for the machine.

Found this link today about Linux’s OOM Killing mechanisms which kind of answered everything.

Taming the OOM killer [LWN.net].

Must read for anyone operating in the cloud space and spawning processes on the go.

I cringe everytime I see a website where the programmers have killed the usability only to prove how much can be achieved in javascript. JS is like a double edged sword. One has to know the balance. One has to understand what makes sense to be pulled asynchronously. And the answer does not always lie in technology. It lies somewhere in the symantics of the operation.

In my company, I use a hosting solution from Firehost. Their support portal is a brilliant example of this. If you have to create a ticket, a new page loads to open up the form of ticket creation (notice: a static html is loaded via a new page load, you say alright). Now when you submit the form, the Javascript submits the form with no updates on the page until the request comes back. On submission of form, the Javascript redirects the page to show ‘Submitted Successfully’. (Yay!)

I come across such marvels frequently enough to get motivated to write about this. I think I may just keep adding the examples to this post and one day publish a chapter on What not to do in Javascript.

Brilliant discussion on JS Same Origin Policy and work arounds.

Came across when I was trying to implement a cross domain JS procedure calls through embedded iFrames.

javascript – Ways to circumvent the same-origin policy – Stack Overflow.

Linux: Download all file from ftp server recursively.

Very often I need to move data directories from one server to another. Normal FTP wouldn’t go recursive into the directories. This is where ncFtp shines. Refer to the link above to see the usage.

In one of my recent projects, I needed to create a mirror server for my production server. The mirror was not to be used for any live activity, it was only
needed to serve as a development environment where the data and issues happening on production could be analyzed and debugged in isolation.
The databases used on the servers were MongoDB and MySQL.

One approach to achieve this could be to enable logging on the databases and setup a Master/Slave
relationship between the live and mirror databases. But this project could tolerate some time lag between live and mirror, so over working the databases
for this did not seem like a good move.

So I went about implementing a very basic solution using shell scripts.

Step 1: Created a shell script on the production server to take dump of both the databases and ftp them to the mirror server
Step 2: Setup a cron on production server to run this script periodically
Step 3: Created a shell script on mirror to restore the databases from the latest available dump ftped to it
Step 4: Setup a cron on mirror to run this script periodically

Here are the two scripts:

Production Script

# user local system defines
ADMINPATH=path_to_the_folder_to_be_used_for_dumps
MYSQL_USERNAME=mysql_username
MYSQL_PWD=mysql_password
MYSQL_DBNAME=mysql_database_name
MONGO_DBNAME=mongodb_database_name

# user mirror system ftp defines
HOSTNAME=mirror_ftp_server
USERNAME=mirror_ftp_username
PASSWORD=mirror_ftp_pwd
REMOTE_DUMP_PATH=remote_dump_path

DAY=`/bin/date +%Y%m%d_%H%M%S`

filename=$ADMINPATH/dumps/$DAY.sql
echo "Creating MYSQL DUMP:  $filename"
mysqldump -u$MYSQL_USERNAME -p$MYSQL_PWD $MYSQL_DBNAME > $filename

echo "Creating MONGO DUMP: mongo_$DAY.tar"
mongodump --db $MONGO_DBNAME --out $ADMINPATH/dumps/mongo_$DAY
cd $ADMINPATH/dumps/mongo_$DAY/
tar -cvzf "$ADMINPATH/dumps/mongo_$DAY.tar" $MONGO_DBNAME/

cd $ADMINPATH
rm -rf $ADMINPATH/dumps/mongo_$DAY

ftp -n $HOSTNAME << EOF
quote USER $USERNAME
quote PASS $PASSWORD

binary
cd $REMOTE_DUMP_PATH
lcd $ADMINPATH/dumps/
put $DAY.sql
put mongo_$DAY.tar
quit
EOF

Mirror Script

# user local system defines
DUMPDIR=path_to_the_folder_where_dumps_are_ftped
MYSQL_USERNAME=mysql_username
MYSQL_PWD=mysql_password
MYSQL_DBNAME=mysql_database_name
MONGO_DBNAME=mongodb_database_name

latestsqldump=`ls -t -r $DUMPDIR/*.sql | tail -n 1`
echo "Going to restore MYSQL from $latestsqldump"
mysql -u$MYSQL_USERNAME -p$MYSQL_PWD $MYSQL_DBNAME < $latestsqldump

latestmongodump=`ls -t -r $DUMPDIR/*.tar | tail -n 1`
echo "Going to restore MONGO DB from $latestmongodump"
rm -rf $DUMPDIR/tmp
mkdir $DUMPDIR/tmp

tar -C $DUMPDIR/tmp/ -xvzf $latestmongodump
mongorestore --db $MONGO_DBNAME $DUMPDIR/tmp/$MONGO_DBNAME/
rm -rf $DUMPDIR/tmp

For all the power they have, they could also be the fastest way to get you fired (or close to it)

If you have written a shell script of some potent, i.e. major server cleanup, patch application across multiple installations etc, the first thing you must do in your script is to bind it to the context. Make sure you (or any one else) don’t end up running it by mistake in a wrong context. (Forget all those generic programming lessons )

Do not perform undoable actions with the script. For e.g. if you got tons of files to replace with a newer version, get your script to make a backup archive for each run. Before replacing, put them in the archive. Let the archive deletion be done later after a manual review.

Will try to post some real scripts and goof-up details to add spice

[If you are a fiesty server programmer, you better take a print of this post and put it in your cabin]

Lost my old blog – much due to GoDaddy’s ridiculous hosting plans and my own laziness to move over the data in time. But anyhow, much has changed since the time I last updated by old blog. Gone are the days when you share your thoughts through your blog. Facebook, Twitter has taken that place very well.

So I am going to use this blog to share stuff which is not appropriate to be shared on any of the social platforms – coding stuff!