Uncategorized

Bash Tips and Tricks

During my day at Jana, when I’m not writing Python, I’m likely writing a Bash script. #!/bin/bash and whatever may follow is integral to our infrastructure. A Bash script at Jana tends to live in one of two places: analytics or deployment. After writing Bash scripts for both purposes, I’ve learned a few tips and tricks from our codebase.

Simplify stdin

We execute SQL queries from Bash scripts in two different ways.

One could could put a query into a query.sql file somewhere on disk, and then feed that file to the mysql command:

#!/bin/bash
mysql -u sam -h 127.0.0.1 -pSecurePassword db < sql_queries/query.sql

But Bash’s here documents remove layer of indirection caused by the filesystem. They provide an easy and elegant way of writing a query directly in the script itself. Then there’s no need for a sql_queries directory or any .sql files! The here document will send its contents to stdin:

#!/bin/bash
mysql -u sam -h 127.0.0.1 -pSecurePassword db <<QUERY
    select * from data where country = “USA” and user = “sam” and company = “jana”;
QUERY

(Feel free to imagine a longer and more complicated query here)

The abstract syntax of a here document looks like this:

any_command <<SOME_DELIMITER
    any sort of input that you would normally give to a command via stdin
SOME_DELIMITER

I like Bash’s here documents because put the SQL queries (or any input to any command) right in front of you; there’s no need to jump between files or worry about getting file paths correct. It does clutter the script itself, so you may want to use here documents for shorter queries.

The “test” cheat-sheet

As I dove into Bash scripting, I was somewhat regularly confounded by lines like this:

[ -f /etc/default/jana ] && . /etc/default/jana

Typing bash -f into Google isn’t very helpful. But typing man test into the terminal is very helpful. Code that uses square brackets and -f, -ne, etc. is actually an invocation of the test command. And its man page is very helpful for Bash scripting. I find myself referring to it often.

Deployment Magic

The combination of Linux’s /etc/init.d services and AWS is really powerful and cool. With an /etc/init.d compliant init script for our services, we can safely restart an EC2 instance with the guarantee that the operating system will restart our services as well. Lately we have been focusing on automating more aspects of our deployments, so writing scripts that can live in /etc/init.d is a big win!

The raw building blocks for such a script can be found in the file /etc/init.d/skeleton. And other details for how your script should behave can be found here: http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html

Linux treats an exit status of 0 from these scripts as a success and anything non-zero as a failure. I’ve found this Bash idiom to be handy when dealing with multiple services or commands:

start_my_service
service_exit_code=$?
start_my_second_service
service_2_exit_code=$?
[ $service_exit_code -eq 0 ] || exit $service_exit_code
[ $service_2_exit_code -eq 0 ] || exit $service_exit_code
exit 0

$? is a special Bash incantation to get the exit code of the most recently run command. Testing that both exit codes are 0 and exiting if either one is non-zero will satisfy the operating system.

Make sure what you want to do is possible!

Sometimes when writing any code, the differences between your production environment and your development environment might not be totally obvious. You might be dealing with OS X versus a Linux distro; working inside of a virtualenv or not; or having the default shell on your development machine be zsh whereas in production its actually sh (which has personally bit me :)).

To avoid init scripts inexplicably failing and subsequent head scratching, add a line like this to the beginning of your script, before you invoke whatever commands you have in mind:

command -v celery >/dev/null 2>&1 || { echo >&2 "Celery not installed.  Aborting."; exit 1; }

This will test to see that the shell can find and execute the celery command without fail. If this fails, then you will likely need to specify an absolute path for the executable you want or think more carefully about your environment.

Discussion

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s