bash Tricks From the Developers of the O'Reilly Network

by Tony Stubblebine

Every week the developers of the O'Reilly Network (that's me, three
developers, and two admins) have a status meeting to check in with our key
managers, decide or rearrange priorities, and work through problems. This is a
dream meeting for managers, questions are answered and plans are laid.



It's fair to say that status meetings aren't a developer's dream. After
several years of weekly meetings, ours were feeling stale. So we agreed to end
the meetings with a round of tips and tricks. First up, bash tricks.



At the end of our meeting, the managers bailed and we stuck around to geek out
over the tricks that make our work easier. Everyone had something to
contribute. I can't recommend this enough! I've learned a lot from books like
Unix Power Tools. But by
sharing directly with your coworkers you get advice that's targeted directly
to the work you do.



Here were our gems, the most useful tips that weren't already common
knowledge among the developers.



pushd/popd


Bash will keep a history of the directories you visit, you just have to ask.
Bash stores the history in a stack and uses the commands pushd
and popd to manage the stack.



pushd dir - move the current directory onto the stack and change to
the dir directory.



popd - pops the top directory off of the stack and moves you into it.



We're opening files all over the file system, internal code, vendor code, templates, configuration files, logs. Because of this we like the ability to take a detour on the file system and still navigate back to our working directory of the day. I think these commands are so useful that I alias'd them in my .bashrc






alias cd="pushd"
alias bd="popd"





Now the cd command manages the stack for me as well as changing directories. Aliasing popd to bd is an easy to remember and easy to type way to move back up the stack, think "change dir" and "back dir".



History



Bash keeps a history of the commands you've run. My group was already comfortable with the up and down arrows to navigate the history, !! to repeat the last command, and !foo to repeat the last command starting with foo.



Our newest admin had a better way, CTRL-R. That does command auto completion. Repeatedly pressing CTRL-R lets you tab through matching commands.



Home/End



CTRL-A takes you to the beginning of the line and CTRL-E takes you to the end of the line. This is probably basic shell knowledge, but I'm probably (hopefully) not the only person who didn't know it.



For Loops


We've got a cluster of machines that we'll sometimes need to loop through. Here's an example from our admins that checks uptime across our cluster.



$ for s in `cat server.list`; do ssh $s uptime; done;



Working with the Previous Command



Sometimes you want to run several commands on the same file, like run ls before deciding if that's the file you want to edit.






ls -l /long/path/to/file.txt
vi /long/path/to/file.txt




Bash provides a shortcut (!$) that holds the last word from the previous command. So in the above you could just write vi !$.



If your last command had a typo you can fix the command and rerun it with this construct, ^foo^bar. That replaces the first occurrence of foo in your command with bar.



Bonus Tip: use Awk



Our admins seem to think awk is pretty useful. And my boss thought it was
so useful, he wrote a book
on it. I can't keep any of the awk syntax in my head beyond printing out a
column from a file.



The normal column delimiter is whitespace. So if you wanted to print out the seventh column in an Apache access log (that's the request url in my logs) you could write:


cat access_log | awk '{print $7}'



You can change the delimiter with -F. So if you wanted to list all the users on your system, you could pull them out of /etc/passwd with:



$ cat /etc/passwd | awk -F: '{print $1}'



The /etc/passwd delimiter is :, which I've indicated
to awk with -F:.



We're doing vim tricks next.




What's your favorite bash trick? How else do you share tricks among developers?


17 Comments

andy-lester
2005-04-21 19:26:57
Last argument
You can also use Esc-period and get the last parm of the previous line. You can repeatedly use Esc-period to scroll back through time with them. That turns out to be even better than $! because you can edit it once it shows up on your command line.
merlyn
2005-04-21 22:03:42
This week's Useless Use of Cat Award Goes to...
Hey, I like your useless use of cat... perhaps you should read why they're useless.
precipice
2005-04-21 22:11:44
Excellent
Fantastic post. Great idea for improving meetings, and great tips.
FBoender
2005-04-22 00:40:26
More key bindings and tricks
Bash will keep a history of the directories you visit, you just have to ask.


You can also always go back to the previous directory you were in by typing cd - without the need to pushd the current directory. Using it more than once cycles between the current and previous directory.


CTRL-A takes you to the beginning of the line and CTRL-E takes you to the end of the line. This is probably basic shell knowledge,


I think it's actually common readline/emacs knowledge, and it works in much more programs than just Bash or a terminal. For instance, you can enable them in Gnome applications by adding the line
gtk-key-theme-name = "Emacs" to the ~/.gtkrc-2.0 file.


Other handy key bindings you can use are:

  • ctrl-u : Cut everything on the current line before the cursor.
  • ctrl-y : 'Yank' (paste) text that was cut using ctrl-u.
  • ctrl-w : Delete the word on the left of the cursor


There's so much usefull knowledge hidden in Bash that, if you spend any time at the command line, you should really get yourself aquinted with. It saves incredible ammounts of time.


Take for example something I wanted to do yesterday. I wanted to now the number of hits on a certain website. I could have installed a tool to parse the Apache access.log, but this was much easier:


$ cat access.log | cut -d"[" -f2 | cut -d"]" -f1 | cut -d"/" -f2 | uniq -c
28905 Mar
16554 Apr


Takes no more than a couple of seconds to write, but saves so much time.


Try reading through the Bash man page. It's huge, but think of all the stuff you'll learn! Or read some online Bash scripting tutorials. Everything from gathering statistics from files to creating thumbnails of images (From the top of my head: for A in *; do convert $A -resize 140x140 th_$A; done) becomes a cinch.

aristotle
2005-04-22 02:14:07
xargs and useless use of cat
for s in `cat server.list`; do ssh $s uptime; done


Do investigate xargs(1). The above example is easier written as


xargs -i ssh {} uptime < server.list


cat access_log | awk '{print $7}'


That’s a useless use of cat there. You can just say


awk '{print $7}' access_log


Some people say they prefer cat(1) because that way they can expand or change the pipe easier. In that case, you can use a little-known feature of redirection operators: you can put them anywhere you want, ie


< access_log awk '{print $7}'

mrchucho@yahoo.com
2005-04-22 04:28:19
vi key bindings
set -o vi


and you can navigate through the line using vi-style key bindings (e.g.g $ end of line, 0 start of line, etc.) Similarly, "ESC k" goes "up" in history and, "j" will go back down. "ESC /" will search history for a string. And so forth!

msporleder
2005-04-22 06:12:31
ugu.com
Get UNIX tips in your email every day, or enter some new ones to get emailed out to me and whoever else does this:
http://www.ugu.com/sui/ugu/show?I=tip.today
msporleder
2005-04-22 06:13:18
ugu.com
msporleder
2005-04-22 06:13:53
ugu.com
This would work better as a link.
tonystubblebine
2005-04-22 08:23:13
vi key bindings
This is a good one! Our other development team here (the people bringing you SafariU and the data that shows up on O'Reilly Radar) swear by it.


I'm a vi user which is why I didn't know the home/end keymappings for Bash. Probably a good idea for any vi users to make this switch.

tonystubblebine
2005-04-22 08:28:58
This week's Useless Use of Cat Award Goes to...
I'm honored! Thanks Randal =)
pga-eti
2005-04-22 10:38:20
should be !$
Instead of $!, use !$, it works much better. :)


$ echo asdf
asdf
$ echo !$
echo asdf
asdf
$ echo $!


$


So $! is an empty variable, while !$ brings back the last argument from the last command.

tonystubblebine
2005-04-22 10:43:07
should be !$
So true. I updated the entry with your correction. Thanks.
bingalls
2005-04-24 17:59:30
basename
basename also supports extensions. Try this, to rename *.xml to *.wsdl (or *.cxx to *.cpp, etc)



for i in *.xml;do;mv $i `basename $i .xml`.wsdl;done

I tried this in zsh, so bash may complain that I have an extra ;.
Be sure to upgrade to the latest bash, which has largely caught up with zsh features like **/* and programmable completion.
pkailasa
2005-04-28 07:45:02
Command substitution
$ for s in `cat server.list`; do ssh $s uptime; done;


Command substution is also done using $(command) notation, which I prefer to the backquotes. It allows commands to be nested (backquotes allow that too, but the inner quotes must be escaped using backslashes, which gets messy.



For example:

$ for s in $(cat server.list); do echo "$s: $(ssh $s uptime)"; done;

or:

# get the uptime for just the first server
$ echo "$(date): $(ssh $(head -1 server.list) uptime)"

rarue
2005-04-28 09:36:36
directory stack
Long ago having aliased "pu" and "po" to pushd and popd, I've moved on to showing my current directory stack with

alias d='dirs -v'

and rolling through the stack with

alias r='pushd +1'

and single numerical values aliased to go to that directory that comes from the dir listing, eg.

alias 3='pushd +3 > /dev/null ; dirs -v'

Another single letter alias I like is "p", which I have used for displaying and setting my PATH variable, but the code is lengthy so I won't burden you with how

p - 4
p --
p ++ /some/dir

does what you want.



dustbo
2005-05-22 19:16:37
No more worrying about cases
The best bash tip I can share is very helpful when working on systems that don't allow filenames to differ only in case (like OSX and Windows):
create a file called .inputrc in your home directory and put this line in it:
set completion-ignore-case on


Now bash tab-completion won't worry about case in filenames. Thus 'cd sit[tab]' would complete to 'cd Sites/'