hax.rb

Where’s the bloat: Tracking ‘require’ memory usage in Ruby and Rails

Among the other awesome things that Ruby Enterprise Edition’s inclusion of the RailsBench GC patch provides is the addition of the allocated_size method on Ruby’s GC constant.

Eric Lindvall recently devised a very clever use for this GC.allocated_size method:

If I wanted to see what was contributing to the large memory footprint of an application on startup, tracking how much memory was allocated during each require would give me a good place to start.

Given the size of GC.allocated_size before and after require calls, we can determine the initial memory profile of any ruby gem, library, file, or Rails model.

To use this to determine the initial memory profile of a beefy Rails application, you’ll first want to download his require_tracking hax. Then, load your production Rails application’s environment with require_tracking turned on:

RAILS_ENV=production ruby -r/tmp/require_tracking.rb  -e "require 'config/environment'"
Memory used by file:
                                     File KB
                            ------------- --------
                      config/environment: 209,623
/srv/myapp/releases/20091230003757/app/models/user: 73,653
/srv/myapp/releases/20091230003757/app/models/store: 70,962
                           ./config/boot: 15,021
/srv/myapp/releases/20091230003757/app/models/discount_code: 9,813
/srv/myapp/releases/20091230003757/app/models/order: 9,390
./config/../vendor/rails/railties/lib/initializer: 9,348
...

71.9 M used by the User model? Holy wow! I know what I’ll be doing today….

A quick way to find slow Rails actions

redsquirrel:

tail -10000 log/production.log | egrep "Completed in [0-9]{2,}"

via RailsMachine support

Using Screen to manage multiple persistent SSH connections

In the scope of my work at Rails Machine, I manage a lot of servers. Thousands, even. That many servers means that I often have dozens of SSH connections open at a time; some tailing log files, some running top, vmstat, or the likes, and some just waiting at a shell prompt for a rapid fire command.

To make things more complicated, I’m not always in the same place all day. Between my home office, the datacenter, Starbucks, etc, my laptop is open and shut plenty of times a day. SSH timeouts used to be my worst enemy. Enter screen…

Put simply, screen is a window manager for virtual terminals. To create a new window in screen, type C-a, c. That means hold down Control, press a, release Control, and press c. To switch between windows, press C-a, space. You can even switch to numbered windows with C-a, [0-9].

Screen is incredibly useful locally as a replacement for tabbed windows in Terminal.app, but it really shines when run from an always-on server. In my case, I run screen on a gateway/firewall server on the Rails Machine internal network, but this will work wherever you can keep an always-on server with access to all the servers you need. With an always-on screen session:

• I never have to worry about SSH timeouts. If run inside a screen session, I know a command will persist forever. If my SSH connection to the gateway times out, all that’s needed is a quick screen -D -RR and my previous session is back up and running.
• I can access the same screen session from multiple computers. If I run downstairs for a quick sammich and want to check on the progress of a test suite run, I can login to the gateway from my kitchen computer, run screen -D -RR, and grab control of the screen session from my laptop upstairs. Everything is exactly as I left it.

All of this is fine and dandy, but a while back I started noticing a common pattern emerging in my use of screen: I’d open a new window (C-a, c), then ssh to a server. Over and over again. Two steps. Then, once my session was done, I’d exit the SSH connection, then exit the local shell again. Again, two steps. That’s one step too many, twice. No good. So, after a little research, I created a custom ssh function in my ~/.bashrc:

function ssh() {
  eval last_arg=\$$#
  screen -t "$last_arg" ssh "$@";
}

Now, when I run ssh user@foo inside a screen session, a new window opens for the SSH connection with it’s title set to ‘user@foo.’ All arguments passed to SSH work as intended, but the screen title only gets the user@host portion of the command. When this SSH session is terminated, the screen window is automatically closed - no need to terminate the local shell containing the SSH process. Nice!

By default, screen doesn’t display the title of all windows, but if you toss this line into your ~/.screenrc, you’ll get a nice status line at the bottom of your screen with the number and title of each open window:

hardstatus alwayslastline "%{gk}[%{G}%H%{g}][%=%{wk}%?%-Lw%?%{R}[%{W}%n*%t%?(%u)%?%{R}]%{w}%?%+Lw%?%?%=%{g}][%{B}%m/%d %{G}%C%a%{g}]"

Put this all together, and you get a screen session that looks like this:

When you need to SSH to a new host, just switch back to the gateway window by typing C-a, 0, type ssh user@host, and a new window will be opened for this SSH connection. The gateway window is left clean, ready to initiate another SSH connection when needed.

As an added bonus, if you toss this in your ~/.bashrc, you’ll get fancy tab-completion on hosts in your ~/.ssh/known_hosts file:

SSH_COMPLETE=( $(cat ~/.ssh/known_hosts | cut -f 1 -d ' ' | sed -e s/,.*//g | uniq | egrep -v [0123456789]) )
complete -o default -W "${SSH_COMPLETE[*]}" ssh

Sick, eh?