Testing is one of the most important parts of software development and helps to ensure bugs don’t get into production and that code can be refactored safely. If you’re working on a team with multiple people with different skill sets, you might have people doing testing who only know windows and development is only using OSX or Linux. We want everyone to be able to test - someone in QA who is familiar with Windows shouldn’t have to throw away all that knowledge, install Linux, and start from scratch. Enter JRuby and John.

John is our tester and he is running windows. He wants to help make sure that when a user goes to http://google.com/ that a button appears with the text “Google Search”. The quick way to do this is to open his browser, navigate to http://google.com/ glance through the page for the button and confirm that its there. John has a problem though, he has 30 other test cases to run and the developers are pushing code to the frontpage several times a day; John now has to continously do this manually everytime code is touched and his test load is piling up.

So let’s help John out and install Sublime Text 2 and JRuby.

Start by downloading the 64-bit version of Sublime Text. Make sure to add the context menu when going through the install process.

Now we’ll visit the JRuby homepage and download the 64 bit installer.

Go through the installer and let JRuby set your path so you can access ruby from cmd.exe

Now when we open cmd.exe and type jruby -v we’ll be able to see that it was installed.

Now that we have our tools installed lets setup our test directory on the Desktop. Inside our testing folder we’ll create a folder called TestDemo for our tests for the Demo project.

Next we’ll open Sublime Text and go to File > Open Folder and navigate to our TestDemo folder and hit open.

Now we can continue making our directory structure inside Sublime Text. Since we’re going to use rspec we need to create a folder called spec to contain all of our tests. Right click on the TestDemo in the tree navigation and click New Folder.

Call the folder spec in the bottom title bar when it prompts you for the folder name.

Next we’ll create our Gemfile which will declare all of our dependencies - so make a file in the project root called Gemfile and put the our dependencies in it:

1source "https://rubygems.org"
2
3gem "rspec"
4gem "selenium"
5gem "selenium-webdriver"
6gem "capybara"

Once we have that file created, open cmd.exe and switch to your project’s root directory.

Type jgem install bundler to install bundler which manages ruby dependencies.

While still at the command prompt we’re going to bundle to install our dependencies:

After that finishes we need to run one last command for selenium to work properly: selenium install

We also need a spec_helper.rb file inside our spec directory.

1require "rspec"
2require "selenium"
3require "capybara/rspec"
4
5Capybara.default_driver =  :selenium

We’ve now setup our rspec folders, our Gemfile with dependencies, and installed them. Now we can write the test that will save John a ton of time.

Chrome comes with a simple tool to get XPath paths so we’re going to use that to get the XPath for the search button. Right click on the “Google Search” button and click Inspect element

Right click on the highlighted element and hit Copy XPath.

Now we’re going to make our spec file and call it homepage_spec.rb and locate it under spec\integration.

Here is a picture showing the directory structure and files:

Here is the spec file with comments explaining each part:

 1# This loads the spec helper file that we required everything in
 2require "spec_helper"
 3
 4# This is the outer level description of the test
 5# For this example it describes going to the homepage of Google.com
 6# Setting the feature type is necessary if you have
 7# Capybara specs outside of the spec\features folder
 8describe "Going to google.com", :type => :feature do
 9
10  # Context is like testing a specific component of the homepage, in this case
11  # its the search button
12  context "The search button" do
13    # This is our actual test where we give it a meaningful test description
14    it "should contain the text 'Google Search'" do
15      visit "http://google.com/" # Opens Firefox and visits google
16      button = find(:xpath, '//*[@id=gbqfba"') # find an object on the page by its XPath path
17      # This uses an rspec assertion saying that the string returned
18      # by button.text is equal to "Google Search"
19      button.text.should eq("Google Seearch")
20
21    end
22  end
23
24end

Now we can tab back to our cmd.exe prompt and run our tests! rspec spec will run all your tests under the spec folder.

Things to take note of

This example scenario is showing how to automate browser testing to do end-to-end tests on a product using rspec. This is by no means everything you can do with rspec and ruby - you can SSH, hit APIs and parse JSON, and do anything you want with the ability to make assertions.

A lot is going on in these examples - there are plenty of resources out there on google and other websites that provide more rspec examples and ruby examples.

We also showed how to add dependencies and install them using bundler. Two of the best resources for finding libraries and other gems is RubyGems and Ruby-Toolbox - the only thing to take note of is anything saying to be a native C extension (they won’t work with JRuby out of the box).

My last note is that you also need to have firefox installed as well - Selenium will work with Chrome but I’ve found it to be a hassle to setup (and unless you really need Chrome), the default of Firefox will work great.

Let’s start out with a simple directory structure:

1.
2├── plugin.rb
3├── main.rb
4└── plugins
5    ├── cat.rb
6    └── dog.rb
7
81 directory, 3 files

All the plugins we will use for our library will be loaded from plugins. Now lets make a simple Plugin class and register our plugins.

 1class Plugin
 2  # Keep the plugin list inside a set so we don't double-load plugins
 3  @plugins = Set.new
 4
 5  def self.plugins
 6    @plugins
 7  end
 8
 9  def self.register_plugins
10    # Iterate over each symbol in the object space
11    Object.constants.each do |klass|
12      # Get the constant from the Kernel using the symbol
13      const = Kernel.const_get(klass)
14      # Check if the plugin has a super class and if the type is Plugin
15      if const.respond_to?(:superclass) and const.superclass == Plugin
16        @plugins << const
17      end
18    end
19  end
20end

We’ve now made a simple class that will contain all of our plugin data when we call register_plugins.

Now for our Dog and Cat classes:

1class DogPlugin < Plugin
2
3  def handle_command(cmd)
4    p "Command received #{cmd}"
5  end
6
7end

1class CatPlugin < Plugin
2
3  def handle_command(cmd)
4    p "Command received #{cmd}"
5  end
6
7end

Now combine this all together in one main entry point and we have a simple plugin system that lets us send messages to each plugin through a set method ( handle_command ).

1require './plugin'
2Dir["./plugins/*.rb"].each { |f| require f }
3Plugin.register_plugins
4
5# Test that we can send a message to each plugin
6Plugin.plugins.each do |plugin|
7  plugin.handle_command('test')
8end

This is a very simple but useful way to make a plugin system to componentize projects like a chat bot for IRC.

Why setuid is Bad

setuid allows a binary to be run as a different user then the one invoking it. For example, ping needs to use low level system interfaces (socket, PF_INET, SOCK_RAW, etc) in order to function properly. We can watch this in action by starting ping in another terminal window ( ping google.com ) and then using strace to see the syscall’s being made:

sudo strace -p PID and we get the following:

1munmap(0x7f329e7ea000, 4096)            = 0stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=185, ...}) = 0
2socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
3connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0

We can find all setuid programs installed by issuing the command:

1sudo find / -xdev \( -perm -4000 \) -type f -print0 -exec ls -l {} \;

This will find all commands that have the root setuid bit set in their permission bit.

setuid list for a few popular operating systems:

Of particular interest in OpenBSD, where a lot of work was done to remove and switch programs from needing to use setuid/gid permissions. OpenIndiana is the worst offender and has the widest vector for attack.

• Ubuntu (22 binaries)
• CentOS (21 binaries)
• OpenBSD (3 binaries)
• OpenIndiana (53 binaries)

setuid escalation is a common attack vector and can allow unprivileged code to be executed by a regular user, and then escalate itself to root and drop you in on the root shell.

Here are a few examples:

CVE-2012-0056: Exploiting /proc/pid/mem

http://blog.zx2c4.com/749 - C code that uses a bug in the way the Linux kernel checked permissions on /proc/pid/mem and then uses that to exploit the su binary to give a root shell.

CVE-2010-3847: Exploiting via $ORIGIN and file descriptors

http://www.exploit-db.com/exploits/15274/ - By exploiting a hole in the way the $ORIGIN is checked, a symlink can be made to a program that uses setuid and exec’d ‘to obtain the file descriptors which then lets arbitrary code injection (in this case a call to system("/bin/bash")).

More of these can be found at http://www.exploit-db.com/shellcode/ and just searching google for setuid exploits.

So you may not want to completely disable the setuid flag on all the binaries for your distribution, but we can turn on some logging to watch when they’re getting called and install a kernel patch that will secure the OS and help prevent 0-days that may prey on setuid vulnerabilities.

How to log setuid calls

I will detail the steps to do this on Ubuntu, but they should apply to the other audit daemons on CentOS.

Let’s first install auditd: sudo apt-get install auditd

Let’s open up /etc/audit/audit.rules, and with a few tweaks with vim, we can insert the list we generated with find into the audit rule set (explanation of each flag after the jump):

 1# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.
 2# The rules are simply the parameters that would be passed
 3# to auditctl.
 4
 5# First rule - delete all
 6-D
 7
 8# Increase the buffers to survive stress events.
 9# Make this bigger for busy systems
10-b 320
11
12# Feel free to add below this line. See auditctl man page
13
14-a always,exit -F path=/usr/lib/pt_chown -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
15-a always,exit -F path=/usr/lib/eject/dmcrypt-get-device -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
16-a always,exit -F path=/usr/lib/dbus-1.0/dbus-daemon-launch-helper -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
17-a always,exit -F path=/usr/lib/openssh/ssh-keysign -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
18-a always,exit -F path=/usr/sbin/uuidd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
19-a always,exit -F path=/usr/sbin/pppd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
20-a always,exit -F path=/usr/bin/at -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
21-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
22-a always,exit -F path=/usr/bin/mtr -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
23-a always,exit -F path=/usr/bin/sudoedit -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
24-a always,exit -F path=/usr/bin/traceroute6.iputils -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
25-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
26-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
27-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
28-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
29-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
30-a always,exit -F path=/bin/fusermount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
31-a always,exit -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
32-a always,exit -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
33-a always,exit -F path=/bin/ping6 -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
34-a always,exit -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
35-a always,exit -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged

1-a: appends the always, and exit rules. This says to always make a log at syscall entry and syscall exit.
2-F
3     path= says filter to the executable being called
4     perm=x says filter on the program being executable
5     auid>= says log all calls for users who have a UID above 500 (regular user accounts start at 1000 generally)
6     auid!=4294967295 sometimes a process may start before the auditd, in which case it will get a auid of 4294967295
7-k passes a filter key that will be put into the record log, in this case its "privileged"

So now when we run ping google.com we can see a full audit trail in /var/log/audit/audit.log:

1type=SYSCALL msg=audit(1361852594.621:48): arch=c000003e syscall=59 success=yes exit=0 a0=f43de8 a1=d40488 a2=ed8008 a3=7fffc9c9a150 items=2 ppid=1464 pid=1631 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts1 ses=6 comm="ping" exe="/bin/ping" key="privileged"type=EXECVE msg=audit(1361852594.621:48): argc=2 a0="ping" a1="google.com"
2type=BPRM_FCAPS msg=audit(1361852594.621:48): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
3type=CWD msg=audit(1361852594.621:48):  cwd="/home/ubuntu"
4type=PATH msg=audit(1361852594.621:48): item=0 name="/bin/ping" inode=131711 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
5type=PATH msg=audit(1361852594.621:48): item=1 name=(null) inode=934 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00

Next steps: Patching and upgrading the kernel with GRSecurity

GRSecurity is an awesome tool in the security-minded system administrators toolbag. It will prevent zero days (like the proc mem exploit explained above ¹ ) by securing which areas a user can access. A full list can be seen at http://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options and http://en.wikipedia.org/wiki/Grsecurity#Miscellaneous_features, I suggest going through these and seeing if you want to continue with this.

The following below is for advanced users. Not responsible for any issues you may run into, please make sure to test this in a staging/test environment.

Here are the steps I followed to install the patch:

 1# Start by downloading the latest kernel
 2wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.39.tar.bz2
 3
 4# Next extract it
 5tar xjvf linux-3.2.39.tar.bz2
 6cd linux-3.2.39
 7
 8# Copy over your current kernel configuration:
 9cp -vi /boot/config-`uname -r` .config
10
11# Updates the config file to match old config and prompts for any new kernel options.
12make oldconfig
13
14# This will make sure only modules get compiled only if they are in your kernel.
15make localmodconfig
16
17# Bring up the configuration menu
18make menuconfig

Once your in the menu config you can browse to the Security section and go to Grsecurity and enable it. I set the configuration method to automatic and then went to Customize. For example, you can now go to Kernel Auditing -> Exec logging to turn on some additional logging to shell activities (WARNING: this will generate a lot of log activity, decide if you want to use this or not). I suggest going through all of these and reading through their menu help descriptions (when selecting one, press the ? key to bring up the help).

Now we’ll finish making the kernel and compiling it:

1# Now we can compile the kernel
2make -j2 # where 2 is the # of CPU's + 1
3
4# Install and load the dynamic kernel modules
5sudo make modules_install
6
7# Finally install kernel
8sudo make install

We can now reboot and boot into our GRsecurity patched kernel!

Hopefully this article has provided some insight into what the setuid flag does, how it has and can be exploited, and what we can do to prevent this in the future.

Here are a few links to useful books on the subject of shellcode and exploits that I reccomend:

Below is the list of setuid binaries on each OS

Ubuntu 12.04 LTS (22)

back to top

 1-rwsr-xr-x 1 root    root        31304 Mar  2  2012 /bin/fusermount-rwsr-xr-x 1 root    root        94792 Mar 30  2012 /bin/mount
 2-rwsr-xr-x 1 root    root        35712 Nov  8  2011 /bin/ping
 3-rwsr-xr-x 1 root    root        40256 Nov  8  2011 /bin/ping6
 4-rwsr-xr-x 1 root    root        36832 Sep 12 18:29 /bin/su
 5-rwsr-xr-x 1 root    root        69096 Mar 30  2012 /bin/umount
 6-rwsr-sr-x 1 daemon  daemon      47928 Oct 25  2011 /usr/bin/at
 7-rwsr-xr-x 1 root    root        41832 Sep 12 18:29 /usr/bin/chfn
 8-rwsr-xr-x 1 root    root        37096 Sep 12 18:29 /usr/bin/chsh
 9-rwsr-xr-x 1 root    root        63848 Sep 12 18:29 /usr/bin/gpasswd
10-rwsr-xr-x 1 root    root        62400 Jul 28  2011 /usr/bin/mtr
11-rwsr-xr-x 1 root    root        32352 Sep 12 18:29 /usr/bin/newgrp
12-rwsr-xr-x 1 root    root        42824 Sep 12 18:29 /usr/bin/passwd
13-rwsr-xr-x 2 root    root        71288 May 31  2012 /usr/bin/sudo
14-rwsr-xr-x 2 root    root        71288 May 31  2012 /usr/bin/sudoedit
15-rwsr-xr-x 1 root    root        18912 Nov  8  2011 /usr/bin/traceroute6.iputils
16-rwsr-xr-- 1 root    messagebus 292944 Oct  3 13:03 /usr/lib/dbus-1.0/dbus-daemon-launch-helper
17-rwsr-xr-x 1 root    root        10408 Dec 13  2011 /usr/lib/eject/dmcrypt-get-device
18-rwsr-xr-x 1 root    root       240984 Apr  2  2012 /usr/lib/openssh/ssh-keysign
19-rwsr-xr-x 1 root    root        10592 Oct  5 16:08 /usr/lib/pt_chown
20-rwsr-xr-- 1 root    dip        325744 Feb  4  2011 /usr/sbin/pppd
21-rwsr-sr-x 1 libuuid libuuid     18856 Mar 30  2012 /usr/sbin/uuidd

CentOS 6.3 (21)

back to top

 1-rwsr-xr-x. 1 root root  76056 Nov  5 05:21 /bin/mount-rwsr-xr-x. 1 root root  40760 Jul 19  2011 /bin/ping
 2-rwsr-xr-x. 1 root root  36488 Jul 19  2011 /bin/ping6
 3-rwsr-xr-x. 1 root root  34904 Jun 22  2012 /bin/su
 4-rwsr-xr-x. 1 root root  50496 Nov  5 05:21 /bin/umount
 5-rwsr-x---. 1 root dbus  46232 Sep 13 13:04 /lib64/dbus-1/dbus-daemon-launch-helper
 6-rwsr-xr-x. 1 root root  10272 Apr 16  2012 /sbin/pam_timestamp_check
 7-rwsr-xr-x. 1 root root  34840 Apr 16  2012 /sbin/unix_chkpwd
 8-rwsr-xr-x. 1 root root  54240 Jan 30  2012 /usr/bin/at
 9-rwsr-xr-x. 1 root root  66352 Dec  7  2011 /usr/bin/chage
10-rws--x--x. 1 root root  20184 Nov  5 05:21 /usr/bin/chfn
11-rws--x--x. 1 root root  20056 Nov  5 05:21 /usr/bin/chsh
12-rwsr-xr-x. 1 root root  47520 Jul 19  2011 /usr/bin/crontab
13-rwsr-xr-x. 1 root root  71480 Dec  7  2011 /usr/bin/gpasswd
14-rwsr-xr-x. 1 root root  36144 Dec  7  2011 /usr/bin/newgrp
15-rwsr-xr-x. 1 root root  30768 Feb 22  2012 /usr/bin/passwd
16---s--x--x. 2 root root 219272 Aug  6  2012 /usr/bin/sudo
17---s--x--x. 2 root root 219272 Aug  6  2012 /usr/bin/sudoedit
18-rwsr-xr-x. 1 root root 224912 Nov  9 07:49 /usr/libexec/openssh/ssh-keysign
19-rws--x--x. 1 root root  14280 Jan 31 06:30 /usr/libexec/pt_chown
20-rwsr-xr-x. 1 root root   9000 Sep 17 05:55 /usr/sbin/usernetctl

OpenBSD 5.2 (3)

back to top

1-r-sr-xr-x  1 root  bin       242808 Aug  1  2012 /sbin/ping-r-sr-xr-x  1 root  bin       263288 Aug  1  2012 /sbin/ping6
2-r-sr-x---  1 root  operator  222328 Aug  1  2012 /sbin/shutdown

OpenIndiana 11 (53)

back to top

 1-rwsr-xr-x   1 root     bin        64232 Jun 30  2012 /sbin/wificonfig--wS--lr-x   1 root     root           0 Dec 11 15:20 /media/.hal-mtab-lock
 2-r-sr-xr-x   1 root     bin       206316 Dec 11 21:00 /usr/lib/ssh/ssh-keysign
 3-rwsr-xr-x   1 root     adm        12140 Jun 30  2012 /usr/lib/acct/accton
 4-r-sr-xr-x   1 root     bin        23200 Jun 30  2012 /usr/lib/fs/ufs/quota
 5-r-sr-xr-x   1 root     bin       111468 Jun 30  2012 /usr/lib/fs/ufs/ufsrestore
 6-r-sr-xr-x   1 root     bin       106964 Jun 30  2012 /usr/lib/fs/ufs/ufsdump
 7-r-sr-xr-x   1 root     bin        18032 Jun 30  2012 /usr/lib/fs/smbfs/umount
 8-r-sr-xr-x   1 root     bin        18956 Jun 30  2012 /usr/lib/fs/smbfs/mount
 9-r-sr-xr-x   1 root     bin        12896 Jun 30  2012 /usr/lib/utmp_update
10-r-sr-xr-x   1 root     bin        35212 Jun 30  2012 /usr/bin/fdformat
11-r-s--x--x   2 root     bin       188080 Jun 30  2012 /usr/bin/sudoedit
12-r-sr-xr-x   1 root     sys        34876 Jun 30  2012 /usr/bin/su
13-r-sr-xr-x   1 root     bin        42504 Jun 30  2012 /usr/bin/login
14-r-sr-xr-x   1 root     bin       257288 Jun 30  2012 /usr/bin/pppd
15-r-sr-xr-x   1 root     sys        46208 Jun 30  2012 /usr/bin/chkey
16-r-sr-xr-x   1 root     sys        29528 Jun 30  2012 /usr/bin/amd64/newtask
17-r-sr-xr-x   2 root     bin        24432 Jun 30  2012 /usr/bin/amd64/w
18-r-sr-xr-x   1 root     bin      3224200 Jun 30  2012 /usr/bin/amd64/Xorg
19-r-sr-xr-x   2 root     bin        24432 Jun 30  2012 /usr/bin/amd64/uptime
20-rwsr-xr-x   1 root     sys        47804 Jun 30  2012 /usr/bin/at
21-r-sr-xr-x   1 root     bin         8028 Jun 30  2012 /usr/bin/mailq
22-r-sr-xr-x   1 root     bin        33496 Jun 30  2012 /usr/bin/rsh
23-r-sr-xr-x   1 root     bin        68704 Jun 30  2012 /usr/bin/rmformat
24-r-sr-sr-x   1 root     sys        31292 Jun 30  2012 /usr/bin/passwd
25-rwsr-xr-x   1 root     sys        23328 Jun 30  2012 /usr/bin/atrm
26-r-sr-xr-x   1 root     bin        97072 Jun 30  2012 /usr/bin/xlock
27-r-sr-xr-x   1 root     bin        78672 Jun 30  2012 /usr/bin/rdist
28-r-sr-xr-x   1 root     bin        27072 Jun 30  2012 /usr/bin/sys-suspend
29-r-sr-xr-x   1 root     bin        29304 Jun 30  2012 /usr/bin/crontab
30-r-sr-xr-x   1 root     bin        53080 Jun 30  2012 /usr/bin/rcp
31-r-s--x--x   2 root     bin       188080 Jun 30  2012 /usr/bin/sudo
32-r-s--x--x   1 uucp     bin        70624 Jun 30  2012 /usr/bin/tip
33-rwsr-xr-x   1 root     sys        18824 Jun 30  2012 /usr/bin/atq
34-r-sr-xr-x   1 root     bin       281732 Jun 30  2012 /usr/bin/xscreensaver
35-r-sr-xr-x   1 root     bin      2767780 Jun 30  2012 /usr/bin/i86/Xorg
36-r-sr-xr-x   1 root     sys        22716 Jun 30  2012 /usr/bin/i86/newtask
37-r-sr-xr-x   2 root     bin        22020 Jun 30  2012 /usr/bin/i86/w
38-r-sr-xr-x   2 root     bin        22020 Jun 30  2012 /usr/bin/i86/uptime
39-rwsr-xr-x   1 root     sys        13636 Jun 30  2012 /usr/bin/newgrp
40-r-sr-xr-x   1 root     bin        39224 Jun 30  2012 /usr/bin/rlogin
41-rwsr-xr-x   1 svctag   daemon    108964 Jun 30  2012 /usr/bin/stclient
42-r-sr-xr-x   1 root     bin        29324 Jun 30  2012 /usr/xpg4/bin/crontab
43-rwsr-xr-x   1 root     sys        47912 Jun 30  2012 /usr/xpg4/bin/at
44-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/deallocate
45-rwsr-xr-x   1 root     sys        32828 Jun 30  2012 /usr/sbin/sacadm
46-r-sr-xr-x   1 root     bin        46512 Jun 30  2012 /usr/sbin/traceroute
47-r-sr-xr-x   1 root     bin        18016 Jun 30  2012 /usr/sbin/i86/whodo
48-r-sr-xr-x   1 root     bin        55584 Jun 30  2012 /usr/sbin/ping
49-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/allocate
50-r-sr-xr-x   1 root     bin        37320 Jun 30  2012 /usr/sbin/pmconfig
51-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/list_devices
52-r-sr-xr-x   1 root     bin        24520 Jun 30  2012 /usr/sbin/amd64/whodo

Table of Contents

Initial Setup

Setting up iptables and Fail2Ban

Fail2Ban

iptables rules

Make shared memory read-only

Setting up Bastille Linux

Configuring Bastille

sysctl hardening

Setting up a chroot environment

Securing nginx inside the chroot

Extras

Initial Setup

Let’s login to our new machine and take some initial steps to secure our system. For this article I’m going to assume your username is ubuntu.

If you need to, setup your sudoers file by adding the following lines to /etc/sudoers:

1ubuntu ALL=(ALL:ALL) ALL # put this in the "User privilege specification" section

Edit your ~/.ssh/authorized_keys and put your public key inside it. Make sure you can login without a password now once your key is in place.

Open up /etc/ssh/sshd_config and make sure these lines exist to secure SSH:

 1# Only allow version 2 communications, version 1 has known vulnerabilities
 2Protocol 2
 3# Disable root login over ssh
 4PermitRootLogin no
 5# Load authorized keys files from a users home directory
 6AuthorizedKeysFile  %h/.ssh/authorized_keys
 7# Don't allow empty passwords to be used to authenticate
 8PermitEmptyPasswords no
 9# Disable password auth, you must use ssh keys
10PasswordAuthentication no

Keep your current session open and restart sshd:

1sudo service ssh restart

Make sure you can login from another terminal. If you can, move on.

Now we need to update and upgrade to make sure all of our packages are up to date and install two pre-requisites for later in the article: build-essential and ntp.

1sudo apt-get update
2sudo apt-get install build-essential ntp
3sudo apt-get upgrade
4sudo reboot

Setting up iptables and Fail2Ban

Fail2Ban

1sudo apt-get install fail2ban

Open up the fail2ban config and change the ban time, destemail, and maxretry /etc/fail2ban/jail.conf:

 1[DEFAULT]
 2ignoreip = 127.0.0.1/8
 3bantime  = 3600
 4maxretry = 2
 5destemail = [email protected]
 6action = %(action_mw)s
 7
 8[ssh]
 9
10enabled  = true
11port     = ssh
12filter   = sshd
13logpath  = /var/log/auth.log
14maxretry = 2

Now restart fail2ban.

1sudo service fail2ban restart

If you try and login from another machine and fail, you should see the ip in iptables.

1# sudo iptables -L
2Chain fail2ban-ssh (1 references)
3target     prot opt source               destination
4DROP       all  --  li203-XX.members.linode.com  anywhere
5RETURN     all  --  anywhere             anywhere

iptables Rules

Here are my default iptables rules, it opens up port 80 and 443 for HTTP/HTTPS communication, and allows port 22. We also allow ping and then log all denied calls and then reject everything else. If you have other services you need to run, such as a game server or something else, you’ll have to add the rules to open up the ports in the iptables config.

/etc/iptables.up.rules

 1*filter
 2
 3# Accepts all established inbound connections
 4 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 5
 6# Allows all outbound traffic
 7# You could modify this to only allow certain traffic
 8 -A OUTPUT -j ACCEPT
 9
10# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
11 -A INPUT -p tcp --dport 443 -j ACCEPT
12 -A INPUT -p tcp --dport 80 -j ACCEPT
13# Allows SSH connections for script kiddies
14# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
15 -A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
16
17# Now you should read up on iptables rules and consider whether ssh access
18# for everyone is really desired. Most likely you will only allow access from certain IPs.
19
20# Allow ping
21 -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
22
23# log iptables denied calls (access via 'dmesg' command)
24 -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
25
26# Reject all other inbound - default deny unless explicitly allowed policy:
27 -A INPUT -j REJECT
28 -A FORWARD -j REJECT
29
30COMMIT

We can load that up into iptables:

1sudo iptables-restore < /etc/iptables.up.rules

Make sure it loads on boot by putting it into the if-up scripts: /etc/network/if-up.d/iptables

1#!/bin/sh
2iptables-restore /etc/iptables.up.rules

Now make it executable:

1chmod +x /etc/network/if-up.d/iptables

Rebooting here is optional, I usually reboot after major changes to make sure everything boots up properly.

If you’re getting hit by scanners or brute-force attacks, you’ll see a line similar to this in your /var/log/syslog:

1Jan 18 03:30:37 localhost kernel: [   79.631680] iptables denied: IN=eth0 OUT= MAC=04:01:01:40:70:01:00:12:f2:c6:e8:00:08:00 SRC=87.13.110.30 DST=192.34.XX.XX LEN=64 TOS=0x00 PREC=0x00 TTL=34 ID=57021 DF PROTO=TCP SPT=1253 DPT=135 WINDOW=53760 RES=0x00 SYN URGP=0

Read only shared memory

A common exploit vector is going through shared memory (which can let you change the UID of running programs and other malicious actions). It can also be used as a place to drop files once an initial breakin has been made. An example of one such exploit is available here.

Open /etc/fstab/:

1tmpfs     /dev/shm     tmpfs     defaults,ro     0     0

Once you do this you need to reboot.

Setting up Bastille Linux

The Bastille Hardening program “locks down” an operating system, proactively configuring the system for increased security and decreasing its susceptibility to compromise. Bastille can also assess a system’s current state of hardening, granularly reporting on each of the security settings with which it works.

Bastille: Installation and Setup

1sudo apt-get install bastille # choose Internet site for postfix
2# configure bastille
3sudo bastille

After you run that command you’ll be prompted to configure your system, here are the options I chose:

Configuring Bastille

• File permissions module: Yes (suid)
• Disable SUID for mount/umount: Yes
• Disable SUID on ping: Yes
• Disable clear-text r-protocols that use IP-based authentication? Yes
• Enforce password aging? No (situation dependent, I have no users accessing my machines except me, and I only allow ssh keys)
• Default umask: Yes
• Umask: 077
• Disable root login on tty’s 1-6: No
• Password protect GRUB prompt: No (situation dependent, I’m on a VPS and would like to get support in case I need it)
• Password protect su mode: Yes
• default-deny on tcp-wrappers and xinetd? No
• Ensure telnet doesn’t run? Yes
• Ensure FTP does not run? Yes
• display authorized use message? No (situation dependent, if you had other users, Yes)
• Put limits on system resource usage? Yes
• Restrict console access to group of users? Yes (then choose root)
• Add additional logging? Yes
• Setup remote logging if you have a remote log host, I don’t so I answered No
• Setup process accounting? Yes
• Disable acpid? Yes
• Deactivate nfs + samba? Yes (situation dependent)
• Stop sendmail from running in daemon mode? No (I have this firewalled off, so I’m not concerned)
• Deactivate apache? Yes
• Disable printing? Yes
• TMPDIR/TMP scripts? No (if a multi-user system, yes)
• Packet filtering script? No (we configured the firewall previously)
• Finished? YES! & reboot

You can verify some of these changes by testing them out, for instance, the SUID change on ping:

Bastille: Verifying changes

1ubuntu@app1:~$ ping google.com
2ping: icmp open socket: Operation not permitted
3ubuntu@app1:~$ sudo ping google.com
4PING google.com (74.125.228.72) 56(84) bytes of data.
564 bytes from iad23s07-in-f8.1e100.net (74.125.228.72): icmp_req=1 ttl=55 time=9.06 ms
6^C
7--- google.com ping statistics ---
81 packets transmitted, 1 received, 0% packet loss, time 0ms
9rtt min/avg/max/mdev = 9.067/9.067/9.067/0.000 ms

Sysctl hardening

Since our machine isn’t running as a router and is going to be running as an application/web server, there are additional steps we can take to secure the machine. Many of these are from the NSA’s security guide, which you can read in its entirety here.

/etc/sysctl.conf http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf Source

 1# Protect ICMP attacks
 2net.ipv4.icmp_echo_ignore_broadcasts = 1
 3
 4# Turn on protection for bad icmp error messages
 5net.ipv4.icmp_ignore_bogus_error_responses = 1
 6
 7# Turn on syncookies for SYN flood attack protection
 8net.ipv4.tcp_syncookies = 1
 9
10# Log suspcicious packets, such as spoofed, source-routed, and redirect
11net.ipv4.conf.all.log_martians = 1
12net.ipv4.conf.default.log_martians = 1
13
14# Disables these ipv4 features, not very legitimate uses
15net.ipv4.conf.all.accept_source_route = 0
16net.ipv4.conf.default.accept_source_route = 0
17
18# Enables RFC-reccomended source validation (dont use on a router)
19net.ipv4.conf.all.rp_filter = 1
20net.ipv4.conf.default.rp_filter = 1
21
22# Make sure no one can alter the routing tables
23net.ipv4.conf.all.accept_redirects = 0
24net.ipv4.conf.default.accept_redirects = 0
25net.ipv4.conf.all.secure_redirects = 0
26net.ipv4.conf.default.secure_redirects = 0
27
28# Host only (we're not a router)
29net.ipv4.ip_forward = 0
30net.ipv4.conf.all.send_redirects = 0
31net.ipv4.conf.default.send_redirects = 0
32
33
34# Turn on execshild
35kernel.exec-shield = 1
36kernel.randomize_va_space = 1
37
38# Tune IPv6
39net.ipv6.conf.default.router_solicitations = 0
40net.ipv6.conf.default.accept_ra_rtr_pref = 0
41net.ipv6.conf.default.accept_ra_pinfo = 0
42net.ipv6.conf.default.accept_ra_defrtr = 0
43net.ipv6.conf.default.autoconf = 0
44net.ipv6.conf.default.dad_transmits = 0
45net.ipv6.conf.default.max_addresses = 1
46
47# Optimization for port usefor LBs
48# Increase system file descriptor limit
49fs.file-max = 65535
50
51# Allow for more PIDs (to reduce rollover problems); may break some programs 32768
52kernel.pid_max = 65536
53
54# Increase system IP port limits
55net.ipv4.ip_local_port_range = 2000 65000
56
57# Increase TCP max buffer size setable using setsockopt()
58net.ipv4.tcp_rmem = 4096 87380 8388608
59net.ipv4.tcp_wmem = 4096 87380 8388608
60
61# Increase Linux auto tuning TCP buffer limits
62# min, default, and max number of bytes to use
63# set max to at least 4MB, or higher if you use very high BDP paths
64net.core.rmem_max = 8388608
65net.core.wmem_max = 8388608
66net.core.netdev_max_backlog = 5000
67net.ipv4.tcp_window_scaling = 1

After making these changes you should reboot.

Setting up a chroot environment

We’ll be setting up a chroot environment to run our web server and applications in. Chroot’s provide isolation from the rest of the operating system, so even in the event of a application compromise, damage can be mitigated.

chroot: Installation and Setup

1sudo apt-get install debootstrap dchroot

Now add this to your /etc/schroot/schroot.conf file, precise is the release of Ubuntu I’m using, so change it if you need to:

/etc/schroot/schroot.conf

1[precise]
2description=Ubuntu Precise LTS
3location=/var/chroot
4priority=3
5users=ubuntu
6groups=sbuild
7root-groups=root

Now bootstrap the chroot with a minimal Ubuntu installation:

1sudo debootstrap --variant=buildd --arch amd64 precise /var/chroot/ http://mirror.anl.gov/pub/ubuntu/
2sudo cp /etc/resolv.conf /var/chroot/etc/resolv.conf
3sudo mount -o bind /proc /var/chroot/proc
4sudo chroot /var/chroot/
5apt-get install ubuntu-minimal
6apt-get update

Add the following to /etc/apt/sources.list inside the chroot:

1deb http://archive.ubuntu.com/ubuntu precise main
2deb http://archive.ubuntu.com/ubuntu precise-updates main
3deb http://security.ubuntu.com/ubuntu precise-security main
4deb http://archive.ubuntu.com/ubuntu precise universe
5deb http://archive.ubuntu.com/ubuntu precise-updates universe

Let’s test out our chroot and install nginx inside of it:

1apt-get update
2apt-get install nginx

Securing nginx inside the chroot

First thing we will do is add a www user for nginx to run under: Adding a application user

1sudo chroot /var/chroot
2useradd www -d /home/www
3mkdir /home/www
4chown -R www.www /home/www

Open up /etc/nginx/nginx.conf and make sure you change user to www inside the chroot:

1user www;

We can now start nginx inside the chroot:

1sudo chroot /var/chroot
2service nginx start

Now if you go to http://your_vm_ip/ you should see “Welcome to nginx!” running inside your fancy new chroot.

We also need to setup ssh to run inside the chroot so we can deploy our applications more easily.

Chroot: sshd

1sudo chroot /var/chroot
2apt-get install openssh-server udev

Since we already have SSH for the main host running on 22, we’re going to run SSH for the chroot on port 2222. We’ll copy over our config from outside the chroot to the chroot.

sshd config

1sudo cp /etc/ssh/sshd_config /var/chroot/etc/ssh/sshd_config

Now open the config and change the bind port to 2222.

We also need to add the rules to our firewall script: /etc/iptables.up.rules

1# Chroot ssh
2 -A INPUT -p tcp -m state --state NEW --dport 2222 -j ACCEPT

Now make a startup script for chroot-precise in /etc/init.d/chroot-precise: /etc/init.d/chroot-precise`

1mount -o bind /proc /var/chroot/proc
2mount -o bind /dev /var/chroot/dev
3mount -o bind /sys /var/chroot/sys
4mount -o bind /dev/pts /var/chroot/dev/pts
5chroot /var/chroot service nginx start
6chroot /var/chroot service ssh start

Set it to executable and to start at boot:

1sudo chmod +x /etc/init.d/chroot-precise
2sudo update-rc.d chroot-precise defaults

Next is to put your public key inside the .ssh/authorized_keys file for the www user inside the chroot so you can ssh and deploy your applications.

If you want, you can test your server and reboot it now to ensure nginx and ssh boot up properly. If it’s not running right now, you start it: sudo /etc/init.d/chroot-precise.

You should now be able to ssh into your chroot and main server without a password.

Extras

I would like to also mention the GRSecurity kernel patch. I had tried several times to install this (two different versions were released while I was writing this) and both make the kernel unable to compile. Hopefully they’ll fix these bugs and I’ll be able to update this article with notes on setting GRSecurity up as well.

I hope this article proved useful to anyone trying to secure a Ubuntu system, and if you liked it please share it!

Rb RFO Status is a simple system to post status updates to your team or customers in a easy to understand format so there is no delay in reporting a reason for outage. It is modeled slightly after the Heroku Status Page.

Source: https://github.com/bluescripts/rb_rfo_status

Download: https://s3.amazonaws.com/josh-opensource/rb_rfo_status-0.1.war

It is licensed under the MIT License so do whatever you want with it!

I’ve already opened up a few issues on Github that are enhancements, but this serves as a super simple application to deploy to keep your customers and team informed of system states.

Installation

Download the .war file and deploy it in your favorite container (Tomcat, etc). Once the war file is extracted you can modify the config settings and start it.

To run migrations on an extracted WAR file:

1cd rb_rfo_status/WEB-INF
2sudo RAILS_ENV=production BUNDLE_WITHOUT=development:test BUNDLE_GEMFILE=Gemfile GEM_HOME=gems java -cp lib/jruby-core-1.7.1.jar:lib/jruby-stdlib-1.7.1.jar:lib/gems-gems-activerecord-jdbc-adapter-1.2.2.1-lib-arjdbc-jdbc-adapter_java.jar:lib/gems-gems-jdbc-mysql-5.1.13-lib-mysql-connector-java-5.1.13.jar org.jruby.Main -S rake db:migrate

Screenshots

Homepage

Creating an Incident

Updating an incident

A resolved incident

Chef is awesome. Being able to recreate your entire environment from a recipe is an inredibly powerful tool, and I had started using Chef a few months ago. When I had initially configured the Chef server I hadn’t paid much attention to the couchdb portion of it until I had a chef-server hiccup. Here are a few things to watch out for when running chef-server:

• Setup CouchDB compaction - Chef had a CouchDB size of 30+GB (after compaction it was only a few megabytes).
• When resizing instances, make sure you setup RabbitMQ to use a NODENAME. If you don’t you’ll run into an issue with RabbitMQ losing the database’s that were setup (by default, they’re based on hostname… so if you resize a EC2 instance the hostname may change, and you’ll either have to do some moving around or manually set the NODENAME to the previous hostname).
• Client’s may fail to validate after this - requiring a regeneration of the validation.pem, which is fine since this file is only used for the initial bootstrap of a server.
• Make sure you run your chef recipes you setup (for instance monitoring) on your chef-server.

I hope these tips will be helpful to other people when they run into a Chef/CouchDB/RabbitMQ issue after a server resize or hostname change. Another really helpful place is #chef on freenode’s IRC servers.

Before we dive into the benchmarks of Resque vs Sidekiq it will first help to have a better understanding of how forking and threading works in Ruby.

Threading vs Forking

Forking

When you fork a process you are creating an entire copy of that process: the address space and all open file descriptors. You get a separate copy of the address space of the parent process, isolating any work done to that fork. If the forked child process does a lot of work and uses a lot of memory, when that child exits the memory gets free’d back to the operating system. If your programming language (MRI Ruby) doesn’t support actual kernel level threading, then this is the only way to spread work out across multiple cores since each process will get scheduled to a different core. You also gain some stability since if a child crashes the parent can just respawn a new fork, however there is a caveat. If the parent dies while there are children that haven’t exited, then those children become zombies.

Forking and Ruby

One important note about forking with Ruby is that the maintainers have done a good job on keeping memory usage down when forking. Ruby implements a copy on write system for memory allocation with child forks.

 1require 'benchmark'
 2
 3fork_pids = []
 4
 5# Lets fill up some memory
 6
 7objs = {}
 8objs['test'] = []
 91_000_000.times do
10  objs['test'] << Object.new
11end
12
13
14
1550.times do
16    fork_pids << Process.fork do
17        sleep 0.1
18    end
19end
20fork_pids.map{|p| Process.waitpid(p) }
21}

We can see this in action here:

However when we start modifying memory inside the child forks, memory quickly grows.

150.times do
2    fork_pids << Process.fork do
3      1_000_000.times do
4        objs << Object.new
5      end
6    end
7end
8fork_pids.map{|p| Process.waitpid(p) }

We’re now creating a million new objects in each forked child:

Threading

Threads on the other hand have considerably less overhead since they share address space, memory, and allow easier communication (versus inter-process communication with forks). Context switching between threads inside the same process is also generally cheaper than scheduling switches between processes. Depending on the runtime being used, any issues that might occur using threads (for instance needing to use lots of memory for a task) can be handled by the garbage collector for the most part. One of the benefits of threading is that you do not have to worry about zombie processes since all threads die when the process dies, avoiding the issue of zombies.

Threading with Ruby

As of 1.9 the GIL (Global Interpreter Lock) is gone! But it’s only been renamed to the GVL (Global VM Lock). The GVL in MRI ruby uses a lock called rb_thread_lock_t which is a mutex around when ruby code can be run. When no ruby objects are being touched, you can actually run ruby threads in parallel before the GVL kicks in again (ie: system level blocking call, IO blocking outside of ruby). After these blocking calls each thread checks the interrupt RUBY_VM_CHECK_INTS.

With MRI ruby threads are pre-emptively scheduled using a function called rb_thread_schedule which schedules an “interrupt” that lets each thread get a fair amount of execution time (every 10 microseconds). [source: thread.c:1018]

We can see an example of the GIL/GVL in action here:

 1threads = []
 2
 3objs = []
 4objs['test'] = []
 51_000_000.times do
 6  objs << Object.new
 7end
 8
 950.times do |num|
10  threads << Thread.new do
11    1_000_000.times do
12      objs << Object.new
13    end
14  end
15end
16
17threads.map(&:join)

Normally this would be an unsafe operation, but since the GIL/GVL exists we don’t have to worry about two threads adding to the same ruby object at once since only one thread can run on the VM at once and it ends up being an atomic operation (although don’t rely on this quirk for thread safety, it definitely doesn’t apply to any other VMs).

Another important note is that the Ruby GC is doing a really horrible job during this benchmark.

The memory kept growing so I had to kill the process after a few seconds.

Threading with JRuby on the JVM

JRuby specifies the use of native threads based on the operating system support using the getNativeThread call [2]. JRuby’s implementation of threads using the JVM means there is no GIL/GVL. This allows CPU bound processes to utilize all cores of a machine without having to deal with forking (which, in the case of resque, can be very expensive).

When trying to execute the GIL safe code above JRuby spits out a concurrency error: ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users

We can either add a mutex around this code or modify it to not worry about concurrent access. I chose the latter:

 1threads = []
 2
 3objs = {}
 4objs['test'] = []
 51_000_000.times do
 6  objs['test'] << Object.new
 7end
 8
 950.times do |num|
10  threads << Thread.new do
11    1_000_000.times do
12      objs[num] = [] if objs[num].nil?
13      objs[num] << Object.new
14    end
15  end
16end
17
18threads.map(&:join)

Compared to the MRI version, ruby running on the JVM was able to make some optimizations and keep memory usage around 800MB for the duration of the test:

Now that we have a better understanding of the differences between forking and threading in Ruby, lets move on to Sidekiq and Resque.

Sidekiq and Resque

Resque’s view of the world

Resque assumes chaos in your environment. It follows the forking model with C and ruby and makes a complete copy of each resque parent when a new job needs to be run. This has its advantages in preventing memory leaks, long running workers, and locking. You run into an issue with forking though when you need to increase the amount of workers on a machine. You end up not having enough spare CPU cycles since the majority are being taken up handling all the forking.

Resque follows a simple fork and do work model, each worker will take a job off the queue and fork a new process to do the job.

Resque @ Github

Sidekiq’s view of the world

Unlike Resque, Sidekiq uses threads and is extremely easy to use as a drop in replacement to Resque since they both work on the same perform method. When you dig into the results below you can see that Sidekiq’s claim of being able to handle a larger number of workers and amount of work is true. Due to using threads and not having to allocate a new stack and address space for each fork, you get that overhead back and are able to do more work with a threaded model.

Sidekiq follows the actor pattern. So compared to Resque which has N workers that fork, Sidekiq has an Actor manager, with N threads and one Fetcher actor which will pop jobs off Redis and hand them to the Manager. Sidekiq handles the “chaos” portion of Resque by catching all exceptions and bubbling them up to an exception handler such as Airbrake or Errbit.

Now that we know how Sidekiq and Resque work we can get on to testing them and comparing the results.

Sidekiq @ Github

The Test Code

The idea behind the test was to pick a CPU bound processing task, in this case SHA256 and apply it across a set of 20 numbers, 150,000 times.

 1require 'sidekiq'
 2require 'resque'
 3require 'digest'
 4
 5
 6# Running:
 7# sidekiq -r ./por.rb -c 240
 8#
 9# require 'sidekiq'
10# require './por'
11# queueing: 150_000.times { Sidekiq::Client.enqueue(POR, [rand(123098)]*20) }
12# queueing: 150_000.times { Resque.enqueue(POR, [rand(123098)]*20) }
13
14class POR
15  include Sidekiq::Worker
16
17  @queue = :por
18
19  def perform(arr)
20    arr.each do |a|
21      Digest::SHA2.new << a.to_s
22    end
23  end
24
25  def self.perform(arr)
26    arr.each do |a|
27      Digest::SHA2.new << a.to_s
28    end
29  end
30
31end

Test Machine

 1      Model Name: Mac Pro
 2      Model Identifier: MacPro4,1
 3      Processor Name: Quad-Core Intel Xeon
 4      Processor Speed: 2.26 GHz
 5      Number of Processors: 2
 6      Total Number of Cores: 8
 7      L2 Cache (per Core): 256 KB
 8      L3 Cache (per Processor): 8 MB
 9      Memory: 12 GB
10      Processor Interconnect Speed: 5.86 GT/s

This gives us a total of 16 cores to use for our testing. I’m also using a Crucial M4 SSD

Results

Time to Process 150,000 sets of 20 numbers

All about the CPU

Resque: 50 workers

Here we can see that the forking is taking its toll on the available CPU we have for processing. Roughly 50% of the CPU is being wasted on forking and scheduling those new processes. Resque took 396 seconds to finish and process 150,000 jobs.

Sidekiq (MRI) 1 process, 50 threads

We’re not fully utilizing the CPU. When running this test it pegged one CPU at 100% usage and kept it there for the duration of the test. We have a slight overhead with system CPU usage. Sidekiq took 312 seconds with 50 threads using MRI Ruby. Lets now take a look at doing things a bit resque-ish, and use multiple sidekiq processes to get more threads scheduled across multiple CPUs.

Sidekiq (MRI) 3 processes, 50 threads

We’re doing better. We’ve cut our processing time roughly in third and we’re utilizing more of our resources (CPUs). 3 Sidekiq processes with 50 threads each (for a total of 150 threads) took 120 seconds to complete 150,000 jobs.

Sidekiq (MRI) 5 processes, 50 threads

As we keep adding more processes that get scheduled to different cores we’re seeing the CPU usage go up even further, however with more processes comes more overhead for process scheduling (versus thread scheduling). We’re still wasting CPU cycles, but we’re completing 150,000 jobs in 98 seconds.

Sidekiq (JRuby) 50 threads

We’re doing much better now with native threads. With 50 OS level threads, we’re completing our set of jobs in 91 seconds.

Sidekiq (JRuby) 150 threads & 240 Threads

We’re no longer seeing a increase in (much) CPU usage and only a slight decrease in processing time. As we keep adding more and more threads we end up running into some thread contention issues with accessing redis and how quickly we can pop things off the queue.

Overview

Even if we stick with the stock MRI ruby and go with Sidekiq, we’re going to see a huge decrease in CPU usage while also gaining a little bit of performance as well.

Sidekiq, overall, provides a cleaner, more object oriented interface (in my opinion) to inspecting jobs and what is going on in the processing queue.

In Resque you would do something like: Resque.size("queue_name"). However, in Sidekiq you would take your class, in this case, POR and call POR.jobs to get the list of jobs for that worker queue. (note: you need to require 'sidekiq/testing' to get access to the jobs method).

The only thing I find missing from Sidekiq that I enjoyed in Resque was the ability to inspect failed jobs in the web UI. However Sidekiq more than makes up for that with the ability to automatically retry failed jobs (although be careful you don’t introduce race conditions and accidentally DOS yourself).

And of course, JRuby comes out on top and gives us the best performance and bang for the buck (although your mileage may vary, depending on the task).

Further Reading

Deploying with JRuby: Deliver Scalable Web Apps using the JVM (Pragmatic Programmers)

JRuby Cookbook

Sidekiq & Resque

Sidekiq

Resque

I saw the question of “How can I prevent a class from being reopened again in Ruby?” pop up on the Ruby mailing list. While this is somewhat against the nature of ruby, it can be accomplished:

{% codeblock lang:ruby %} class Foo def Foo.method_added(name) raise “This class is closed for modification” end end

class Foo def testing p “test” end end {% endcodeblock %}

This will raise an exception anytime someone tries to reopen the class.

When you work on your code and are finished for the day, is what you have committed worry free? If another developer were to push your code in the middle of the night, would they be calling you at 3am?

Let’s see how we can improve our development cycle with testing so we can avoid those early morning calls. We’ll go over some of the basics with a simple project to start.

The most important part about TDD is getting quick feedback based on our desired design (the feedback loop).

Here is an example of how fast the tests run:

While this is a somewhat contrived example for the reddit cli we’re making, this can be applied equally as well when writing Rails applications. Only load the parts you need (ActionMailer, ActiveSupport, etc), usually you don’t need to load the entire rails stack. This can make your tests run in milliseconds instead of seconds. This lets you get feedback right away.

Before we go further into the testing discussion, lets setup a spec helper.

{% codeblock spec/spec_helper.rb lang:ruby %} require ‘rspec’ require ‘vcr’ require ‘pry’ VCR.configure do |c| c.cassette_library_dir = ‘fixtures/vcr_cassettes’ c.hook_into :fakeweb# or :fakeweb end {% endcodeblock %}

Now how do we start doing TDD? We first start with a failing test.

{% codeblock Reddit API Spec (Pass 1) - spec/lib/reddit_api_spec lang:ruby %} require ‘spec_helper’ require ‘./lib/reddit_api’

describe RedditApi do let(:reddit) { RedditApi.new(‘ProgrammerHumor’) } context “#initializing” do it “should form the correct endpoint” do reddit.url.should eq “http://reddit.com/r/ProgrammerHumor/.json?after=" end end end {% endcodeblock %}

When we create a new instance of the Reddit API we want to pass it a subreddit, and then we want to make sure it builds the URL properly.

{% codeblock Reddit API (Pass 1) - lib/reddit_api.rb lang:ruby %} require ‘json’ require ‘rest-client’

class RedditApi REDDIT_URL = “http://reddit.com/r/" attr_reader :url, :stories def initialize(subreddit) @subreddit = subreddit @after = “” @url = “#{REDDIT_URL}#{subreddit}/.json?after=#{@after}” end end {% endcodeblock %}

Next we want to make the actual HTTP request to the Reddit api and process it.

{% codeblock Reddit API Spec (Pass 2) - spec/lib/reddit_api_spec lang:ruby %} require ‘spec_helper’ require ‘./lib/reddit_api’

describe RedditApi do let(:reddit) { RedditApi.new(‘ProgrammerHumor’) } context “#initializing” do it “should form the correct endpoint” do VCR.use_cassette(‘reddit_programmer_humor’) do reddit.url.should eq “http://reddit.com/r/ProgrammerHumor/.json?after=" end end end

1context "#fetching" do
2    it "should fetch the first page of stories" do
3        VCR.use_cassette('reddit_programmer_humor') do
4            reddit.stories.count.should eq(25)
5        end
6    end
7end

end {% endcodeblock %}

We’ve now added a VCR wrapper and added an expectation that the reddit api will return a list of stories. We use VCR here to again ensure that our tests run fast. Once we make the first request, future runs will take milliseconds and will hit our VCR tape instead of the API.

Now we need to introduce three new areas: requesting, processing, and a Story object class.

{% codeblock Story - lib/story.rb lang:ruby %} Story = Struct.new(:title, :score, :comments, :url) {% endcodeblock %}

{% codeblock Reddit API (Pass 2) - lib/reddit_api.rb lang:ruby %} require ‘json’ require ‘rest-client’ require ‘./lib/story’

class RedditApi REDDIT_URL = “http://reddit.com/r/" attr_reader :url, :stories def initialize(subreddit) @subreddit = subreddit @after = “” @url = “#{REDDIT_URL}#{subreddit}/.json?after=#{@after}” request process_request end

 1def request
 2    @request_response = JSON.parse(RestClient.get(@url))
 3end
 4
 5def process_request
 6    @stories = []
 7    @request_response['data']['children'].each do |red|
 8        d = red['data']
 9        @stories << Story.new(d['title'], d['score'],
10                              d['num_comments'], d['url'])
11    end
12    @after = @request_response['data']['after']
13end

end {% endcodeblock %}

What can we do now? The API lets us make a full request and get a list of Story struct objects back. We’ll be using this array of structs later on to build the CLi.

The only thing left for this simple CLI a way to get to the next page. Let’s add our failing spec:

{% codeblock Reddit API Spec (Pass 3) - spec/lib/reddit_api_spec lang:ruby %} require ‘spec_helper’ require ‘./lib/reddit_api’

describe RedditApi do let(:reddit) { RedditApi.new(‘ProgrammerHumor’) } context “#initializing” do it “should form the correct endpoint” do VCR.use_cassette(‘reddit_programmer_humor’) do reddit.url.should eq “http://reddit.com/r/ProgrammerHumor/.json?after=" end end end

 1context "#fetching" do
 2    it "should fetch the first page of stories" do
 3        VCR.use_cassette('reddit_programmer_humor') do
 4            reddit.stories.count.should eq(25)
 5        end
 6    end
 7
 8    it "should fetch the second page of stories" do
 9        VCR.use_cassette('reddit_programmer_humor_p2') do
10            reddit.next.stories.count.should eq(25)
11        end
12    end
13end

end {% endcodeblock %}

And let’s make the test pass:

{% codeblock Reddit API (Pass 3) - lib/reddit_api.rb lang:ruby %} require ‘json’ require ‘rest-client’ require ‘./lib/story’

class RedditApi REDDIT_URL = “http://reddit.com/r/" attr_reader :url, :stories def initialize(subreddit) @subreddit = subreddit @after = “” @url = “#{REDDIT_URL}#{subreddit}/.json?after=#{@after}” request process_request end

 1def next
 2    @url = "#{REDDIT_URL}#{@subreddit}/.json?after=#{@after}"
 3    request
 4    process_request
 5    self
 6end
 7
 8def request
 9    @request_response = JSON.parse(RestClient.get(@url))
10end
11
12def process_request
13    @stories = []
14    @request_response['data']['children'].each do |red|
15        d = red['data']
16        @stories << Story.new(d['title'], d['score'],
17                              d['num_comments'], d['url'])
18    end
19    @after = @request_response['data']['after']
20end

end {% endcodeblock %}

We also allow method chaining since we return self after calling next (so you could chain next’s for instance).

Another important principal to keep in mind is the “Tell, Dont Ask” rule. Without tests, we might have gone this route:

{% codeblock bad_example.rb lang:ruby %} @reddit = Reddit.new(‘ProgrammerHumor’)

User presses next

@reddit.url = “http://reddit.com/r/ProgrammerHumor/.json?after=sometoken" {% endcodeblock %}

Not only would we not be telling the object what we want, we would be modifying the internal state of an object as well. By implementing a next method we abstract the idea of a URL and any tokens we may need to keep track of away from the consumer. Doing TDD adds a little extra step of “Thinking” more about what we want our interfaces to be. What’s easier? Calling next or modifying the internal state?

I’m kind of cheating a bit here. I found a nice “table” gem that outputs what you send in as a formatted table (think MySQL console output). Let’s just make sure everything is being sent around properly and STDOUT is printing the correct contents:

{% codeblock Reddit CLI Spec (Pass 1) - spec/lib/reddit-cli.rb lang:ruby %} require ‘spec_helper’ require ‘stringio’ require ‘./lib/reddit-cli’

describe RedditCli do let(:subreddit) { “ProgrammerHumor” } context “#initializing” do before(:all) do $stdout = @fakeout = StringIO.new end

 1    it "should print out a story" do
 2        api_response = double(RedditApi)
 3        api_response.stub!(:stories =>
 4                           [Story.new("StoryTitle", "Score",
 5                                      "Comments", "URL")])
 6        $stdin.should_receive(:gets).and_return("q")
 7        cli = RedditCli.new(api_response)
 8        $stdout = STDOUT
 9        @fakeout.string.include?('StoryTitle').should be_true
10    end
11end

end {% endcodeblock %}

We’re doing several things here. First we’re taking $stdout and putting it (temporarily) into a instance variable so we can see what gets outputted. Next we’re mocking out the RedditApi since we dont actually need to hit that class or the VCR tapes, we just need to stub out the expected results (stories) and pass the response object along to the CLI class. And finally once we’re finished we set $stdout back to the proper constant.

And the class for output:

{% codeblock Reddit CLI (Pass 1) - lib/reddit-cli.rb lang:ruby %} require ‘./lib/reddit_api’ require ‘terminal-table’ class RedditCli def initialize(api) @rows = [] @api = api @stories = api.stories print_stories print “\nType ? for help\n” prompt end

 1def print_stories
 2    @stories.each_with_index {|x, i| @rows << [i, x.score, x.comments, x.title[0..79] ] }
 3    puts Terminal::Table.new :headings=> ['#', 'Score', 'Comments', 'Title'], :rows => @rows
 4end
 5
 6def prompt
 7    print "\n?> "
 8    input = STDIN.gets.chomp
 9    case input
10    when "?"
11        p "Type the # of a story to open it in your browser"
12        p "Type n to go to the next page"
13        prompt
14    when "quit", "q"
15    when "n"
16        @rows = []
17        @stories = @api.next.stories
18        print_stories
19        prompt
20    else
21        print "#=> Oepning: #{@stories[input.to_i].url}"
22        `open #{@stories[input.to_i].url}`
23        prompt
24    end
25end

end {% endcodeblock %}

And finally, a little wrapper in the root directory:

{% codeblock Wrapper - reddit-cli.rb lang:ruby %} require ‘./lib/reddit_api’ require ‘./lib/reddit-cli’

subreddit = ARGV[0] RedditCli.new(RedditApi.new(subreddit)) {% endcodeblock %}

An Important Note

When working with external resources, whether it be a gem or a remote API, it’s important to wrap those endpoints in your own abstraction. For instance, with our Reddit CLI we could have avoided those first 2 classes entirely, written everything in the CLI display class, and worked with the raw JSON. But what happens when Reddit changes their API? If this CLI class was huge or incoporated many other components, this could be quite a big code change. Instead, what we wrote encapsulates the API inside a RedditApi class that returns a generic Story struct we can work with and pass around. We don’t care if the API changes in the CLI, or in any other code. If the API changes, we only have to update the one API class to mold the new API to the output we were already generating.

End Result & Source Code

I was working on my blog and moving some posts around when I kept getting a Psych::SyntaxError when generating it with Jekyll and ruby 1.9.x. Unfortunately the default stack trace doesn’t provide much information on what file was causing the issue, so a quick way to find out is opening up irb:

{% codeblock Example to run in irb - sample.rb lang:ruby %} require ‘yaml’ Dir.foreach(“source/_posts”).each {|f| YAML.load_file(“source/_posts/” + f) unless f == “.” || f == “..” } {% endcodeblock %}