setuid allows a binary to be run as a different user then the one invoking it. For example, ping needs to use low level system interfaces (socket, PF_INET, SOCK_RAW, etc) in order to function properly. We can watch this in action by starting ping in another terminal window ( ping google.com ) and then using strace to see the syscall’s being made:
We can find all setuid programs installed by issuing the command:
How to find all setuid programs
1
sudo find / -xdev \( -perm -4000 \) -type f -print0 -exec ls -l {}\;
This will find all commands that have the root setuid bit set in their permission bit.
setuid list for a few popular operating systems:
Of particular interest in OpenBSD, where a lot of work was done to remove and switch programs from needing to use setuid/gid permissions. OpenIndiana is the worst offender and has the widest vector for attack.
setuid escalation is a common attack vector and can allow unprivileged code to be executed by a regular user, and then escalate itself to root and drop you in on the root shell.
Here are a few examples:
CVE-2012-0056: Exploiting /proc/pid/mem
http://blog.zx2c4.com/749 - C code that uses a bug in the way the Linux kernel checked permissions on /proc/pid/mem and then uses that to exploit the su binary to give a root shell.
CVE-2010-3847: Exploiting via $ORIGIN and file descriptors
http://www.exploit-db.com/exploits/15274/ - By exploiting a hole in the way the $ORIGIN is checked, a symlink can be made to a program that uses setuid and exec‘d ‘to obtain the file descriptors which then lets arbitrary code injection (in this case a call to system("/bin/bash")).
So you may not want to completely disable the setuid flag on all the binaries for your distribution, but we can turn on some logging to watch when they’re getting called and install a kernel patch that will secure the OS and help prevent 0-days that may prey on setuid vulnerabilities.
How to log setuid calls
I will detail the steps to do this on Ubuntu, but they should apply to the other audit daemons on CentOS.
Let’s first install auditd: sudo apt-get install auditd
Let’s open up /etc/audit/audit.rules, and with a few tweaks with vim, we can insert the list we generated with find into the audit rule set (explanation of each flag after the jump):
# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.# The rules are simply the parameters that would be passed# to auditctl.# First rule - delete all-D
# Increase the buffers to survive stress events.# Make this bigger for busy systems-b 320
# Feel free to add below this line. See auditctl man page-a always,exit -F path=/usr/lib/pt_chown -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/eject/dmcrypt-get-device -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/dbus-1.0/dbus-daemon-launch-helper -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/openssh/ssh-keysign -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/uuidd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/pppd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/at -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/mtr -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudoedit -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/traceroute6.iputils -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/fusermount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/ping6 -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
audtid Options Explained
1234567
-a: appends the always, and exit rules. This says to always make a log at syscall entry and syscall exit.
-F
path= says filter to the executable being called
perm=x says filter on the program being executable
auid>= says log all calls for users who have a UID above 500 (regular user accounts start at 1000 generally)
auid!=4294967295 sometimes a process may start before the auditd, in which case it will get a auid of 4294967295
-k passes a filter key that will be put into the record log, in this case its "privileged"
So now when we run ping google.com we can see a full audit trail in /var/log/audit/audit.log:
The following below is for advanced users. Not responsible for any issues you may run into, please make sure to test this in a staging/test environment.
Here are the steps I followed to install the patch:
123456789101112131415161718
# Start by downloading the latest kernelwget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.39.tar.bz2
# Next extract ittar xjvf linux-3.2.39.tar.bz2
cd linux-3.2.39
# Copy over your current kernel configuration:cp -vi /boot/config-`uname -r` .config
# Updates the config file to match old config and prompts for any new kernel options.make oldconfig
# This will make sure only modules get compiled only if they are in your kernel. make localmodconfig
# Bring up the configuration menumake menuconfig
Once your in the menu config you can browse to the Security section and go to Grsecurity and enable it. I set the configuration method to automatic and then went to Customize. For example, you can now go to Kernel Auditing -> Exec logging to turn on some additional logging to shell activities (WARNING: this will generate a lot of log activity, decide if you want to use this or not). I suggest going through all of these and reading through their menu help descriptions (when selecting one, press the ? key to bring up the help).
Now we’ll finish making the kernel and compiling it:
12345678
# Now we can compile the kernelmake -j2 # where 2 is the # of CPU's + 1# Install and load the dynamic kernel modulessudo make modules_install
# Finally install kernelsudo make install
We can now reboot and boot into our GRsecurity patched kernel!
Hopefully this article has provided some insight into what the setuid flag does, how it has and can be exploited, and what we can do to prevent this in the future.
Here are a few links to useful books on the subject of shellcode and exploits that I reccomend:
Let’s login to our new machine and take some initial steps to secure our system. For this article I’m going to assume your username is ubuntu.
If you need to, setup your sudoers file by adding the following lines:
/etc/sudoers
1
ubuntu ALL=(ALL:ALL) ALL # put this in the "User privilege specification" section
Edit your ~/.ssh/authorized_keys and put your public key inside it. Make sure you can login without a password now once your key is in place.
Open up /etc/ssh/sshd_config and make sure these lines exist to secure SSH:
/etc/ssh/sshd_config
12345678910
# Only allow version 2 communications, version 1 has known vulnerabilitiesProtocol 2
# Disable root login over sshPermitRootLogin no
# Load authorized keys files from a users home directoryAuthorizedKeysFile %h/.ssh/authorized_keys
# Don't allow empty passwords to be used to authenticatePermitEmptyPasswords no
# Disable password auth, you must use ssh keysPasswordAuthentication no
Keep your current session open and restart sshd:
/etc/ssh/sshd_config
1
sudo service ssh restart
Make sure you can login from another terminal. If you can, move on.
Now we need to update and upgrade to make sure all of our packages are up to date and install two pre-requisites for later in the article: build-essential and ntp.
If you try and login from another machine and fail, you should see the ip in iptables.
/etc/fail2ban/jail.conf
12345
# sudo iptables -LChain fail2ban-ssh (1 references)target prot opt source destination
DROP all -- li203-XX.members.linode.com anywhere
RETURN all -- anywhere anywhere
iptables Rules
Here are my default iptables rules, it opens up port 80 and 443 for HTTP/HTTPS communication, and allows port 22.
We also allow ping and then log all denied calls and then reject everything else. If you have other services you need to run, such as a game server or something else, you’ll have to add the rules to open up the ports in the iptables config.
*filter
# Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allows all outbound traffic
# You could modify this to only allow certain traffic
-A OUTPUT -j ACCEPT
# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 443 -j ACCEPT
-A INPUT -p tcp --dport 80 -j ACCEPT
# Allows SSH connections for script kiddies
# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Now you should read up on iptables rules and consider whether ssh access
# for everyone is really desired. Most likely you will only allow access from certain IPs.
# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
# log iptables denied calls (access via 'dmesg' command)
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
# Reject all other inbound - default deny unless explicitly allowed policy:
-A INPUT -j REJECT
-A FORWARD -j REJECT
COMMIT
We can load that up into iptables:
1
sudo iptables-restore < /etc/iptables.up.rules
Make sure it loads on boot by putting it into the if-up scripts:
/etc/network/if-up.d/iptables
12
#!/bin/shiptables-restore /etc/iptables.up.rules
Now make it executable:
1
chmod +x /etc/network/if-up.d/iptables
Rebooting here is optional, I usually reboot after major changes to make sure everything boots up properly.
If you’re getting hit by scanners or brute-force attacks, you’ll see a line similar to this in your /var/log/syslog:
A common exploit vector is going through shared memory (which can let you change the UID of running programs and other malicious actions). It can also be used as a place to drop files once an initial breakin has been made. An example of one such exploit is available here.
Open /etc/fstab/:
/etc/fstab
1
tmpfs /dev/shm tmpfs defaults,ro 0 0
Once you do this you need to reboot.
Setting up Bastille Linux
The Bastille Hardening program “locks down” an operating system, proactively configuring the system for increased security and decreasing its susceptibility to compromise. Bastille can also assess a system’s current state of hardening, granularly reporting on each of the security settings with which it works.
Bastille: Installation and Setup
123
sudo apt-get install bastille # choose Internet site for postfix# configure bastillesudo bastille
After you run that command you’ll be prompted to configure your system, here are the options I chose:
Configuring Bastille
File permissions module: Yes (suid)
Disable SUID for mount/umount: Yes
Disable SUID on ping: Yes
Disable clear-text r-protocols that use IP-based authentication? Yes
Enforce password aging? No (situation dependent, I have no users accessing my machines except me, and I only allow ssh keys)
Default umask: Yes
Umask: 077
Disable root login on tty’s 1-6: No
Password protect GRUB prompt: No (situation dependent, I’m on a VPS and would like to get support in case I need it)
Password protect su mode: Yes
default-deny on tcp-wrappers and xinetd? No
Ensure telnet doesn’t run? Yes
Ensure FTP does not run? Yes
display authorized use message? No (situation dependent, if you had other users, Yes)
Put limits on system resource usage? Yes
Restrict console access to group of users? Yes (then choose root)
Add additional logging? Yes
Setup remote logging if you have a remote log host, I don’t so I answered No
Setup process accounting? Yes
Disable acpid? Yes
Deactivate nfs + samba? Yes (situation dependent)
Stop sendmail from running in daemon mode? No (I have this firewalled off, so I’m not concerned)
Deactivate apache? Yes
Disable printing? Yes
TMPDIR/TMP scripts? No (if a multi-user system, yes)
Packet filtering script? No (we configured the firewall previously)
Finished? YES! & reboot
You can verify some of these changes by testing them out, for instance, the SUID change on ping:
Bastille: Verifying changes
123456789
ubuntu@app1:~$ ping google.com
ping: icmp open socket: Operation not permitted
ubuntu@app1:~$ sudo ping google.com
PING google.com (74.125.228.72) 56(84) bytes of data.
64 bytes from iad23s07-in-f8.1e100.net (74.125.228.72): icmp_req=1 ttl=55 time=9.06 ms
^C
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 9.067/9.067/9.067/0.000 ms
Sysctl hardening
Since our machine isn’t running as a router and is going to be running as an application/web server, there are additional
steps we can take to secure the machine. Many of these are from the NSA’s security guide, which you can read in its entirety
here.
# Protect ICMP attacksnet.ipv4.icmp_echo_ignore_broadcasts = 1
# Turn on protection for bad icmp error messagesnet.ipv4.icmp_ignore_bogus_error_responses = 1
# Turn on syncookies for SYN flood attack protectionnet.ipv4.tcp_syncookies = 1
# Log suspcicious packets, such as spoofed, source-routed, and redirectnet.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
# Disables these ipv4 features, not very legitimate usesnet.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Enables RFC-reccomended source validation (dont use on a router)net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Make sure no one can alter the routing tablesnet.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
# Host only (we're not a router)net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Turn on execshildkernel.exec-shield = 1
kernel.randomize_va_space = 1
# Tune IPv6net.ipv6.conf.default.router_solicitations = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.default.dad_transmits = 0
net.ipv6.conf.default.max_addresses = 1
# Optimization for port usefor LBs# Increase system file descriptor limitfs.file-max = 65535
# Allow for more PIDs (to reduce rollover problems); may break some programs 32768kernel.pid_max = 65536
# Increase system IP port limitsnet.ipv4.ip_local_port_range = 2000 65000
# Increase TCP max buffer size setable using setsockopt()net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
# Increase Linux auto tuning TCP buffer limits# min, default, and max number of bytes to use# set max to at least 4MB, or higher if you use very high BDP pathsnet.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
After making these changes you should reboot.
Setting up a chroot environment
We’ll be setting up a chroot environment to run our web server and applications in. Chroot’s provide isolation from the rest of the operating system, so even in the event of a application compromise, damage can be mitigated.
chroot: Installation and Setup
1
sudo apt-get install debootstrap dchroot
Now add this to your /etc/schroot/schroot.conf file, precise is the release of Ubuntu I’m using, so change it if you need to:
Add the following to /etc/apt/sources.list inside the chroot:
12345
deb http://archive.ubuntu.com/ubuntu precise main
deb http://archive.ubuntu.com/ubuntu precise-updates main
deb http://security.ubuntu.com/ubuntu precise-security main
deb http://archive.ubuntu.com/ubuntu precise universe
deb http://archive.ubuntu.com/ubuntu precise-updates universe
Let’s test out our chroot and install nginx inside of it:
12
apt-get update
apt-get install nginx
Securing nginx inside the chroot
First thing we will do is add a www user for nginx to run under:
Since we already have SSH for the main host running on 22, we’re going to run SSH for the chroot on port 2222. We’ll copy over our config from outside the chroot to the chroot.
Now open the config and change the bind port to 2222.
We also need to add the rules to our firewall script:
/etc/iptables.up.rules
12
# Chroot ssh -A INPUT -p tcp -m state --state NEW --dport 2222 -j ACCEPT
Now make a startup script for chroot-precise in `/etc/init.d/chroot-precise:
/etc/init.d/chroot-precise
123456
mount -o bind /proc /var/chroot/proc
mount -o bind /dev /var/chroot/dev
mount -o bind /sys /var/chroot/sys
mount -o bind /dev/pts /var/chroot/dev/pts
chroot /var/chroot service nginx start
chroot /var/chroot service ssh start
Next is to put your public key inside the .ssh/authorized_keys file for the www user inside the chroot so you can ssh and deploy your applications.
If you want, you can test your server and reboot it now to ensure nginx and ssh boot up properly. If it’s not running right now, you start it: sudo /etc/init.d/chroot-precise.
You should now be able to ssh into your chroot and main server without a password.
Extras
I would like to also mention the GRSecurity kernel patch. I had tried several times to install this (two different versions were released while I was writing this) and both make the kernel unable to compile. Hopefully they’ll fix these bugs and I’ll be able to update this article with notes on setting GRSecurity up as well.
I hope this article proved useful to anyone trying to secure a Ubuntu system, and if you liked it please share it!
Rb RFO Status is a simple system to post status updates to your team or customers in a easy to understand format so there is no delay in reporting a reason for outage.
It is modeled slightly after the Heroku Status Page.
It is licensed under the MIT License so do whatever you want with it!
I’ve already opened up a few issues on Github that are enhancements, but this serves as a super simple application to deploy to keep your customers and team informed of system states.
Installation
Download the .war file and deploy it in your favorite container (Tomcat, etc). Once the war file is extracted you can modify the config settings and start it.
Chef is awesome. Being able to recreate your entire environment from a recipe is an inredibly powerful tool, and I had started using Chef a few months ago. When I had initially configured the Chef server I hadn’t paid much attention to the couchdb portion of it until I had a chef-server hiccup. Here are a few things to watch out for when running chef-server:
Setup CouchDB compaction - Chef had a CouchDB size of 30+GB (after compaction it was only a few megabytes).
When resizing instances, make sure you setup RabbitMQ to use a NODENAME. If you don’t you’ll run into an issue with RabbitMQ losing the database’s that were setup (by default, they’re based on hostname… so if you resize a EC2 instance the hostname may change, and you’ll either have to do some moving around or manually set the NODENAME to the previous hostname).
Client’s may fail to validate after this - requiring a regeneration of the validation.pem, which is fine since this file is only used for the initial bootstrap of a server.
Make sure you run your chef recipes you setup (for instance monitoring) on your chef-server.
I hope these tips will be helpful to other people when they run into a Chef/CouchDB/RabbitMQ issue after a server resize or hostname change. Another really helpful place is #chef on freenode’s IRC servers.
Before we dive into the benchmarks of Resque vs Sidekiq it will first help to have a better understanding of how forking and threading works in Ruby.
Threading vs Forking
Forking
When you fork a process you are creating an entire copy of that process: the address space and all open file descriptors. You get a separate copy of the address space of the parent process, isolating any work done to that fork. If the forked child process does a lot of work and uses a lot of memory, when that child exits the memory gets free’d back to the operating system. If your programming language (MRI Ruby) doesn’t support actual kernel level threading, then this is the only way to spread work out across multiple cores since each process will get scheduled to a different core. You also gain some stability since if a child crashes the parent can just respawn a new fork, however there is a caveat. If the parent dies while there are children that haven’t exited, then those children become zombies.
Forking and Ruby
One important note about forking with Ruby is that the maintainers have done a good job on keeping memory usage down when forking. Ruby implements a copy on write system for memory allocation with child forks.
123456789101112131415161718192021
require'benchmark'fork_pids=[]# Lets fill up some memoryobjs={}objs['test']=[]1_000_000.timesdoobjs['test']<<Object.newend50.timesdofork_pids<<Process.forkdosleep0.1endendfork_pids.map{|p|Process.waitpid(p)}}
We can see this in action here:
However when we start modifying memory inside the child forks, memory quickly grows.
We’re now creating a million new objects in each forked child:
Threading
Threads on the other hand have considerably less overhead since they share address space, memory, and allow easier communication (versus inter-process communication with forks). Context switching between threads inside the same process is also generally cheaper than scheduling switches between processes. Depending on the runtime being used, any issues that might occur using threads (for instance needing to use lots of memory for a task) can be handled by the garbage collector for the most part. One of the benefits of threading is that you do not have to worry about zombie processes since all threads die when the process dies, avoiding the issue of zombies.
Threading with Ruby
As of 1.9 the GIL (Global Interpreter Lock) is gone! But it’s only been renamed to the GVL (Global VM Lock). The GVL in MRI ruby uses a lock called rb_thread_lock_t which is a mutex around when ruby code can be run. When no ruby objects are being touched, you can actually run ruby threads in parallel before the GVL kicks in again (ie: system level blocking call, IO blocking outside of ruby). After these blocking calls each thread checks the interrupt RUBY_VM_CHECK_INTS.
With MRI ruby threads are pre-emptively scheduled using a function called rb_thread_schedule which schedules an “interrupt” that lets each thread get a fair amount of execution time (every 10 microseconds). [source: thread.c:1018]
We can see an example of the GIL/GVL in action here:
Normally this would be an unsafe operation, but since the GIL/GVL exists we don’t have to worry about two threads adding to the same ruby object at once since only one thread can run on the VM at once and it ends up being an atomic operation (although don’t rely on this quirk for thread safety, it definitely doesn’t apply to any other VMs).
Another important note is that the Ruby GC is doing a really horrible job during this benchmark.
The memory kept growing so I had to kill the process after a few seconds.
Threading with JRuby on the JVM
JRuby specifies the use of native threads based on the operating system support using the getNativeThread call [2]. JRuby’s implementation of threads using the JVM means there is no GIL/GVL. This allows CPU bound processes to utilize all cores of a machine without having to deal with forking (which, in the case of resque, can be very expensive).
When trying to execute the GIL safe code above JRuby spits out a concurrency error: ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users
We can either add a mutex around this code or modify it to not worry about concurrent access. I chose the latter:
Compared to the MRI version, ruby running on the JVM was able to make some optimizations and keep memory usage around 800MB for the duration of the test:
Now that we have a better understanding of the differences between forking and threading in Ruby, lets move on to Sidekiq and Resque.
Sidekiq and Resque
Resque’s view of the world
Resque assumes chaos in your environment. It follows the forking model with C and ruby and makes a complete copy of each resque parent when a new job needs to be run. This has its advantages in preventing memory leaks, long running workers, and locking. You run into an issue with forking though when you need to increase the amount of workers on a machine. You end up not having enough spare CPU cycles since the majority are being taken up handling all the forking.
Resque follows a simple fork and do work model, each worker will take a job off the queue and fork a new process to do the job.
Unlike Resque, Sidekiq uses threads and is extremely easy to use as a drop in replacement to Resque since they both work on the same perform method. When you dig into the results below you can see that Sidekiq’s claim of being able to handle a larger number of workers and amount of work is true. Due to using threads and not having to allocate a new stack and address space for each fork, you get that overhead back and are able to do more work with a threaded model.
Sidekiq follows the actor pattern. So compared to Resque which has N workers that fork, Sidekiq has an Actor manager, with N threads and one Fetcher actor which will pop jobs off Redis and hand them to the Manager. Sidekiq handles the “chaos” portion of Resque by catching all exceptions and bubbling them up to an exception handler such as Airbrake or Errbit.
Now that we know how Sidekiq and Resque work we can get on to testing them and comparing the results.
Model Name: Mac Pro
Model Identifier: MacPro4,1
Processor Name: Quad-Core Intel Xeon
Processor Speed: 2.26 GHz
Number of Processors: 2
Total Number of Cores: 8
L2 Cache (per Core): 256 KB
L3 Cache (per Processor): 8 MB
Memory: 12 GB
Processor Interconnect Speed: 5.86 GT/s
This gives us a total of 16 cores to use for our testing. I’m also using a Crucial M4 SSD
Results
Time to Process 150,000 sets of 20 numbers
Type
Time to Completion (seconds)
Sidekiq (JRuby) 150 Threads
88
Sidekiq (JRuby) 240 Threads
89
Sidekiq (JRuby) 50 Threads
91
Sidekiq (MRI) 5x50
98
Sidekiq (MRI) 3x50
120
Sidekiq (MRI) 50
312
Resque 50
396
All about the CPU
Resque: 50 workers
Here we can see that the forking is taking its toll on the available CPU we have for processing. Roughly 50% of the CPU is being wasted on forking and scheduling those new processes. Resque took 396 seconds to finish and process 150,000 jobs.
Sidekiq (MRI) 1 process, 50 threads
We’re not fully utilizing the CPU. When running this test it pegged one CPU at 100% usage and kept it there for the duration of the test. We have a slight overhead with system CPU usage. Sidekiq took 312 seconds with 50 threads using MRI Ruby. Lets now take a look at doing things a bit resque-ish, and use multiple sidekiq processes to get more threads scheduled across multiple CPUs.
Sidekiq (MRI) 3 processes, 50 threads
We’re doing better. We’ve cut our processing time roughly in third and we’re utilizing more of our resources (CPUs). 3 Sidekiq processes with 50 threads each (for a total of 150 threads) took 120 seconds to complete 150,000 jobs.
Sidekiq (MRI) 5 processes, 50 threads
As we keep adding more processes that get scheduled to different cores we’re seeing the CPU usage go up even further, however with more processes comes more overhead for process scheduling (versus thread scheduling). We’re still wasting CPU cycles, but we’re completing 150,000 jobs in 98 seconds.
Sidekiq (JRuby) 50 threads
We’re doing much better now with native threads. With 50 OS level threads, we’re completing our set of jobs in 91 seconds.
Sidekiq (JRuby) 150 threads & 240 Threads
We’re no longer seeing a increase in (much) CPU usage and only a slight decrease in processing time. As we keep adding more and more threads we end up running into some thread contention issues with accessing redis and how quickly we can pop things off the queue.
Overview
Even if we stick with the stock MRI ruby and go with Sidekiq, we’re going to see a huge decrease in CPU usage while also gaining a little bit of performance as well.
Sidekiq, overall, provides a cleaner, more object oriented interface (in my opinion) to inspecting jobs and what is going on in the processing queue.
In Resque you would do something like: Resque.size("queue_name"). However, in Sidekiq you would take your class, in this case, POR and call POR.jobs to get the list of jobs for that worker queue. (note: you need to require 'sidekiq/testing' to get access to the jobs method).
The only thing I find missing from Sidekiq that I enjoyed in Resque was the ability to inspect failed jobs in the web UI. However Sidekiq more than makes up for that with the ability to automatically retry failed jobs (although be careful you don’t introduce race conditions and accidentally DOS yourself).
And of course, JRuby comes out on top and gives us the best performance and bang for the buck (although your mileage may vary, depending on the task).
I saw the question of “How can I prevent a class from being reopened again in Ruby?” pop up on the Ruby mailing list. While this is somewhat against the nature of ruby, it can be accomplished:
1234567891011
classFoodefFoo.method_added(name)raise"This class is closed for modification"endendclassFoodeftestingp"test"endend
This will raise an exception anytime someone tries to reopen the class.
When you work on your code and are finished for the day, is what you have committed worry free? If another developer were to push your code in the middle of the night, would they be calling you at 3am?
Let’s see how we can improve our development cycle with testing so we can avoid those early morning calls. We’ll go over some of the basics with a simple project to start.
The most important part about TDD is getting quick feedback based on our desired design (the feedback loop).
Here is an example of how fast the tests run:
While this is a somewhat contrived example for the reddit cli we’re making, this can be applied equally as well when writing Rails applications. Only load the parts you need (ActionMailer, ActiveSupport, etc), usually you don’t need to load the entire rails stack. This can make your tests run in milliseconds instead of seconds. This lets you get feedback right away.
Before we go further into the testing discussion, lets setup a spec helper.
spec/spec_helper.rb
1234567
require'rspec'require'vcr'require'pry'VCR.configuredo|c|c.cassette_library_dir='fixtures/vcr_cassettes'c.hook_into:fakeweb# or :fakewebend
Now how do we start doing TDD? We first start with a failing test.
Reddit API Spec (Pass 1) - spec/lib/reddit_api_spec
1234567891011
require'spec_helper'require'./lib/reddit_api'describeRedditApidolet(:reddit){RedditApi.new('ProgrammerHumor')}context"#initializing"doit"should form the correct endpoint"doreddit.url.shouldeq"http://reddit.com/r/ProgrammerHumor/.json?after="endendend
When we create a new instance of the Reddit API we want to pass it a subreddit, and then we want to make sure it builds the URL properly.
Next we want to make the actual HTTP request to the Reddit api and process it.
Reddit API Spec (Pass 2) - spec/lib/reddit_api_spec
123456789101112131415161718192021
require'spec_helper'require'./lib/reddit_api'describeRedditApidolet(:reddit){RedditApi.new('ProgrammerHumor')}context"#initializing"doit"should form the correct endpoint"doVCR.use_cassette('reddit_programmer_humor')doreddit.url.shouldeq"http://reddit.com/r/ProgrammerHumor/.json?after="endendendcontext"#fetching"doit"should fetch the first page of stories"doVCR.use_cassette('reddit_programmer_humor')doreddit.stories.count.shouldeq(25)endendendend
We’ve now added a VCR wrapper and added an expectation that the reddit api will return a list of stories. We use VCR here to again ensure that our tests run fast. Once we make the first request, future runs will take milliseconds and will hit our VCR tape instead of the API.
Now we need to introduce three new areas: requesting, processing, and a Story object class.
What can we do now? The API lets us make a full request and get a list of Story struct objects back. We’ll be using this array of structs later on to build the CLi.
The only thing left for this simple CLI a way to get to the next page. Let’s add our failing spec:
Reddit API Spec (Pass 3) - spec/lib/reddit_api_spec
123456789101112131415161718192021222324252627
require'spec_helper'require'./lib/reddit_api'describeRedditApidolet(:reddit){RedditApi.new('ProgrammerHumor')}context"#initializing"doit"should form the correct endpoint"doVCR.use_cassette('reddit_programmer_humor')doreddit.url.shouldeq"http://reddit.com/r/ProgrammerHumor/.json?after="endendendcontext"#fetching"doit"should fetch the first page of stories"doVCR.use_cassette('reddit_programmer_humor')doreddit.stories.count.shouldeq(25)endendit"should fetch the second page of stories"doVCR.use_cassette('reddit_programmer_humor_p2')doreddit.next.stories.count.shouldeq(25)endendendend
We also allow method chaining since we return self after calling next (so you could chain next’s for instance).
Another important principal to keep in mind is the “Tell, Dont Ask” rule. Without tests, we might have gone this route:
bad_example.rb
123
@reddit=Reddit.new('ProgrammerHumor')# User presses next @reddit.url="http://reddit.com/r/ProgrammerHumor/.json?after=sometoken"
Not only would we not be telling the object what we want, we would be modifying the internal state of an object as well. By implementing a next method we abstract the idea of a URL and any tokens we may need to keep track of away from the consumer. Doing TDD adds a little extra step of “Thinking” more about what we want our interfaces to be. What’s easier? Calling next or modifying the internal state?
I’m kind of cheating a bit here. I found a nice “table” gem that outputs what you send in as a formatted table (think MySQL console output). Let’s just make sure everything is being sent around properly and STDOUT is printing the correct contents:
Reddit CLI Spec (Pass 1) - spec/lib/reddit-cli.rb
123456789101112131415161718192021222324
require'spec_helper'require'stringio'require'./lib/reddit-cli'describeRedditClidolet(:subreddit){"ProgrammerHumor"}context"#initializing"dobefore(:all)do$stdout=@fakeout=StringIO.newendit"should print out a story"doapi_response=double(RedditApi)api_response.stub!(:stories=>[Story.new("StoryTitle","Score","Comments","URL")])$stdin.should_receive(:gets).and_return("q")cli=RedditCli.new(api_response)$stdout=STDOUT@fakeout.string.include?('StoryTitle').shouldbe_trueendendend
We’re doing several things here. First we’re taking $stdout and putting it (temporarily) into a instance variable so we can see what gets outputted. Next we’re mocking out the RedditApi since we dont actually need to hit that class or the VCR tapes, we just need to stub out the expected results (stories) and pass the response object along to the CLI class. And finally once we’re finished we set $stdout back to the proper constant.
require'./lib/reddit_api'require'terminal-table'classRedditClidefinitialize(api)@rows=[]@api=api@stories=api.storiesprint_storiesprint"\nType ? for help\n"promptenddefprint_stories@stories.each_with_index{|x,i|@rows<<[i,x.score,x.comments,x.title[0..79]]}putsTerminal::Table.new:headings=>['#','Score','Comments','Title'],:rows=>@rowsenddefpromptprint"\n?> "input=STDIN.gets.chompcaseinputwhen"?"p"Type the # of a story to open it in your browser"p"Type n to go to the next page"promptwhen"quit","q"when"n"@rows=[]@stories=@api.next.storiesprint_storiespromptelseprint"#=> Oepning: #{@stories[input.to_i].url}"`open #{@stories[input.to_i].url}`promptendendend
And finally, a little wrapper in the root directory:
When working with external resources, whether it be a gem or a remote API, it’s important to wrap those endpoints in your own abstraction. For instance, with our Reddit CLI we could have avoided those first 2 classes entirely, written everything in the CLI display class, and worked with the raw JSON. But what happens when Reddit changes their API? If this CLI class was huge or incoporated many other components, this could be quite a big code change. Instead, what we wrote encapsulates the API inside a RedditApi class that returns a generic Story struct we can work with and pass around. We don’t care if the API changes in the CLI, or in any other code. If the API changes, we only have to update the one API class to mold the new API to the output we were already generating.
I was working on my blog and moving some posts around when I kept getting a Psych::SyntaxError when generating it with Jekyll and ruby 1.9.x. Unfortunately the default stack trace doesn’t provide much information on what file was causing the issue, so a quick way to find out is opening up irb:
Now you can access your parameters being passed in as instance variables for an object. You can extract this out into a method to apply to all objects or just make a simple extension to include it in files that you wanted to use it in. While this is a trivial example, for methods with longer signatures this becomes a more appealing approach. I’ll probably extract this out into a gem and post it here later.
