Josh Rendek

<3 Ruby

Go-lang Compare *ssh.Request.Type Against a String

I was working on the agent for SSH Pot and ran into something interesting last night. A lot of the brute force attempts attempt to run a command like this:

Example Exec
1
ssh user@host 'uname'

This is different than:

Example Shell
1
2
ssh user@host
$ uname

The first command is executing a command then exiting, the second is actually logging in and giving the user a shell. The first requests a exec subsystem and the second requests a shell subsystem - so there are two ways to handle it.

broken_handler.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
  for req := range in {
      ok := true
      logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
      switch req.Type {
      case "shell":
          req.Reply(ok, nil)
      case "exec":
          if string(req.Payload) == string("uname") {
              channel.Write([]byte("\n\rLinux\n\r"))
          }

          channel.Close()
      }
  }
}

When logging in my logfile it would show something like:

Log
1
[request exec]: uname

And even when comparing the two side by side with something like this:

log.go
1
logfile.Println("["+string(req.Payload)+"]:["+"uname"+"]")

I would get this output:

Log
1
[uname]:[uname]

Yet the comparison on line 9 would not get hit. After sitting and thinking about it for a while I decided to print the bytes out:

Log
1
2
INFO: 2014/07/07 23:15:18 sshd.go:157: [0 0 0 5 117 110 97 109 101]
INFO: 2014/07/07 23:15:18 sshd.go:158: [117 110 97 109 101]

Aha! So for some reason req.Payload is padded with 3 null bytes and a ENQ byte (hex 5).

Here is the corrected version removing the correct bytes - now the string comparison works:

working_handler.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
  for req := range in {
      ok := true
      logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
      switch req.Type {
      case "shell":
          req.Reply(ok, nil)
      case "exec":
          if string(req.Payload[4:]) == string("uname") {
              channel.Write([]byte("\n\rLinux\n\r"))
          }

          channel.Close()
      }
  }
}

Go-lang: Mocking exec.Command Using Interfaces

This is a short example showing how to use an interface to ease testing, and how to use an interface with running shell commands / other programs and providing mock output.

Source on Github

Here is our main file that actually runs the commands and prints out “hello”.

example.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main

import (
  "fmt"
  "os/exec"
)

// first argument is the command, like cat or echo,
// the second is the list of args to pass to it
type Runner interface {
  Run(string, ...string) ([]byte, error)
}

type RealRunner struct{}

var runner Runner

// the real runner for the actual program, actually execs the command
func (r RealRunner) Run(command string, args ...string) ([]byte, error) {
  out, err := exec.Command(command, args...).CombinedOutput()
  return out, err
}

func Hello() string {
  out, err := runner.Run("echo", "hello")
  if err != nil {
      panic(err)
  }
  return string(out)
}

func main() {
  runner = RealRunner{}
  fmt.Println(Hello())
}

Here is our test file. We start by defining our TestRunner type and implementing the Run(...) interface for it.

This function builds up a command to run the current test file and run the TestHelperProcess function passing along all the args you originally sent. This lets you do things like return different output for different commands you want to run.

The TestHelperProcess function exits when run in the context of the test file, but runs when specified in the files arguments.

example_test.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main

import (
  "fmt"
  "os"
  "os/exec"
  "testing"
)

type TestRunner struct{}

func (r TestRunner) Run(command string, args ...string) ([]byte, error) {
  cs := []string{"-test.run=TestHelperProcess", "--"}
  cs = append(cs, args...)
  cmd := exec.Command(os.Args[0], cs...)
  cmd.Env = []string{"GO_WANT_HELPER_PROCESS=1"}
  out, err := cmd.CombinedOutput()
  return out, err
}

func TestHello(t *testing.T) {
  runner = TestRunner{}
  out := Hello()
  if out == "testing helper process" {
      t.Logf("out was eq to %s", string(out))
  }
}

func TestHelperProcess(*testing.T) {
  if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
      return
  }
  defer os.Exit(0)
  fmt.Println("testing helper process")
}

Hopefully this helps someone else! I had a hard time finding some good, short examples on the internet that combined both interfaces and mocking like this.

More examples from os/exec/exec_test.go

A Useful Logger in Go

Small function that will print out useful information when invoked:

log.go
1
2
3
4
5
6
7
8
9
10
func logMsg(msg string) {
  pc, _, _, _ := runtime.Caller(1)
  caller := runtime.FuncForPC(pc).Name()
  _, file, line, _ := runtime.Caller(0)
  sp := strings.Split(file, "/")
  short_path := sp[len(sp)-2 : len(sp)]
  path_line := fmt.Sprintf("[%s/%s:%d]", short_path[0], short_path[1], line)
  log_string := fmt.Sprintf("[%s]%s{%s}:: %s", time.Now(), path_line, caller, msg)
  fmt.Println(log_string)
}

Sample output:

1
2
[2014-06-10 01:38:45.812215998 +0000 UTC][src/trex-client.go:15]{main.main}:: checking jobs - finish
[2014-06-10 01:38:47.329650331 +0000 UTC][src/trex-client.go:15]{main.main}:: building package list - start

Motivation to Work on New Projects

Whenever I have spare time ( often around Christmas or when I’m on vacation/traveling ), I tend to fill it with working on projects I’ve built up in my backlog. I’m also really trying to keep a continuous streak of OSS commits going on Github (something about filling that chart up makes me want to work harder). Here’s my process and how I go about working on personal projects and try to stay motivated - if you have any ideas I’d love to hear them in the comments!

Have a backlog

I use Evernote for all my ideas and project notes:

Evernote

I have two columns - one for things in progress or to do and one for projects that are done ( with a link to any github repos I published ). When I have some downtime but don’t feel like actually writing any code - I’ll write out plans for what the project needs (use cases, backend needs, software I plan on using, etc) and do research and store all that as a sub-note in Evernote (you can see that with the light green link to the HAProxy Frontend ) under the main page. Plus I can easily share these with friends for feedback by just copying the share URL.

Use small milestones to build up bigger ones

For instance, when I was working on the code for http://ifcfg.net/ I decided there were two major components I would need to create: the web api to access the data, and then a backing library to do some web scraping to gather BGP data. I started out writing a small scraper in Scala for scraping BGP and looking glass info (which involved learning some more SBT, and selenium apis for Scala) and then moved onto learning a small amount of the Play! framework and exposing my library via that api. This let me focus on one small component at a time and finish it ( I have a habit of leaving personal projects unfinished or taking a long time to finish them if I let the scope creep beyond what I deemed as minimum requirements ).

Pick an interesting project

There are some areas I just don’t have an interest in - like writing an application to track golf scores.

So pick something you like - I love doing backend systems and APIs - pick something your passionate about already or a topic you want to learn more about.

Learn

If I’m working on a personal project and not learning anything new (even if its just a new way to test, for instance) - I get bored, really quickly. I’ve been stemming this by trying to pick up new languages as I work on projects and working on projects with broader goals.

For instance, my latest project I’m working on is Patchasaurus ( yes there isn’t a readme yet ). I know theres a gap in the systems world for (open source) patch management, especially focused on Ubuntu and Debian - so I decided to write a small initial version of one. I had been playing around with Go at work (and boy is it nice to get a HTTP API running in a few MB of RAM) and decided to write the agent for patchasaurus in that ( nicknamed trex ). I’ve been learning how to cross compile programs in Go, what libraries don’t work with cross-compilation (looking at you os/user) and a nice work flow for testing these while developing them ( sshfs is great for this with VirtualBox or Vagrant ). I also chose to use Rails 4.1 as the management interface since I wanted to stay up to date with the new Rails features - turns out spring is very nice and a great improvement over the guard work flow I’ve used before.

Don’t focus on processes versus getting things done

I’m a big fan of testing, and TDD, however I’m not always in the mood to do it. Sometimes I just want to see results and I’ll go back and refactor and test later. Picking what works for you on a specific project/component, and getting it done I think is much more important than rigidly following a specific set of guidelines on every project you do ( aka: test first, setup CI before any code, etc ).

Don’t get in a rut

Staring at HackerNews or Reddit all day can be daunting - try and not focus on what everyone else is doing and instead focus on what you’re getting done and how you’re improving yourself.

Also don’t let this influence your technology choices. Sometimes there are articles trending for AngularJS or Ruby on Rails - stick with what you picked ( unless you really want to learn that new tech ) - or figure out ways to incorporate that into smaller components of your project. Don’t throw away all that progress just because you saw a few posts reach the page!

Take breaks

Don’t spend all day coding - take breaks, go for a walk, a run, play with your dog, play a video game - something that can give you a moment to breathe and think about something else or give you time to re-focus on the grand vision you’ve been laboring over. Figure out what works for you to relax and do it to break up that screen glow tan you’re getting.

Talk about what you’re working on

Talk with friends to brainstorm ideas, pair up on some problems, see if theres a more idiomatic way to do a function in the language your using ( for example, I spent some time trying to see if there were any map() equivalents on #go-nuts), and blog about what you’re doing if that’s your style.

Knowing people are using code and software I’ve written is a huge motivating factor to working on future projects ( star/watch counts on Github, downloads on RubyGems, traffic to my blog, etc).

Finish!!

Yes it can be hard, but figure out what finished means to you, and do it. Publish it on Github, submit it to HackerNews, post it to reddit, get it hooked into TravisCI - make sure you come to the finish line of each component or project you’re working on. Building up these small accomplishments can help set a streak for the future so you have the motivation to power through and get items done.

Sometimes you’re more interested in getting an application finished than on the deployment process - throw it on Heroku, a shared hosting provider, etc. There’s nothing wrong with some shared hosting for a small project. Don’t let things like deployment stop you from finishing!

2 Patterns for Refactoring With Your Ruby Application

When working on a rails application you can sometimes find duplicated or very similar code between two different controllers (for instance a UI element and an API endpoint). Realizing that you have this duplication there are several things you can do. I’m going to go over how to extract this code out into the query object pattern 1 and clean up our constructor using the builder pattern 2 adapted to ruby.

I’m going to make a few assumptions here, but this should be applicable to any data access layer of your application. I’m also assuming you’re using something like Kaminari for pagination and have a model for People.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
11
def index
  page = params[:page] || 1
  per_page = params[:per_page] || 50
  name = params[:name]
  sort = params[:sort_by] || 'last_name'
  direction = params[:sort_direction] || 'asc'

  query = People
  query = query.where(name: name) if name.present?
  @results = query.order("#{sort} #{direction}").page(page).per_page(per_page)
end

So we see this duplicated elsehwere in the code base and we want to clean it up. Lets first start by extracting this out into a new class called PeopleQuery.

I usually put these under app/queries in my rails application.

people_query.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class PeopleQuery
  attr_accessor :page, :per_page, :name, :sort, :direction, :query
  def initialize(page, per_page, name, sort, direction)
    self.page = page || 1
    self.per_page = per_page || 50
    self.name = name
    self.sort = sort || 'last_name'
    self.direction = direction || 'asc'
    self.query = People
  end

  def build
    self.query = self.query.where(name: self.name) if self.name.present?
    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
  end
end

Now our controller looks like this:

dummy_controller.rb
1
2
3
4
def index
  query = PeopleQuery.new(params[:page], params[:per_page], params[:name], params[:sort], params[:direction])
  @results = query.build
end

Much better! We’ve decoupled our control from our data access object (People/ActiveRecord), moved some of the query logic outside of the controller and into a specific class meant to deal with building it. But that constructor doesn’t look very nice. We can do better since we’re using ruby.

Our new PeopleQuery class will look like this and will use a block to initialize itself instead of a long list of constructor arguments.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class PeopleQuery
  attr_accessor :page, :per_page, :name, :sort, :direction, :query
  def initialize(&block)
    yield self
    self.page ||= 1
    self.per_page =|| 50
    self.sort ||= 'last_name'
    self.direction ||= 'asc'
    self.query = People
  end

  def build
    self.query = self.query.where(name: self.name) if self.name.present?
    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
  end
end

We yield first to let the caller set the values and then after yielding we set our default values if they weren’t passed in. There is another method of doing this with instance_eval but you end up losing variable scope and the constructor looks worse since you have to start passing around the params variable to get access to it, so we’re going to stick with yield.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
def index
  query = PeopleQuery.new do |query|
    query.page = params[:page]
    query.per_page = params[:per_page]
    query.name = params[:name]
    query.sort = params[:sort]
    query.direction = params[:direction]
  end
  @results = query.build
end

And that’s it! We’ve de-duplicated some code (remember we assumed dummy controller’s index method was duplicated elsewhere in an API call in a seperate namespaced controller), extracted out a common query object, decoupled our controller from ActiveRecord, and built up a nice way to construct the query object using the builder pattern.

Parsing HTML in Scala

Is there ever a confusing amount of information out there on parsing HTML in Scala. Here is the list of possible ways I ran across:

  • Hope the document is valid XHTML and use scala.xml.XML to parse it
  • If the document isn’t valid XHTML use something like TagSoup and hope it parses again
  • Still think its valid XHTML? Try using scalaz’s XML parser

All of the answers I found on Google pointed to some type of XML parsing, which won’t always work. Coming from Ruby I know there are tools out there like Selenium that can simulate a web browser for you and give you a rich interface to interact with the returned HTML.

So I went on Maven and found the two Selenium web drivers I wanted for my project and added them to my libraryDependencies:

1
2
"org.seleniumhq.webdriver" % "webdriver-selenium" % "0.9.7376",
"org.seleniumhq.webdriver" % "webdriver-htmlunit" % "0.9.7376"

The project I’m working on is to parse Looking Glass websites for BGP information and AS peering, so I wanted to scrape the data. I also didn’t want to have to use a full blown web browser (ala Selenium + Firefox for instance) - so I stuck with the HtmlUnit driver for the implementation.

Here is a quick code snippet that lets me grab AS #’s and Peer names from an AS:

AS.scala
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
val url = "http://example.com/AS" + as.toString

val driver = new HtmlUnitDriver
// Proxy for BetaMax when writing tests
if (_port != null) {
  driver.setProxy("localhost", _port)
}
driver.get(url)

val peers = driver.findElementsByXPath("//*[@id=\"table_peers4\"]/tbody/tr/td[position() = 1 or position() = 2]")

// zip up the list in pairs so List(a,b,c,d) becomes List((a,b), (c,d))
for(peer <- peers zip peers.tail) {
  println(peer)
}

No XML to muck with and I get some nice selectors to query the document for. Remember if the source you want data from doesn’t have an API, HTML is an API! Just be respectful of how you query and interact with them (ie: Don’t do 100 requests/second, cache/record responses while writing tests, etc).

Getting Started With Scala

Recently I’ve been getting into more Java and (attempting to) Scala development. I always got annoyed with the Scala ecosystem for development and would get fed up and just go back to writing straight Java (coughsbtcough). Today I decided to write down everything I did and get a sane process going for Scala development with SBT.

I decided to write a small Scala client for OpenWeatherMap - here is what I went through.

A brief guide on naming conventions is here. I found this useful just to reference conventions since not everything is the same as Ruby (camelCase vs snake_case for instance).

Setting up and starting a project

First make sure you hava a JVM installed, Scala, and SBT. I’ll be using Scala 2.10.2 and SBT 0.12.1 since that is what I have installed.

One of the nice things I like about Ruby on Rails is the project generation ( aka: rails new project [opts] ) so I was looking for something similar with Scala.

Enter giter8: https://github.com/n8han/giter8

giter8 runs through SBT and has templates available for quickstart.

Follow the install instructions and install giter8 into SBT globally and load SBT to make sure it downloads and installs.

Once you do that you can pick a template from the list, or go with the one I chose: fayimora/basic-scala-project which sets up the directories properly and also sets up ScalaTest, a testing framework with a DSL similar to RSpec.

To setup your project you need to run:

1
g8 fayimora/basic-scala-project

You’ll be prompted with several questions and then your project will be made. Switch into that directory and run sbt test to make sure the simple HelloWorld passes and everything with SBT is working.

Setting up IntelliJ

For Java and Scala projects I stick with IntelliJ over my usual vim. When using Java IntelliJ is good about picking up library and class path’s and resolving dependencies (especially if you are using Maven). However there isn’t a good SBT plugin (as of writing this) that manages to do all this inside IntelliJ.

The best plugin for SBT I’ve found that does this is sbt-idea. You’re going to need to make a project/plugins.sbt file:

plugins.sbt
1
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")

and now you can generate your .idea files by running: sbt gen-idea

IntelliJ should now resolve your project dependencies and you can start coding your project.

Final Result

scala-weather - A simple to use OpenWeatherMap client in Scala set up with Travis-CI and CodeClimate. This is just the first of several projects I plan on working on / open sourcing to get my feet wet with Scala more.

Useful libraries

Notes

By default Bee Client will log everything to STDOUT - you’ll need to configure logback with an XML file located in src/main/resources/logback.xml:

src/main/resources/logback/xml
1
2
3
4
5
6
7
8
9
10
11
12
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="ERROR">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

From 0 to Testing on Windows With JRuby

Testing is one of the most important parts of software development and helps to ensure bugs don’t get into production and that code can be refactored safely. If you’re working on a team with multiple people with different skill sets, you might have people doing testing who only know windows and development is only using OSX or Linux. We want everyone to be able to test - someone in QA who is familiar with Windows shouldn’t have to throw away all that knowledge, install Linux, and start from scratch. Enter JRuby and John.

John is our tester and he is running windows. He wants to help make sure that when a user goes to http://google.com/ that a button appears with the text “Google Search”. The quick way to do this is to open his browser, navigate to http://google.com/ glance through the page for the button and confirm that its there. John has a problem though, he has 30 other test cases to run and the developers are pushing code to the frontpage several times a day; John now has to continously do this manually everytime code is touched and his test load is piling up.

So let’s help John out and install Sublime Text 2 and JRuby.

Start by downloading the 64-bit version of Sublime Text. Make sure to add the context menu when going through the install process.

Now we’ll visit the JRuby homepage and download the 64 bit installer.

Go through the installer and let JRuby set your path so you can access ruby from cmd.exe

Now when we open cmd.exe and type jruby -v we’ll be able to see that it was installed.

Now that we have our tools installed lets setup our test directory on the Desktop. Inside our testing folder we’ll create a folder called TestDemo for our tests for the Demo project.

Next we’ll open Sublime Text and go to File > Open Folder and navigate to our TestDemo folder and hit open.

Now we can continue making our directory structure inside Sublime Text. Since we’re going to use rspec we need to create a folder called spec to contain all of our tests. Right click on the TestDemo in the tree navigation and click New Folder.

Call the folder spec in the bottom title bar when it prompts you for the folder name.

Next we’ll create our Gemfile which will declare all of our dependencies - so make a file in the project root called Gemfile and put the our dependencies in it:

Gemfile
1
2
3
4
5
6
source "https://rubygems.org"

gem "rspec"
gem "selenium"
gem "selenium-webdriver"
gem "capybara"

Once we have that file created, open cmd.exe and switch to your project’s root directory.

Type jgem install bundler to install bundler which manages ruby dependencies.

While still at the command prompt we’re going to bundle to install our dependencies:

After that finishes we need to run one last command for selenium to work properly: selenium install

We also need a spec_helper.rb file inside our spec directory.

specs\spec_helper.rb
1
2
3
4
5
require "rspec"
require "selenium"
require "capybara/rspec"

Capybara.default_driver =  :selenium

We’ve now setup our rspec folders, our Gemfile with dependencies, and installed them. Now we can write the test that will save John a ton of time.

Chrome comes with a simple tool to get XPath paths so we’re going to use that to get the XPath for the search button. Right click on the “Google Search” button and click Inspect element

Right click on the highlighted element and hit Copy XPath.

Now we’re going to make our spec file and call it homepage_spec.rb and locate it under spec\integration.

Here is a picture showing the directory structure and files:

Here is the spec file with comments explaining each part:

spec\integration\homepage_spec.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# This loads the spec helper file that we required everything in
require "spec_helper"

# This is the outer level description of the test
# For this example it describes going to the homepage of Google.com
# Setting the feature type is necessary if you have
# Capybara specs outside of the spec\features folder
describe "Going to google.com", :type => :feature do

  # Context is like testing a specific component of the homepage, in this case
  # its the search button
  context "The search button" do
    # This is our actual test where we give it a meaningful test description
    it "should contain the text 'Google Search'" do
      visit "http://google.com/" # Opens Firefox and visits google
      button = find(:xpath, '//*[@id=gbqfba"') # find an object on the page by its XPath path
      # This uses an rspec assertion saying that the string returned
      # by button.text is equal to "Google Search"
      button.text.should eq("Google Seearch")

    end
  end

end

Now we can tab back to our cmd.exe prompt and run our tests! rspec spec will run all your tests under the spec folder.

Things to take note of

This example scenario is showing how to automate browser testing to do end-to-end tests on a product using rspec. This is by no means everything you can do with rspec and ruby - you can SSH, hit APIs and parse JSON, and do anything you want with the ability to make assertions.

A lot is going on in these examples - there are plenty of resources out there on google and other websites that provide more rspec examples and ruby examples.

We also showed how to add dependencies and install them using bundler. Two of the best resources for finding libraries and other gems is RubyGems and Ruby-Toolbox - the only thing to take note of is anything saying to be a native C extension (they won’t work with JRuby out of the box).

My last note is that you also need to have firefox installed as well - Selenium will work with Chrome but I’ve found it to be a hassle to setup (and unless you really need Chrome), the default of Firefox will work great.

A Simple Ruby Plugin System

Let’s start out with a simple directory structure:

1
2
3
4
5
6
7
8
.
├── plugin.rb
├── main.rb
└── plugins
    ├── cat.rb
    └── dog.rb

1 directory, 3 files

All the plugins we will use for our library will be loaded from plugins. Now lets make a simple Plugin class and register our plugins.

plugin.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class Plugin
  # Keep the plugin list inside a set so we don't double-load plugins
  @plugins = Set.new

  def self.plugins
    @plugins
  end

  def self.register_plugins
    # Iterate over each symbol in the object space
    Object.constants.each do |klass|
      # Get the constant from the Kernel using the symbol
      const = Kernel.const_get(klass)
      # Check if the plugin has a super class and if the type is Plugin
      if const.respond_to?(:superclass) and const.superclass == Plugin
        @plugins << const
      end
    end
  end
end

We’ve now made a simple class that will contain all of our plugin data when we call register_plugins.

Now for our Dog and Cat classes:

dog.rb
1
2
3
4
5
6
7
class DogPlugin < Plugin

  def handle_command(cmd)
    p "Command received #{cmd}"
  end

end
cat.rb
1
2
3
4
5
6
7
class CatPlugin < Plugin

  def handle_command(cmd)
    p "Command received #{cmd}"
  end

end

Now combine this all together in one main entry point and we have a simple plugin system that lets us send messages to each plugin through a set method ( handle_command ).

main.rb
1
2
3
4
5
6
7
8
require './plugin'
Dir["./plugins/*.rb"].each { |f| require f }
Plugin.register_plugins

# Test that we can send a message to each plugin
Plugin.plugins.each do |plugin|
  plugin.handle_command('test')
end

This is a very simple but useful way to make a plugin system to componentize projects like a chat bot for IRC.

Why Setuid Is Bad and What You Can Do

Why setuid is Bad

setuid allows a binary to be run as a different user then the one invoking it. For example, ping needs to use low level system interfaces (socket, PF_INET, SOCK_RAW, etc) in order to function properly. We can watch this in action by starting ping in another terminal window ( ping google.com ) and then using strace to see the syscall’s being made:

sudo strace -p PID and we get the following:

strace output
1
2
3
munmap(0x7f329e7ea000, 4096)            = 0stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=185, ...}) = 0
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0

We can find all setuid programs installed by issuing the command:

How to find all setuid programs
1
sudo find / -xdev \( -perm -4000 \) -type f -print0 -exec ls -l {} \;

This will find all commands that have the root setuid bit set in their permission bit.

setuid list for a few popular operating systems:

Of particular interest in OpenBSD, where a lot of work was done to remove and switch programs from needing to use setuid/gid permissions. OpenIndiana is the worst offender and has the widest vector for attack.

setuid escalation is a common attack vector and can allow unprivileged code to be executed by a regular user, and then escalate itself to root and drop you in on the root shell.

Here are a few examples:

CVE-2012-0056: Exploiting /proc/pid/mem

http://blog.zx2c4.com/749 - C code that uses a bug in the way the Linux kernel checked permissions on /proc/pid/mem and then uses that to exploit the su binary to give a root shell.

CVE-2010-3847: Exploiting via $ORIGIN and file descriptors

http://www.exploit-db.com/exploits/15274/ - By exploiting a hole in the way the $ORIGIN is checked, a symlink can be made to a program that uses setuid and exec‘d ‘to obtain the file descriptors which then lets arbitrary code injection (in this case a call to system("/bin/bash")).

More of these can be found at http://www.exploit-db.com/shellcode/ and just searching google for setuid exploits.

So you may not want to completely disable the setuid flag on all the binaries for your distribution, but we can turn on some logging to watch when they’re getting called and install a kernel patch that will secure the OS and help prevent 0-days that may prey on setuid vulnerabilities.

How to log setuid calls

I will detail the steps to do this on Ubuntu, but they should apply to the other audit daemons on CentOS.

Let’s first install auditd: sudo apt-get install auditd

Let’s open up /etc/audit/audit.rules, and with a few tweaks with vim, we can insert the list we generated with find into the audit rule set (explanation of each flag after the jump):

/etc/audit/audit.rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# This file contains the auditctl rules that are loaded# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# Feel free to add below this line. See auditctl man page

-a always,exit -F path=/usr/lib/pt_chown -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/eject/dmcrypt-get-device -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/dbus-1.0/dbus-daemon-launch-helper -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/lib/openssh/ssh-keysign -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/uuidd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/sbin/pppd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/at -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/mtr -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudoedit -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/traceroute6.iputils -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/sudo -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/chfn -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/gpasswd -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/usr/bin/newgrp -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/fusermount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/ping6 -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
audtid Options Explained
1
2
3
4
5
6
7
-a: appends the always, and exit rules. This says to always make a log at syscall entry and syscall exit.
-F
     path= says filter to the executable being called
     perm=x says filter on the program being executable
     auid>= says log all calls for users who have a UID above 500 (regular user accounts start at 1000 generally)
     auid!=4294967295 sometimes a process may start before the auditd, in which case it will get a auid of 4294967295
-k passes a filter key that will be put into the record log, in this case its "privileged"

So now when we run ping google.com we can see a full audit trail in /var/log/audit/audit.log:

auditd output
1
2
3
4
5
type=SYSCALL msg=audit(1361852594.621:48): arch=c000003e syscall=59 success=yes exit=0 a0=f43de8 a1=d40488 a2=ed8008 a3=7fffc9c9a150 items=2 ppid=1464 pid=1631 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts1 ses=6 comm="ping" exe="/bin/ping" key="privileged"type=EXECVE msg=audit(1361852594.621:48): argc=2 a0="ping" a1="google.com"
type=BPRM_FCAPS msg=audit(1361852594.621:48): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=CWD msg=audit(1361852594.621:48):  cwd="/home/ubuntu"
type=PATH msg=audit(1361852594.621:48): item=0 name="/bin/ping" inode=131711 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1361852594.621:48): item=1 name=(null) inode=934 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00

Next steps: Patching and upgrading the kernel with GRSecurity

GRSecurity is an awesome tool in the security-minded system administrators toolbag. It will prevent zero days (like the proc mem exploit explained above 1 ) by securing which areas a user can access. A full list can be seen at http://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options and http://en.wikipedia.org/wiki/Grsecurity#Miscellaneous_features, I suggest going through these and seeing if you want to continue with this.

The following below is for advanced users. Not responsible for any issues you may run into, please make sure to test this in a staging/test environment.

Here are the steps I followed to install the patch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Start by downloading the latest kernel
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.39.tar.bz2

# Next extract it
tar xjvf linux-3.2.39.tar.bz2
cd linux-3.2.39

# Copy over your current kernel configuration:
cp -vi /boot/config-`uname -r` .config

# Updates the config file to match old config and prompts for any new kernel options.
make oldconfig

# This will make sure only modules get compiled only if they are in your kernel. 
make localmodconfig

# Bring up the configuration menu
make menuconfig

Once your in the menu config you can browse to the Security section and go to Grsecurity and enable it. I set the configuration method to automatic and then went to Customize. For example, you can now go to Kernel Auditing -> Exec logging to turn on some additional logging to shell activities (WARNING: this will generate a lot of log activity, decide if you want to use this or not). I suggest going through all of these and reading through their menu help descriptions (when selecting one, press the ? key to bring up the help).

Now we’ll finish making the kernel and compiling it:

1
2
3
4
5
6
7
8
# Now we can compile the kernel
make -j2 # where 2 is the # of CPU's + 1

# Install and load the dynamic kernel modules
sudo make modules_install

# Finally install kernel
sudo make install

We can now reboot and boot into our GRsecurity patched kernel!

Hopefully this article has provided some insight into what the setuid flag does, how it has and can be exploited, and what we can do to prevent this in the future.

Here are a few links to useful books on the subject of shellcode and exploits that I reccomend:

Below is the list of setuid binaries on each OS

Ubuntu 12.04 LTS (22)

back to top

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
-rwsr-xr-x 1 root    root        31304 Mar  2  2012 /bin/fusermount-rwsr-xr-x 1 root    root        94792 Mar 30  2012 /bin/mount
-rwsr-xr-x 1 root    root        35712 Nov  8  2011 /bin/ping
-rwsr-xr-x 1 root    root        40256 Nov  8  2011 /bin/ping6
-rwsr-xr-x 1 root    root        36832 Sep 12 18:29 /bin/su
-rwsr-xr-x 1 root    root        69096 Mar 30  2012 /bin/umount
-rwsr-sr-x 1 daemon  daemon      47928 Oct 25  2011 /usr/bin/at
-rwsr-xr-x 1 root    root        41832 Sep 12 18:29 /usr/bin/chfn
-rwsr-xr-x 1 root    root        37096 Sep 12 18:29 /usr/bin/chsh
-rwsr-xr-x 1 root    root        63848 Sep 12 18:29 /usr/bin/gpasswd
-rwsr-xr-x 1 root    root        62400 Jul 28  2011 /usr/bin/mtr
-rwsr-xr-x 1 root    root        32352 Sep 12 18:29 /usr/bin/newgrp
-rwsr-xr-x 1 root    root        42824 Sep 12 18:29 /usr/bin/passwd
-rwsr-xr-x 2 root    root        71288 May 31  2012 /usr/bin/sudo
-rwsr-xr-x 2 root    root        71288 May 31  2012 /usr/bin/sudoedit
-rwsr-xr-x 1 root    root        18912 Nov  8  2011 /usr/bin/traceroute6.iputils
-rwsr-xr-- 1 root    messagebus 292944 Oct  3 13:03 /usr/lib/dbus-1.0/dbus-daemon-launch-helper
-rwsr-xr-x 1 root    root        10408 Dec 13  2011 /usr/lib/eject/dmcrypt-get-device
-rwsr-xr-x 1 root    root       240984 Apr  2  2012 /usr/lib/openssh/ssh-keysign
-rwsr-xr-x 1 root    root        10592 Oct  5 16:08 /usr/lib/pt_chown
-rwsr-xr-- 1 root    dip        325744 Feb  4  2011 /usr/sbin/pppd
-rwsr-sr-x 1 libuuid libuuid     18856 Mar 30  2012 /usr/sbin/uuidd

CentOS 6.3 (21)

back to top

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
-rwsr-xr-x. 1 root root  76056 Nov  5 05:21 /bin/mount-rwsr-xr-x. 1 root root  40760 Jul 19  2011 /bin/ping
-rwsr-xr-x. 1 root root  36488 Jul 19  2011 /bin/ping6
-rwsr-xr-x. 1 root root  34904 Jun 22  2012 /bin/su
-rwsr-xr-x. 1 root root  50496 Nov  5 05:21 /bin/umount
-rwsr-x---. 1 root dbus  46232 Sep 13 13:04 /lib64/dbus-1/dbus-daemon-launch-helper
-rwsr-xr-x. 1 root root  10272 Apr 16  2012 /sbin/pam_timestamp_check
-rwsr-xr-x. 1 root root  34840 Apr 16  2012 /sbin/unix_chkpwd
-rwsr-xr-x. 1 root root  54240 Jan 30  2012 /usr/bin/at
-rwsr-xr-x. 1 root root  66352 Dec  7  2011 /usr/bin/chage
-rws--x--x. 1 root root  20184 Nov  5 05:21 /usr/bin/chfn
-rws--x--x. 1 root root  20056 Nov  5 05:21 /usr/bin/chsh
-rwsr-xr-x. 1 root root  47520 Jul 19  2011 /usr/bin/crontab
-rwsr-xr-x. 1 root root  71480 Dec  7  2011 /usr/bin/gpasswd
-rwsr-xr-x. 1 root root  36144 Dec  7  2011 /usr/bin/newgrp
-rwsr-xr-x. 1 root root  30768 Feb 22  2012 /usr/bin/passwd
---s--x--x. 2 root root 219272 Aug  6  2012 /usr/bin/sudo
---s--x--x. 2 root root 219272 Aug  6  2012 /usr/bin/sudoedit
-rwsr-xr-x. 1 root root 224912 Nov  9 07:49 /usr/libexec/openssh/ssh-keysign
-rws--x--x. 1 root root  14280 Jan 31 06:30 /usr/libexec/pt_chown
-rwsr-xr-x. 1 root root   9000 Sep 17 05:55 /usr/sbin/usernetctl

OpenBSD 5.2 (3)

back to top

1
2
-r-sr-xr-x  1 root  bin       242808 Aug  1  2012 /sbin/ping-r-sr-xr-x  1 root  bin       263288 Aug  1  2012 /sbin/ping6
-r-sr-x---  1 root  operator  222328 Aug  1  2012 /sbin/shutdown

OpenIndiana 11 (53)

back to top

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
-rwsr-xr-x   1 root     bin        64232 Jun 30  2012 /sbin/wificonfig--wS--lr-x   1 root     root           0 Dec 11 15:20 /media/.hal-mtab-lock
-r-sr-xr-x   1 root     bin       206316 Dec 11 21:00 /usr/lib/ssh/ssh-keysign
-rwsr-xr-x   1 root     adm        12140 Jun 30  2012 /usr/lib/acct/accton
-r-sr-xr-x   1 root     bin        23200 Jun 30  2012 /usr/lib/fs/ufs/quota
-r-sr-xr-x   1 root     bin       111468 Jun 30  2012 /usr/lib/fs/ufs/ufsrestore
-r-sr-xr-x   1 root     bin       106964 Jun 30  2012 /usr/lib/fs/ufs/ufsdump
-r-sr-xr-x   1 root     bin        18032 Jun 30  2012 /usr/lib/fs/smbfs/umount
-r-sr-xr-x   1 root     bin        18956 Jun 30  2012 /usr/lib/fs/smbfs/mount
-r-sr-xr-x   1 root     bin        12896 Jun 30  2012 /usr/lib/utmp_update
-r-sr-xr-x   1 root     bin        35212 Jun 30  2012 /usr/bin/fdformat
-r-s--x--x   2 root     bin       188080 Jun 30  2012 /usr/bin/sudoedit
-r-sr-xr-x   1 root     sys        34876 Jun 30  2012 /usr/bin/su
-r-sr-xr-x   1 root     bin        42504 Jun 30  2012 /usr/bin/login
-r-sr-xr-x   1 root     bin       257288 Jun 30  2012 /usr/bin/pppd
-r-sr-xr-x   1 root     sys        46208 Jun 30  2012 /usr/bin/chkey
-r-sr-xr-x   1 root     sys        29528 Jun 30  2012 /usr/bin/amd64/newtask
-r-sr-xr-x   2 root     bin        24432 Jun 30  2012 /usr/bin/amd64/w
-r-sr-xr-x   1 root     bin      3224200 Jun 30  2012 /usr/bin/amd64/Xorg
-r-sr-xr-x   2 root     bin        24432 Jun 30  2012 /usr/bin/amd64/uptime
-rwsr-xr-x   1 root     sys        47804 Jun 30  2012 /usr/bin/at
-r-sr-xr-x   1 root     bin         8028 Jun 30  2012 /usr/bin/mailq
-r-sr-xr-x   1 root     bin        33496 Jun 30  2012 /usr/bin/rsh
-r-sr-xr-x   1 root     bin        68704 Jun 30  2012 /usr/bin/rmformat
-r-sr-sr-x   1 root     sys        31292 Jun 30  2012 /usr/bin/passwd
-rwsr-xr-x   1 root     sys        23328 Jun 30  2012 /usr/bin/atrm
-r-sr-xr-x   1 root     bin        97072 Jun 30  2012 /usr/bin/xlock
-r-sr-xr-x   1 root     bin        78672 Jun 30  2012 /usr/bin/rdist
-r-sr-xr-x   1 root     bin        27072 Jun 30  2012 /usr/bin/sys-suspend
-r-sr-xr-x   1 root     bin        29304 Jun 30  2012 /usr/bin/crontab
-r-sr-xr-x   1 root     bin        53080 Jun 30  2012 /usr/bin/rcp
-r-s--x--x   2 root     bin       188080 Jun 30  2012 /usr/bin/sudo
-r-s--x--x   1 uucp     bin        70624 Jun 30  2012 /usr/bin/tip
-rwsr-xr-x   1 root     sys        18824 Jun 30  2012 /usr/bin/atq
-r-sr-xr-x   1 root     bin       281732 Jun 30  2012 /usr/bin/xscreensaver
-r-sr-xr-x   1 root     bin      2767780 Jun 30  2012 /usr/bin/i86/Xorg
-r-sr-xr-x   1 root     sys        22716 Jun 30  2012 /usr/bin/i86/newtask
-r-sr-xr-x   2 root     bin        22020 Jun 30  2012 /usr/bin/i86/w
-r-sr-xr-x   2 root     bin        22020 Jun 30  2012 /usr/bin/i86/uptime
-rwsr-xr-x   1 root     sys        13636 Jun 30  2012 /usr/bin/newgrp
-r-sr-xr-x   1 root     bin        39224 Jun 30  2012 /usr/bin/rlogin
-rwsr-xr-x   1 svctag   daemon    108964 Jun 30  2012 /usr/bin/stclient
-r-sr-xr-x   1 root     bin        29324 Jun 30  2012 /usr/xpg4/bin/crontab
-rwsr-xr-x   1 root     sys        47912 Jun 30  2012 /usr/xpg4/bin/at
-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/deallocate
-rwsr-xr-x   1 root     sys        32828 Jun 30  2012 /usr/sbin/sacadm
-r-sr-xr-x   1 root     bin        46512 Jun 30  2012 /usr/sbin/traceroute
-r-sr-xr-x   1 root     bin        18016 Jun 30  2012 /usr/sbin/i86/whodo
-r-sr-xr-x   1 root     bin        55584 Jun 30  2012 /usr/sbin/ping
-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/allocate
-r-sr-xr-x   1 root     bin        37320 Jun 30  2012 /usr/sbin/pmconfig
-r-sr-xr-x   3 root     bin        41276 Jun 30  2012 /usr/sbin/list_devices
-r-sr-xr-x   1 root     bin        24520 Jun 30  2012 /usr/sbin/amd64/whodo