{Josh Rendek}

<3 Ruby & Go

Rails Force HTTPS for Your Application

This is your setup:

  • Nginx redirecting HTTP to HTTPS
  • Unicorn/puma/(etc) being reverse proxied to via nginx

What you may not realize:

When you’re submitting a form and using the redirect_to or respond_with @object, location: [] methods rails may not pick up that you want to use HTTPS as the protocol, adding this to your application_controller.rb ensures every URL generated by rails will go to a secure protocol:

application_controller.rb
1
2
3
4
def default_url_options(options={})
    options.merge!(protocol: 'https') unless Rails.env.development? || Rails.env.test?
    options
end

I ran into issues with rails redirecting to plain HTTP when doing some uploading utilizing iframes (and Chrome rightfully blocking the redirect back to http). Hope this helps someone else! No need to use any special gems.

Note: This only applies if you want all actions to be HTTPS though.

Go-lang Compare *ssh.Request.Type Against a String

I was working on the agent for SSH Pot and ran into something interesting last night. A lot of the brute force attempts attempt to run a command like this:

Example Exec
1
ssh user@host 'uname'

This is different than:

Example Shell
1
2
ssh user@host
$ uname

The first command is executing a command then exiting, the second is actually logging in and giving the user a shell. The first requests a exec subsystem and the second requests a shell subsystem - so there are two ways to handle it.

broken_handler.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
  for req := range in {
      ok := true
      logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
      switch req.Type {
      case "shell":
          req.Reply(ok, nil)
      case "exec":
          if string(req.Payload) == string("uname") {
              channel.Write([]byte("\n\rLinux\n\r"))
          }

          channel.Close()
      }
  }
}

When logging in my logfile it would show something like:

Log
1
[request exec]: uname

And even when comparing the two side by side with something like this:

log.go
1
logfile.Println("["+string(req.Payload)+"]:["+"uname"+"]")

I would get this output:

Log
1
[uname]:[uname]

Yet the comparison on line 9 would not get hit. After sitting and thinking about it for a while I decided to print the bytes out:

Log
1
2
INFO: 2014/07/07 23:15:18 sshd.go:157: [0 0 0 5 117 110 97 109 101]
INFO: 2014/07/07 23:15:18 sshd.go:158: [117 110 97 109 101]

Aha! So for some reason req.Payload is padded with 3 null bytes and a ENQ byte (hex 5).

Here is the corrected version removing the correct bytes - now the string comparison works:

working_handler.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
  for req := range in {
      ok := true
      logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
      switch req.Type {
      case "shell":
          req.Reply(ok, nil)
      case "exec":
          if string(req.Payload[4:]) == string("uname") {
              channel.Write([]byte("\n\rLinux\n\r"))
          }

          channel.Close()
      }
  }
}

Go-lang: Mocking exec.Command Using Interfaces

This is a short example showing how to use an interface to ease testing, and how to use an interface with running shell commands / other programs and providing mock output.

Source on Github

Here is our main file that actually runs the commands and prints out “hello”.

example.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main

import (
  "fmt"
  "os/exec"
)

// first argument is the command, like cat or echo,
// the second is the list of args to pass to it
type Runner interface {
  Run(string, ...string) ([]byte, error)
}

type RealRunner struct{}

var runner Runner

// the real runner for the actual program, actually execs the command
func (r RealRunner) Run(command string, args ...string) ([]byte, error) {
  out, err := exec.Command(command, args...).CombinedOutput()
  return out, err
}

func Hello() string {
  out, err := runner.Run("echo", "hello")
  if err != nil {
      panic(err)
  }
  return string(out)
}

func main() {
  runner = RealRunner{}
  fmt.Println(Hello())
}

Here is our test file. We start by defining our TestRunner type and implementing the Run(...) interface for it.

This function builds up a command to run the current test file and run the TestHelperProcess function passing along all the args you originally sent. This lets you do things like return different output for different commands you want to run.

The TestHelperProcess function exits when run in the context of the test file, but runs when specified in the files arguments.

example_test.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main

import (
  "fmt"
  "os"
  "os/exec"
  "testing"
)

type TestRunner struct{}

func (r TestRunner) Run(command string, args ...string) ([]byte, error) {
  cs := []string{"-test.run=TestHelperProcess", "--"}
  cs = append(cs, args...)
  cmd := exec.Command(os.Args[0], cs...)
  cmd.Env = []string{"GO_WANT_HELPER_PROCESS=1"}
  out, err := cmd.CombinedOutput()
  return out, err
}

func TestHello(t *testing.T) {
  runner = TestRunner{}
  out := Hello()
  if out == "testing helper process" {
      t.Logf("out was eq to %s", string(out))
  }
}

func TestHelperProcess(*testing.T) {
  if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
      return
  }
  defer os.Exit(0)
  fmt.Println("testing helper process")
}

Hopefully this helps someone else! I had a hard time finding some good, short examples on the internet that combined both interfaces and mocking like this.

More examples from os/exec/exec_test.go

A Useful Logger in Go

Small function that will print out useful information when invoked:

log.go
1
2
3
4
5
6
7
8
9
10
func logMsg(msg string) {
  pc, _, _, _ := runtime.Caller(1)
  caller := runtime.FuncForPC(pc).Name()
  _, file, line, _ := runtime.Caller(0)
  sp := strings.Split(file, "/")
  short_path := sp[len(sp)-2 : len(sp)]
  path_line := fmt.Sprintf("[%s/%s:%d]", short_path[0], short_path[1], line)
  log_string := fmt.Sprintf("[%s]%s %s:: %s", time.Now(), path_line, caller, msg)
  fmt.Println(log_string)
}

Sample output:

1
2
[2014-06-10 01:38:45.812215998 +0000 UTC][src/trex-client.go:15]{main.main}:: checking jobs - finish
[2014-06-10 01:38:47.329650331 +0000 UTC][src/trex-client.go:15]{main.main}:: building package list - start

Motivation to Work on New Projects

Whenever I have spare time ( often around Christmas or when I’m on vacation/traveling ), I tend to fill it with working on projects I’ve built up in my backlog. I’m also really trying to keep a continuous streak of OSS commits going on Github (something about filling that chart up makes me want to work harder). Here’s my process and how I go about working on personal projects and try to stay motivated - if you have any ideas I’d love to hear them in the comments!

Have a backlog

I use Evernote for all my ideas and project notes:

Evernote

I have two columns - one for things in progress or to do and one for projects that are done ( with a link to any github repos I published ). When I have some downtime but don’t feel like actually writing any code - I’ll write out plans for what the project needs (use cases, backend needs, software I plan on using, etc) and do research and store all that as a sub-note in Evernote (you can see that with the light green link to the HAProxy Frontend ) under the main page. Plus I can easily share these with friends for feedback by just copying the share URL.

Use small milestones to build up bigger ones

For instance, when I was working on the code for http://ifcfg.net/ I decided there were two major components I would need to create: the web api to access the data, and then a backing library to do some web scraping to gather BGP data. I started out writing a small scraper in Scala for scraping BGP and looking glass info (which involved learning some more SBT, and selenium apis for Scala) and then moved onto learning a small amount of the Play! framework and exposing my library via that api. This let me focus on one small component at a time and finish it ( I have a habit of leaving personal projects unfinished or taking a long time to finish them if I let the scope creep beyond what I deemed as minimum requirements ).

Pick an interesting project

There are some areas I just don’t have an interest in - like writing an application to track golf scores.

So pick something you like - I love doing backend systems and APIs - pick something your passionate about already or a topic you want to learn more about.

Learn

If I’m working on a personal project and not learning anything new (even if its just a new way to test, for instance) - I get bored, really quickly. I’ve been stemming this by trying to pick up new languages as I work on projects and working on projects with broader goals.

For instance, my latest project I’m working on is Patchasaurus ( yes there isn’t a readme yet ). I know theres a gap in the systems world for (open source) patch management, especially focused on Ubuntu and Debian - so I decided to write a small initial version of one. I had been playing around with Go at work (and boy is it nice to get a HTTP API running in a few MB of RAM) and decided to write the agent for patchasaurus in that ( nicknamed trex ). I’ve been learning how to cross compile programs in Go, what libraries don’t work with cross-compilation (looking at you os/user) and a nice work flow for testing these while developing them ( sshfs is great for this with VirtualBox or Vagrant ). I also chose to use Rails 4.1 as the management interface since I wanted to stay up to date with the new Rails features - turns out spring is very nice and a great improvement over the guard work flow I’ve used before.

Don’t focus on processes versus getting things done

I’m a big fan of testing, and TDD, however I’m not always in the mood to do it. Sometimes I just want to see results and I’ll go back and refactor and test later. Picking what works for you on a specific project/component, and getting it done I think is much more important than rigidly following a specific set of guidelines on every project you do ( aka: test first, setup CI before any code, etc ).

Don’t get in a rut

Staring at HackerNews or Reddit all day can be daunting - try and not focus on what everyone else is doing and instead focus on what you’re getting done and how you’re improving yourself.

Also don’t let this influence your technology choices. Sometimes there are articles trending for AngularJS or Ruby on Rails - stick with what you picked ( unless you really want to learn that new tech ) - or figure out ways to incorporate that into smaller components of your project. Don’t throw away all that progress just because you saw a few posts reach the page!

Take breaks

Don’t spend all day coding - take breaks, go for a walk, a run, play with your dog, play a video game - something that can give you a moment to breathe and think about something else or give you time to re-focus on the grand vision you’ve been laboring over. Figure out what works for you to relax and do it to break up that screen glow tan you’re getting.

Talk about what you’re working on

Talk with friends to brainstorm ideas, pair up on some problems, see if theres a more idiomatic way to do a function in the language your using ( for example, I spent some time trying to see if there were any map() equivalents on #go-nuts), and blog about what you’re doing if that’s your style.

Knowing people are using code and software I’ve written is a huge motivating factor to working on future projects ( star/watch counts on Github, downloads on RubyGems, traffic to my blog, etc).

Finish!!

Yes it can be hard, but figure out what finished means to you, and do it. Publish it on Github, submit it to HackerNews, post it to reddit, get it hooked into TravisCI - make sure you come to the finish line of each component or project you’re working on. Building up these small accomplishments can help set a streak for the future so you have the motivation to power through and get items done.

Sometimes you’re more interested in getting an application finished than on the deployment process - throw it on Heroku, a shared hosting provider, etc. There’s nothing wrong with some shared hosting for a small project. Don’t let things like deployment stop you from finishing!

2 Patterns for Refactoring With Your Ruby Application

When working on a rails application you can sometimes find duplicated or very similar code between two different controllers (for instance a UI element and an API endpoint). Realizing that you have this duplication there are several things you can do. I’m going to go over how to extract this code out into the query object pattern 1 and clean up our constructor using the builder pattern 2 adapted to ruby.

I’m going to make a few assumptions here, but this should be applicable to any data access layer of your application. I’m also assuming you’re using something like Kaminari for pagination and have a model for People.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
11
def index
  page = params[:page] || 1
  per_page = params[:per_page] || 50
  name = params[:name]
  sort = params[:sort_by] || 'last_name'
  direction = params[:sort_direction] || 'asc'

  query = People
  query = query.where(name: name) if name.present?
  @results = query.order("#{sort} #{direction}").page(page).per_page(per_page)
end

So we see this duplicated elsehwere in the code base and we want to clean it up. Lets first start by extracting this out into a new class called PeopleQuery.

I usually put these under app/queries in my rails application.

people_query.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class PeopleQuery
  attr_accessor :page, :per_page, :name, :sort, :direction, :query
  def initialize(page, per_page, name, sort, direction)
    self.page = page || 1
    self.per_page = per_page || 50
    self.name = name
    self.sort = sort || 'last_name'
    self.direction = direction || 'asc'
    self.query = People
  end

  def build
    self.query = self.query.where(name: self.name) if self.name.present?
    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
  end
end

Now our controller looks like this:

dummy_controller.rb
1
2
3
4
def index
  query = PeopleQuery.new(params[:page], params[:per_page], params[:name], params[:sort], params[:direction])
  @results = query.build
end

Much better! We’ve decoupled our control from our data access object (People/ActiveRecord), moved some of the query logic outside of the controller and into a specific class meant to deal with building it. But that constructor doesn’t look very nice. We can do better since we’re using ruby.

Our new PeopleQuery class will look like this and will use a block to initialize itself instead of a long list of constructor arguments.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class PeopleQuery
  attr_accessor :page, :per_page, :name, :sort, :direction, :query
  def initialize(&block)
    yield self
    self.page ||= 1
    self.per_page =|| 50
    self.sort ||= 'last_name'
    self.direction ||= 'asc'
    self.query = People
  end

  def build
    self.query = self.query.where(name: self.name) if self.name.present?
    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
  end
end

We yield first to let the caller set the values and then after yielding we set our default values if they weren’t passed in. There is another method of doing this with instance_eval but you end up losing variable scope and the constructor looks worse since you have to start passing around the params variable to get access to it, so we’re going to stick with yield.

dummy_controller.rb
1
2
3
4
5
6
7
8
9
10
def index
  query = PeopleQuery.new do |query|
    query.page = params[:page]
    query.per_page = params[:per_page]
    query.name = params[:name]
    query.sort = params[:sort]
    query.direction = params[:direction]
  end
  @results = query.build
end

And that’s it! We’ve de-duplicated some code (remember we assumed dummy controller’s index method was duplicated elsewhere in an API call in a seperate namespaced controller), extracted out a common query object, decoupled our controller from ActiveRecord, and built up a nice way to construct the query object using the builder pattern.

Parsing HTML in Scala

Is there ever a confusing amount of information out there on parsing HTML in Scala. Here is the list of possible ways I ran across:

  • Hope the document is valid XHTML and use scala.xml.XML to parse it
  • If the document isn’t valid XHTML use something like TagSoup and hope it parses again
  • Still think its valid XHTML? Try using scalaz’s XML parser

All of the answers I found on Google pointed to some type of XML parsing, which won’t always work. Coming from Ruby I know there are tools out there like Selenium that can simulate a web browser for you and give you a rich interface to interact with the returned HTML.

So I went on Maven and found the two Selenium web drivers I wanted for my project and added them to my libraryDependencies:

1
2
"org.seleniumhq.webdriver" % "webdriver-selenium" % "0.9.7376",
"org.seleniumhq.webdriver" % "webdriver-htmlunit" % "0.9.7376"

The project I’m working on is to parse Looking Glass websites for BGP information and AS peering, so I wanted to scrape the data. I also didn’t want to have to use a full blown web browser (ala Selenium + Firefox for instance) - so I stuck with the HtmlUnit driver for the implementation.

Here is a quick code snippet that lets me grab AS #’s and Peer names from an AS:

AS.scala
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
val url = "http://example.com/AS" + as.toString

val driver = new HtmlUnitDriver
// Proxy for BetaMax when writing tests
if (_port != null) {
  driver.setProxy("localhost", _port)
}
driver.get(url)

val peers = driver.findElementsByXPath("//*[@id=\"table_peers4\"]/tbody/tr/td[position() = 1 or position() = 2]")

// zip up the list in pairs so List(a,b,c,d) becomes List((a,b), (c,d))
for(peer <- peers zip peers.tail) {
  println(peer)
}

No XML to muck with and I get some nice selectors to query the document for. Remember if the source you want data from doesn’t have an API, HTML is an API! Just be respectful of how you query and interact with them (ie: Don’t do 100 requests/second, cache/record responses while writing tests, etc).

Getting Started With Scala

Recently I’ve been getting into more Java and (attempting to) Scala development. I always got annoyed with the Scala ecosystem for development and would get fed up and just go back to writing straight Java (coughsbtcough). Today I decided to write down everything I did and get a sane process going for Scala development with SBT.

I decided to write a small Scala client for OpenWeatherMap - here is what I went through.

A brief guide on naming conventions is here. I found this useful just to reference conventions since not everything is the same as Ruby (camelCase vs snake_case for instance).

Setting up and starting a project

First make sure you hava a JVM installed, Scala, and SBT. I’ll be using Scala 2.10.2 and SBT 0.12.1 since that is what I have installed.

One of the nice things I like about Ruby on Rails is the project generation ( aka: rails new project [opts] ) so I was looking for something similar with Scala.

Enter giter8: https://github.com/n8han/giter8

giter8 runs through SBT and has templates available for quickstart.

Follow the install instructions and install giter8 into SBT globally and load SBT to make sure it downloads and installs.

Once you do that you can pick a template from the list, or go with the one I chose: fayimora/basic-scala-project which sets up the directories properly and also sets up ScalaTest, a testing framework with a DSL similar to RSpec.

To setup your project you need to run:

1
g8 fayimora/basic-scala-project

You’ll be prompted with several questions and then your project will be made. Switch into that directory and run sbt test to make sure the simple HelloWorld passes and everything with SBT is working.

Setting up IntelliJ

For Java and Scala projects I stick with IntelliJ over my usual vim. When using Java IntelliJ is good about picking up library and class path’s and resolving dependencies (especially if you are using Maven). However there isn’t a good SBT plugin (as of writing this) that manages to do all this inside IntelliJ.

The best plugin for SBT I’ve found that does this is sbt-idea. You’re going to need to make a project/plugins.sbt file:

plugins.sbt
1
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")

and now you can generate your .idea files by running: sbt gen-idea

IntelliJ should now resolve your project dependencies and you can start coding your project.

Final Result

scala-weather - A simple to use OpenWeatherMap client in Scala set up with Travis-CI and CodeClimate. This is just the first of several projects I plan on working on / open sourcing to get my feet wet with Scala more.

Useful libraries

Notes

By default Bee Client will log everything to STDOUT - you’ll need to configure logback with an XML file located in src/main/resources/logback.xml:

src/main/resources/logback/xml
1
2
3
4
5
6
7
8
9
10
11
12
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="ERROR">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

From 0 to Testing on Windows With JRuby

Testing is one of the most important parts of software development and helps to ensure bugs don’t get into production and that code can be refactored safely. If you’re working on a team with multiple people with different skill sets, you might have people doing testing who only know windows and development is only using OSX or Linux. We want everyone to be able to test - someone in QA who is familiar with Windows shouldn’t have to throw away all that knowledge, install Linux, and start from scratch. Enter JRuby and John.

John is our tester and he is running windows. He wants to help make sure that when a user goes to http://google.com/ that a button appears with the text “Google Search”. The quick way to do this is to open his browser, navigate to http://google.com/ glance through the page for the button and confirm that its there. John has a problem though, he has 30 other test cases to run and the developers are pushing code to the frontpage several times a day; John now has to continously do this manually everytime code is touched and his test load is piling up.

So let’s help John out and install Sublime Text 2 and JRuby.

Start by downloading the 64-bit version of Sublime Text. Make sure to add the context menu when going through the install process.

Now we’ll visit the JRuby homepage and download the 64 bit installer.

Go through the installer and let JRuby set your path so you can access ruby from cmd.exe

Now when we open cmd.exe and type jruby -v we’ll be able to see that it was installed.

Now that we have our tools installed lets setup our test directory on the Desktop. Inside our testing folder we’ll create a folder called TestDemo for our tests for the Demo project.

Next we’ll open Sublime Text and go to File > Open Folder and navigate to our TestDemo folder and hit open.

Now we can continue making our directory structure inside Sublime Text. Since we’re going to use rspec we need to create a folder called spec to contain all of our tests. Right click on the TestDemo in the tree navigation and click New Folder.

Call the folder spec in the bottom title bar when it prompts you for the folder name.

Next we’ll create our Gemfile which will declare all of our dependencies - so make a file in the project root called Gemfile and put the our dependencies in it:

Gemfile
1
2
3
4
5
6
source "https://rubygems.org"

gem "rspec"
gem "selenium"
gem "selenium-webdriver"
gem "capybara"

Once we have that file created, open cmd.exe and switch to your project’s root directory.

Type jgem install bundler to install bundler which manages ruby dependencies.

While still at the command prompt we’re going to bundle to install our dependencies:

After that finishes we need to run one last command for selenium to work properly: selenium install

We also need a spec_helper.rb file inside our spec directory.

specs\spec_helper.rb
1
2
3
4
5
require "rspec"
require "selenium"
require "capybara/rspec"

Capybara.default_driver =  :selenium

We’ve now setup our rspec folders, our Gemfile with dependencies, and installed them. Now we can write the test that will save John a ton of time.

Chrome comes with a simple tool to get XPath paths so we’re going to use that to get the XPath for the search button. Right click on the “Google Search” button and click Inspect element

Right click on the highlighted element and hit Copy XPath.

Now we’re going to make our spec file and call it homepage_spec.rb and locate it under spec\integration.

Here is a picture showing the directory structure and files:

Here is the spec file with comments explaining each part:

spec\integration\homepage_spec.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# This loads the spec helper file that we required everything in
require "spec_helper"

# This is the outer level description of the test
# For this example it describes going to the homepage of Google.com
# Setting the feature type is necessary if you have
# Capybara specs outside of the spec\features folder
describe "Going to google.com", :type => :feature do

  # Context is like testing a specific component of the homepage, in this case
  # its the search button
  context "The search button" do
    # This is our actual test where we give it a meaningful test description
    it "should contain the text 'Google Search'" do
      visit "http://google.com/" # Opens Firefox and visits google
      button = find(:xpath, '//*[@id=gbqfba"') # find an object on the page by its XPath path
      # This uses an rspec assertion saying that the string returned
      # by button.text is equal to "Google Search"
      button.text.should eq("Google Seearch")

    end
  end

end

Now we can tab back to our cmd.exe prompt and run our tests! rspec spec will run all your tests under the spec folder.

Things to take note of

This example scenario is showing how to automate browser testing to do end-to-end tests on a product using rspec. This is by no means everything you can do with rspec and ruby - you can SSH, hit APIs and parse JSON, and do anything you want with the ability to make assertions.

A lot is going on in these examples - there are plenty of resources out there on google and other websites that provide more rspec examples and ruby examples.

We also showed how to add dependencies and install them using bundler. Two of the best resources for finding libraries and other gems is RubyGems and Ruby-Toolbox - the only thing to take note of is anything saying to be a native C extension (they won’t work with JRuby out of the box).

My last note is that you also need to have firefox installed as well - Selenium will work with Chrome but I’ve found it to be a hassle to setup (and unless you really need Chrome), the default of Firefox will work great.