If you’re using bundler for your ruby or rails project and docker you will run into docker having to install your gems everytime. You can either make a base image that has the bundle cache already on it, or you can make a small cache step in your Dockerfile.

Here I’ve setup a cache user and host to store the cache tar. It will attempt to download and untar it, run bundle, then attempt to tar and re-upload it.

1RUN scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email protected]:~/project.tar.gz . || true
2RUN tar xzf project.tar.gz || true
3RUN bundle install --deployment --without development test
4RUN tar czf project.tar.gz vendor
5RUN scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no project.tar.gz [email protected]:~/ || true

Doing this cut build times for my image from a few minutes to a few seconds. If you have any other tricks for speeding up builds, let me know!

Here is an example of a clean way to communicate between React components without getting stuck passing @prop callbacks all around. Inspired by looking at the new Flux React utilities.

We’re going to start off with a simple HAML file:

1%script{src: "https://cdnjs.cloudflare.com/ajax/libs/react/0.12.2/react-with-addons.js"}
2%div{data: { ui: 'alerts' }}
3%div{data: { ui: 'widgets' }}
4:javascript
5  React.renderComponent(Widgets(), document.querySelector('[data-ui="widgets"]'))
6  React.renderComponent(Alerts(), document.querySelector('[data-ui="alerts"]'))

Next comes our Widget component.

1{div, button} = React.DOM
2
3Widgets = React.createClass
4	render: ->
5    div className: 'widget',
6    	button className: 'btn btn-primary', onClick: (=> @_sendMsg('Testing')), 'Click Me'
7	_sendMsg: (msg) ->
8    $('[data-ui="alerts"]').trigger("message", ["Widget clicked."])

On line 1 we’re defining some easy helper methods to access the React.DOM object - otherwise on every line we’d be writing something like React.DOM.div or whichever element we were going to call.

Line 4 is our render method. Everytime state gets mutated or the component is loaded, this method is called.

On line 6 we’re creating an anonymous function but passing in the local scope using a fat arrow => so we can access our other functions in the class. We call it inside an anonymous function so we can pass an argument to it, in this case the message.

Line 7 is our function that fires the event. I’m using the _sendMsg syntax to denote it is a private function. The first argument to the jQuery event emitter is the event name, followed by a list of arguments.

Now lets write our Alert handler and go through it line by line.

 1{div} = React.DOM
 2Alerts = React.createClass
 3  messageTimeout: 5000
 4  getInitialState: ->
 5    messages: []
 6
 7  componentDidMount: ->
 8    $('[data-ui="alerts"]').on 'message', (event, msg) =>
 9      msgs = @state.messages
10      msgs.push(msg)
11      @setState(messages: msgs)
12
13  componentDidUpdate: ->
14    @state.messages.map (msg, index) =>
15      setTimeout(( => @_removeMsg(index)), @messageTimeout)
16
17  render: ->
18    div {},
19      @state.messages.map (msg, index) =>
20        div className: 'alert alert-info',
21          msg
22
23  _removeMsg: (index) ->
24    msgs = @state.messages
25    msgs.splice(index, 1)
26    @setState(messages: msgs)

Line 1 we’re doing the same thing as before, creating a little helper method.

Line 3 is a class variable (we also could have used props here but I went with the class variable instead).

Line 4 is a function that defines the initial state of the component once it is mounted on the page. Here we are saying that there is an empty messages array.

Line 7 is a life cycle event of a React component, called componentDidMount which is called after the component has been rendered into the DOM and mounted. Here we are telling jQuery to bind to any events that are triggered on the [data-ui="alerts"] object and process them. We take the current messages from @state.messages, push the newest message on to the end and then finally call @setState to mutate the components state.

Now the next part on line 13 is how we can gracefully remove messages after they have been rendered. componentDidUpdate is another React life cycle event and is called after a render occurs (and renders occur because the component was updated). We iterate over each message using the map function and call setTimeout with an anonymous function that calls @_removeMsg and passes in an index. @messageTimeout is how we access the class variable defined at the top of the file.

Line 17 is a render call to display all the messages. Note that it is wrapped in a div because you can’t return a collection of objects from render, it must a single root element with nodes underneath.

Line 23 is our message removal function. We set @state.messages to a local variable, remove one element at index and then mutate the state by setting it to our local variable with @setState.

Below is an example of the final product.

I’d like to thank my friend/co-worker Robert Pearce for getting me into React and showing me that everything doesn’t need to be jQuery!

Moved to my new hydroponics website: http://hydronerd.com/

This is your setup:

• Nginx redirecting HTTP to HTTPS
• Unicorn/puma/(etc) being reverse proxied to via nginx

What you may not realize:

When you’re submitting a form and using the redirect_to or respond_with @object, location: [] methods rails may not pick up that you want to use HTTPS as the protocol, adding this to your application_controller.rb ensures every URL generated by rails will go to a secure protocol:

1def default_url_options(options={})
2    options.merge!(protocol: 'https') unless Rails.env.development? || Rails.env.test?
3    options
4end

I ran into issues with rails redirecting to plain HTTP when doing some uploading utilizing iframes (and Chrome rightfully blocking the redirect back to http). Hope this helps someone else! No need to use any special gems.

Note: This only applies if you want all actions to be HTTPS though.

I was working on the agent for SSH Pot and ran into something interesting last night. A lot of the brute force attempts attempt to run a command like this:

1ssh user@host 'uname'

This is different than:

1ssh user@host
2$ uname

The first command is executing a command then exiting, the second is actually logging in and giving the user a shell. The first requests a exec subsystem and the second requests a shell subsystem - so there are two ways to handle it.

 1func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
 2	for req := range in {
 3		ok := true
 4		logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
 5		switch req.Type {
 6		case "shell":
 7			req.Reply(ok, nil)
 8		case "exec":
 9			if string(req.Payload) == string("uname") {
10				channel.Write([]byte("\n\rLinux\n\r"))
11			}
12
13			channel.Close()
14		}
15	}
16}

When logging in my logfile it would show something like:

1[request exec]: uname

And even when comparing the two side by side with something like this:

1logfile.Println("["+string(req.Payload)+"]:["+"uname"+"]")

I would get this output:

1[uname]:[uname]

Yet the comparison on line 9 would not get hit. After sitting and thinking about it for a while I decided to print the bytes out:

1INFO: 2014/07/07 23:15:18 sshd.go:157: [0 0 0 5 117 110 97 109 101]
2INFO: 2014/07/07 23:15:18 sshd.go:158: [117 110 97 109 101]

Aha! So for some reason req.Payload is padded with 3 null bytes and a ENQ byte (hex 5).

Here is the corrected version removing the correct bytes - now the string comparison works:

 1func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
 2	for req := range in {
 3		ok := true
 4		logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
 5		switch req.Type {
 6		case "shell":
 7			req.Reply(ok, nil)
 8		case "exec":
 9			if string(req.Payload[4:]) == string("uname") {
10				channel.Write([]byte("\n\rLinux\n\r"))
11			}
12
13			channel.Close()
14		}
15	}
16}

This is a short example showing how to use an interface to ease testing, and how to use an interface with running shell commands / other programs and providing mock output.

Here is our main file that actually runs the commands and prints out “hello”.

 1package main
 2
 3import (
 4	"fmt"
 5	"os/exec"
 6)
 7
 8// first argument is the command, like cat or echo,
 9// the second is the list of args to pass to it
10type Runner interface {
11	Run(string, ...string) ([]byte, error)
12}
13
14type RealRunner struct{}
15
16var runner Runner
17
18// the real runner for the actual program, actually execs the command
19func (r RealRunner) Run(command string, args ...string) ([]byte, error) {
20	out, err := exec.Command(command, args...).CombinedOutput()
21	return out, err
22}
23
24func Hello() string {
25	out, err := runner.Run("echo", "hello")
26	if err != nil {
27		panic(err)
28	}
29	return string(out)
30}
31
32func main() {
33	runner = RealRunner{}
34	fmt.Println(Hello())
35}

Here is our test file. We start by defining our TestRunner type and implementing the Run(...) interface for it.

This function builds up a command to run the current test file and run the TestHelperProcess function passing along all the args you originally sent. This lets you do things like return different output for different commands you want to run.

The TestHelperProcess function exits when run in the context of the test file, but runs when specified in the files arguments.

 1package main
 2
 3import (
 4	"fmt"
 5	"os"
 6	"os/exec"
 7	"testing"
 8)
 9
10type TestRunner struct{}
11
12func (r TestRunner) Run(command string, args ...string) ([]byte, error) {
13	cs := []string{"-test.run=TestHelperProcess", "--"}
14	cs = append(cs, args...)
15	cmd := exec.Command(os.Args[0], cs...)
16	cmd.Env = []string{"GO_WANT_HELPER_PROCESS=1"}
17	out, err := cmd.CombinedOutput()
18	return out, err
19}
20
21func TestHello(t *testing.T) {
22	runner = TestRunner{}
23	out := Hello()
24	if out == "testing helper process" {
25		t.Logf("out was eq to %s", string(out))
26	}
27}
28
29func TestHelperProcess(*testing.T) {
30	if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
31		return
32	}
33	defer os.Exit(0)
34	fmt.Println("testing helper process")
35}

Hopefully this helps someone else! I had a hard time finding some good, short examples on the internet that combined both interfaces and mocking like this.

Small function that will print out useful information when invoked:

 1func logMsg(msg string) {
 2	pc, _, _, _ := runtime.Caller(1)
 3	caller := runtime.FuncForPC(pc).Name()
 4	_, file, line, _ := runtime.Caller(0)
 5	sp := strings.Split(file, "/")
 6	short_path := sp[len(sp)-2 : len(sp)]
 7	path_line := fmt.Sprintf("[%s/%s:%d]", short_path[0], short_path[1], line)
 8	log_string := fmt.Sprintf("[%s]%s %s:: %s", time.Now(), path_line, caller, msg)
 9	fmt.Println(log_string)
10}

Sample output:

1[2014-06-10 01:38:45.812215998 +0000 UTC][src/trex-client.go:15]{main.main}:: checking jobs - finish
2[2014-06-10 01:38:47.329650331 +0000 UTC][src/trex-client.go:15]{main.main}:: building package list - start

Whenever I have spare time ( often around Christmas or when I’m on vacation/traveling ), I tend to fill it with working on projects I’ve built up in my backlog. I’m also really trying to keep a continuous streak of OSS commits going on Github (something about filling that chart up makes me want to work harder). Here’s my process and how I go about working on personal projects and try to stay motivated - if you have any ideas I’d love to hear them in the comments!

Have a backlog

I use Evernote for all my ideas and project notes:

I have two columns - one for things in progress or to do and one for projects that are done ( with a link to any github repos I published ). When I have some downtime but don’t feel like actually writing any code - I’ll write out plans for what the project needs (use cases, backend needs, software I plan on using, etc) and do research and store all that as a sub-note in Evernote (you can see that with the light green link to the HAProxy Frontend ) under the main page. Plus I can easily share these with friends for feedback by just copying the share URL.

Use small milestones to build up bigger ones

For instance, when I was working on the code for http://ifcfg.net/ I decided there were two major components I would need to create: the web api to access the data, and then a backing library to do some web scraping to gather BGP data. I started out writing a small scraper in Scala for scraping BGP and looking glass info (which involved learning some more SBT, and selenium apis for Scala) and then moved onto learning a small amount of the Play! framework and exposing my library via that api. This let me focus on one small component at a time and finish it ( I have a habit of leaving personal projects unfinished or taking a long time to finish them if I let the scope creep beyond what I deemed as minimum requirements ).

Pick an interesting project

There are some areas I just don’t have an interest in - like writing an application to track golf scores.

So pick something you like - I love doing backend systems and APIs - pick something your passionate about already or a topic you want to learn more about.

Learn

If I’m working on a personal project and not learning anything new (even if its just a new way to test, for instance) - I get bored, really quickly. I’ve been stemming this by trying to pick up new languages as I work on projects and working on projects with broader goals.

For instance, my latest project I’m working on is Patchasaurus ( yes there isn’t a readme yet ). I know theres a gap in the systems world for (open source) patch management, especially focused on Ubuntu and Debian - so I decided to write a small initial version of one. I had been playing around with Go at work (and boy is it nice to get a HTTP API running in a few MB of RAM) and decided to write the agent for patchasaurus in that ( nicknamed trex ). I’ve been learning how to cross compile programs in Go, what libraries don’t work with cross-compilation (looking at you os/user) and a nice work flow for testing these while developing them ( sshfs is great for this with VirtualBox or Vagrant ). I also chose to use Rails 4.1 as the management interface since I wanted to stay up to date with the new Rails features - turns out spring is very nice and a great improvement over the guard work flow I’ve used before.

Don’t focus on processes versus getting things done

I’m a big fan of testing, and TDD, however I’m not always in the mood to do it. Sometimes I just want to see results and I’ll go back and refactor and test later. Picking what works for you on a specific project/component, and getting it done I think is much more important than rigidly following a specific set of guidelines on every project you do ( aka: test first, setup CI before any code, etc ).

Don’t get in a rut

Staring at HackerNews or Reddit all day can be daunting - try and not focus on what everyone else is doing and instead focus on what you’re getting done and how you’re improving yourself.

Also don’t let this influence your technology choices. Sometimes there are articles trending for AngularJS or Ruby on Rails - stick with what you picked ( unless you really want to learn that new tech ) - or figure out ways to incorporate that into smaller components of your project. Don’t throw away all that progress just because you saw a few posts reach the page!

Take breaks

Don’t spend all day coding - take breaks, go for a walk, a run, play with your dog, play a video game - something that can give you a moment to breathe and think about something else or give you time to re-focus on the grand vision you’ve been laboring over. Figure out what works for you to relax and do it to break up that screen glow tan you’re getting.

Talk about what you’re working on

Talk with friends to brainstorm ideas, pair up on some problems, see if theres a more idiomatic way to do a function in the language your using ( for example, I spent some time trying to see if there were any map() equivalents on #go-nuts), and blog about what you’re doing if that’s your style.

Knowing people are using code and software I’ve written is a huge motivating factor to working on future projects ( star/watch counts on Github, downloads on RubyGems, traffic to my blog, etc).

Finish!!

Yes it can be hard, but figure out what finished means to you, and do it. Publish it on Github, submit it to HackerNews, post it to reddit, get it hooked into TravisCI - make sure you come to the finish line of each component or project you’re working on. Building up these small accomplishments can help set a streak for the future so you have the motivation to power through and get items done.

Sometimes you’re more interested in getting an application finished than on the deployment process - throw it on Heroku, a shared hosting provider, etc. There’s nothing wrong with some shared hosting for a small project. Don’t let things like deployment stop you from finishing!

When working on a rails application you can sometimes find duplicated or very similar code between two different controllers (for instance a UI element and an API endpoint). Realizing that you have this duplication there are several things you can do. I’m going to go over how to extract this code out into the query object pattern 1 and clean up our constructor using the builder pattern 2 adapted to ruby.

I’m going to make a few assumptions here, but this should be applicable to any data access layer of your application. I’m also assuming you’re using something like Kaminari for pagination and have a model for People.

 1
 2def index
 3  page = params[:page] || 1
 4  per_page = params[:per_page] || 50
 5  name = params[:name]
 6  sort = params[:sort_by] || 'last_name'
 7  direction = params[:sort_direction] || 'asc'
 8
 9  query = People
10  query = query.where(name: name) if name.present?
11  @results = query.order("#{sort} #{direction}").page(page).per_page(per_page)
12end

So we see this duplicated elsehwere in the code base and we want to clean it up. Lets first start by extracting this out into a new class called PeopleQuery.

I usually put these under app/queries in my rails application.

 1
 2class PeopleQuery
 3  attr_accessor :page, :per_page, :name, :sort, :direction, :query
 4  def initialize(page, per_page, name, sort, direction)
 5    self.page = page || 1
 6    self.per_page = per_page || 50
 7    self.name = name
 8    self.sort = sort || 'last_name'
 9    self.direction = direction || 'asc'
10    self.query = People
11  end
12
13  def build
14    self.query = self.query.where(name: self.name) if self.name.present?
15    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
16  end
17end

Now our controller looks like this:

1
2def index
3  query = PeopleQuery.new(params[:page], params[:per_page], params[:name], params[:sort], params[:direction])
4  @results = query.build
5end

Much better! We’ve decoupled our control from our data access object (People/ActiveRecord), moved some of the query logic outside of the controller and into a specific class meant to deal with building it. But that constructor doesn’t look very nice. We can do better since we’re using ruby.

Our new PeopleQuery class will look like this and will use a block to initialize itself instead of a long list of constructor arguments.

 1class PeopleQuery
 2  attr_accessor :page, :per_page, :name, :sort, :direction, :query
 3  def initialize(&block)
 4    yield self
 5    self.page ||= 1
 6    self.per_page =|| 50
 7    self.sort ||= 'last_name'
 8    self.direction ||= 'asc'
 9    self.query = People
10  end
11
12  def build
13    self.query = self.query.where(name: self.name) if self.name.present?
14    self.query.order("#{self.sort} #{self.direction}").page(self.page).per_page(self.per_page)
15  end
16end

We yield first to let the caller set the values and then after yielding we set our default values if they weren’t passed in. There is another method of doing this with instance_eval but you end up losing variable scope and the constructor looks worse since you have to start passing around the params variable to get access to it, so we’re going to stick with yield.

 1
 2def index
 3  query = PeopleQuery.new do |query|
 4    query.page = params[:page]
 5    query.per_page = params[:per_page]
 6    query.name = params[:name]
 7    query.sort = params[:sort]
 8    query.direction = params[:direction]
 9  end
10  @results = query.build
11end

And that’s it! We’ve de-duplicated some code (remember we assumed dummy controller’s index method was duplicated elsewhere in an API call in a seperate namespaced controller), extracted out a common query object, decoupled our controller from ActiveRecord, and built up a nice way to construct the query object using the builder pattern.

Is there ever a confusing amount of information out there on parsing HTML in Scala. Here is the list of possible ways I ran across:

• Hope the document is valid XHTML and use scala.xml.XML to parse it
• If the document isn’t valid XHTML use something like TagSoup and hope it parses again
• Still think its valid XHTML? Try using scalaz’s XML parser

All of the answers I found on Google pointed to some type of XML parsing, which won’t always work. Coming from Ruby I know there are tools out there like Selenium that can simulate a web browser for you and give you a rich interface to interact with the returned HTML.

So I went on Maven and found the two Selenium web drivers I wanted for my project and added them to my libraryDependencies:

1    "org.seleniumhq.webdriver" % "webdriver-selenium" % "0.9.7376",
2    "org.seleniumhq.webdriver" % "webdriver-htmlunit" % "0.9.7376"

The project I’m working on is to parse Looking Glass websites for BGP information and AS peering, so I wanted to scrape the data. I also didn’t want to have to use a full blown web browser (ala Selenium + Firefox for instance) - so I stuck with the HtmlUnit driver for the implementation.

Here is a quick code snippet that lets me grab AS #’s and Peer names from an AS:

 1val url = "http://example.com/AS" + as.toString
 2
 3val driver = new HtmlUnitDriver
 4// Proxy for BetaMax when writing tests
 5if (_port != null) {
 6  driver.setProxy("localhost", _port)
 7}
 8driver.get(url)
 9
10val peers = driver.findElementsByXPath("//*[@id=\"table_peers4\"]/tbody/tr/td[position() = 1 or position() = 2]")
11
12// zip up the list in pairs so List(a,b,c,d) becomes List((a,b), (c,d))
13for(peer <- peers zip peers.tail) {
14  println(peer)
15}

No XML to muck with and I get some nice selectors to query the document for. Remember if the source you want data from doesn’t have an API, HTML is an API! Just be respectful of how you query and interact with them (ie: Don’t do 100 requests/second, cache/record responses while writing tests, etc).