{ Josh Rendek }

<3 Ruby & Go

I was setting up the ELK stack and had quite a fun time trying to get upstart to log to syslog WITH a log tag ( aka: my-application ) so it could be filtered inside Kibana.

Here is a working example for STDOUT and STDERR:

 1respawn
 2respawn limit 15 5
 3
 4start on runlevel [2345]
 5stop on runlevel [06]
 6
 7setuid app-user
 8setgid app-user
 9
10script
11  # Redirect stdout to syslog
12  mkfifo /tmp/app-stdout-fifo
13  ( logger -p user.info -t your-app-tag </tmp/app-stdout-fifo & )
14  exec 1>/tmp/app-stdout-fifo
15  rm /tmp/app-stdout-fifo
16
17  # Redirect stderr to syslog
18  mkfifo /tmp/app-stderr-fifo
19  ( logger -p user.err  -t your-app-tag </tmp/app-stderr-fifo & )
20  exec 2>/tmp/app-stderr-fifo
21  rm /tmp/app-stderr-fifo
22
23  exec ./your-app-binary
24end script

Hope this helps someone else, there as a lot of mis-leading and broken examples on Google & StackOverflow.

Golang Performance Tips

Sep 20, 2015 - 5 minutes

Below is some advice and notes that I wish I had when writing Go to deal with high amounts of requests (20k+/second). Have any extra tips? Leave them in the comments!

Kernel Tuning

Step 1 is making sure your host OS isn’t going to keel over when you start making thousands of requests/second or hammering the CPU.

Update /etc/sysctl.conf to have these lines:

1net.ipv4.tcp_tw_reuse = 1
2net.ipv4.tcp_tw_recycle = 1
3net.ipv4.ip_local_port_range = 50000

ip_local_port_range - at the default of 30,000 and not modifying the tw_reuse and tw_recycle properties, we’re effectively limited to 500 connections/second to a server. If this is still not enough you can configure additional IP’s on the server and cycle between them.

tcp_tw_reuse will re-use an existing connection that is in TIME-WAIT for outgoing connections.

tcp_tw_recycle enables sockets to be recycled faster once they reach the TIME-WAIT state for both incoming and outgoing connections. Make sure you’re not running anything through a NAT or this can cause problems with connections.

Vinent Bernat has a great explanation with state diagrams on his blog.

Next up are file descriptors. I prefer defining these in the init or upstart scripts, so you would call ulimit -n 102400 and then call your go binary in the upstart script that way it is set before running. (Note: this will only work if the user has been properly given permissions to up their limit in /etc/security/limits.d.

Upstart also provides a mechanism to set file limits in the job stanza.

Golang Tuning

Utilizing all CPUs ( < Go 1.5 )

You can use all the go-routines in the world and not use all your CPU cores and threads. In order to let your go program utilize all operating-system level threads, we need to tell the go runtime about them:

1runtime.GOMAXPROCS(runtime.NumCPU())

This is no longer necessary as of Go 1.5 and is done automatically.

Finish what you start

Make sure you call .Close() on your responses, and make sure you read the entire body. The documentation for net/http/response explicitly says that “it is the caller’s responsibility to close Body” and that “neither ReadResponse nor Response.Write ever closes a connection.” net/http/response.go

Don’t be intimidated

You want to do things fast! But your confused by all the options for concurrency in go. Channels? Goroutines? Libraries to manage them? Stick with a simple worker pattern for best results. I’ve found many libraries that claim to manage concurrency for you (limiting running routines, or providing some interface to queueing jobs) fall short, break, or not utilize all CPU cores.

Here is a simple worker pattern that uses nothing but the standard library:

 1tasks := make(chan someDataStruct, 40)
 2var wg sync.WaitGroup
 3
 4for i := 0; i < 40; i++ {
 5	wg.Add(1)
 6	go func() {
 7		for data := range tasks {
 8			// do some work on data
 9		}
10		wg.Done()
11	}()
12}
13
14// Push to it like this:
15tasks <- someData
16
17// Finish like this
18close(tasks)
19wg.Wait()

First, we make a channel containing someDataStruct as the type to be sent/received over it. We give it a buffer size of 40. Since we only have 40 routines spinning up, no more than 40 can be worked on at once.

When a caller is trying to push data to this channel and all slots are full, it will block until a slot is free, so keep this in mind and change accordingly if you need to.

Next we make a WaitGroup which will wait for all of our goroutines to finish. When we loop 40 times and say wg.Add(1) we’re telling the WaitGroup that we’re expecting 40 goroutines, and to wait for them to finish.

Next we iterate over data coming in our tasks channel and do some process on it (this is obviously where your program specific logic or function calls go).

When no more data is available on the channel we call wg.Done() which tells the WaitGroup a routine has finished.

Pushing data is simple by passing an instance of someDataStruct into the tasks channel.

Almost done! We now want to wait for everything to finish before our program exits. close(tasks) marks the channel as closed - and any other callers who try and send to it will get a nice fat error message.

Finally wg.Wait() says to wait until all 40 wg.Done()’s have been called.

Errors

One of my favorite things about go is that its fast, real fast. Make sure you test, test, and test some more! Always make sure you fail gracefully (if a HTTP connection failed and you need to re-process a job, for instance) and push jobs back onto their queues when a failure is detected. If you have an unexpected race condition or other errors (run out of file descriptors, etc) go will very quickly churn through your job queue.

But what about…

There are lots of other considerations, like what you’re running this against. On small elasticsearch clusters using these patterns to send data from go daemons to ES, I’ve been able to hit 50k requests/second with still plenty of room to grow.

You may need to pay extra attention to what libraries your using: how many redis connections can you have open? How many do you need?

Are you using keep-alive connections for HTTP? Is your receiver setup properly (nginx configs, etc)?

Is your MySQL or PostgreSQL server tuned to allow this many connections? Make sure you use connection pooling!

Lastly: Monitor all the things!

Send your data somewhere. I prefer StatsD, InfluxDB and Grafana for my monitoring stack. There is a ready-to-use go library quipo/statsd that I haven’t had issues with. One important thing to do is throw any data sends into a goroutine otherwise you might notice a slowdown while it tries to send the data.

Whether you use Grafana or anything else, its important to monitor. Without metrics on how your systems are running (ops/s, latency, etc) you have no insight into whether or not new changes have affected the overall throughput of your system.

Have any extra tips? Leave them in the comments below!

Using a custom HTTP Dialer in Go

Sep 19, 2015 - 1 minutes

Let’s make a function to generate an HTTP client for us using a custom dialer:

 1var DefaultDialer = &net.Dialer{}
 2
 3func GetHttpClient() http.Client {
 4	tr := &http.Transport{
 5		Dial:                DefaultDialer.Dial,
 6	}
 7
 8	client := http.Client{Transport: tr}
 9	return client
10}

Can you spot the bug?

By omitting the Timeout, KeepAlive timeouts in the first example, we’ve introduced a very subtle bug.

There is also another bug if you don’t handle TLS timeouts as well.

net/Dialer has some documentation on this.

Without providing a KeepAlive and a Timeout value, you could end up with connections that hang indefinitely. By omitting the TLS handshake timeout, the daemon would also hang trying to re-negotiate the SSL connection.

In my case this was causing a very random and hard to reproduce issue where the program would hang indefinitely.

Some good debugging tips are using strace to see what syscall its stuck in, and if your daemon is running in the foreground, using a SIGQUIT signal.

Here is a working version:

 1var DefaultDialer = &net.Dialer{Timeout: 2 * time.Second, KeepAlive: 2 * time.Second}
 2
 3func GetHttpClient() http.Client {
 4	tr := &http.Transport{
 5		Dial:                DefaultDialer.Dial,
 6		TLSHandshakeTimeout: 2 * time.Second,
 7}
 8
 9	client := http.Client{Transport: tr}
10	return client
11}

Faster docker builds using a cache

Jun 17, 2015 - 1 minutes

If you’re using bundler for your ruby or rails project and docker you will run into docker having to install your gems everytime. You can either make a base image that has the bundle cache already on it, or you can make a small cache step in your Dockerfile.

Here I’ve setup a cache user and host to store the cache tar. It will attempt to download and untar it, run bundle, then attempt to tar and re-upload it.

1RUN scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email protected]:~/project.tar.gz . || true
2RUN tar xzf project.tar.gz || true
3RUN bundle install --deployment --without development test
4RUN tar czf project.tar.gz vendor
5RUN scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no project.tar.gz [email protected]:~/ || true

Doing this cut build times for my image from a few minutes to a few seconds. If you have any other tricks for speeding up builds, let me know!

Here is an example of a clean way to communicate between React components without getting stuck passing @prop callbacks all around. Inspired by looking at the new Flux React utilities.

We’re going to start off with a simple HAML file:

1%script{src: "https://cdnjs.cloudflare.com/ajax/libs/react/0.12.2/react-with-addons.js"}
2%div{data: { ui: 'alerts' }}
3%div{data: { ui: 'widgets' }}
4:javascript
5  React.renderComponent(Widgets(), document.querySelector('[data-ui="widgets"]'))
6  React.renderComponent(Alerts(), document.querySelector('[data-ui="alerts"]'))

Next comes our Widget component.

1{div, button} = React.DOM
2
3Widgets = React.createClass
4	render: ->
5    div className: 'widget',
6    	button className: 'btn btn-primary', onClick: (=> @_sendMsg('Testing')), 'Click Me'
7	_sendMsg: (msg) ->
8    $('[data-ui="alerts"]').trigger("message", ["Widget clicked."])

On line 1 we’re defining some easy helper methods to access the React.DOM object - otherwise on every line we’d be writing something like React.DOM.div or whichever element we were going to call.

Line 4 is our render method. Everytime state gets mutated or the component is loaded, this method is called.

On line 6 we’re creating an anonymous function but passing in the local scope using a fat arrow => so we can access our other functions in the class. We call it inside an anonymous function so we can pass an argument to it, in this case the message.

Line 7 is our function that fires the event. I’m using the _sendMsg syntax to denote it is a private function. The first argument to the jQuery event emitter is the event name, followed by a list of arguments.

Now lets write our Alert handler and go through it line by line.

 1{div} = React.DOM
 2Alerts = React.createClass
 3  messageTimeout: 5000
 4  getInitialState: ->
 5    messages: []
 6
 7  componentDidMount: ->
 8    $('[data-ui="alerts"]').on 'message', (event, msg) =>
 9      msgs = @state.messages
10      msgs.push(msg)
11      @setState(messages: msgs)
12
13  componentDidUpdate: ->
14    @state.messages.map (msg, index) =>
15      setTimeout(( => @_removeMsg(index)), @messageTimeout)
16
17  render: ->
18    div {},
19      @state.messages.map (msg, index) =>
20        div className: 'alert alert-info',
21          msg
22
23  _removeMsg: (index) ->
24    msgs = @state.messages
25    msgs.splice(index, 1)
26    @setState(messages: msgs)

Line 1 we’re doing the same thing as before, creating a little helper method.

Line 3 is a class variable (we also could have used props here but I went with the class variable instead).

Line 4 is a function that defines the initial state of the component once it is mounted on the page. Here we are saying that there is an empty messages array.

Line 7 is a life cycle event of a React component, called componentDidMount which is called after the component has been rendered into the DOM and mounted. Here we are telling jQuery to bind to any events that are triggered on the [data-ui="alerts"] object and process them. We take the current messages from @state.messages, push the newest message on to the end and then finally call @setState to mutate the components state.

Now the next part on line 13 is how we can gracefully remove messages after they have been rendered. componentDidUpdate is another React life cycle event and is called after a render occurs (and renders occur because the component was updated). We iterate over each message using the map function and call setTimeout with an anonymous function that calls @_removeMsg and passes in an index. @messageTimeout is how we access the class variable defined at the top of the file.

Line 17 is a render call to display all the messages. Note that it is wrapped in a div because you can’t return a collection of objects from render, it must a single root element with nodes underneath.

Line 23 is our message removal function. We set @state.messages to a local variable, remove one element at index and then mutate the state by setting it to our local variable with @setState.

Below is an example of the final product.

See the Pen ZYoEWg by Josh Rendek (@joshrendek) on CodePen.

I’d like to thank my friend/co-worker Robert Pearce for getting me into React and showing me that everything doesn’t need to be jQuery!

Rails force HTTPS for your application

Aug 22, 2014 - 1 minutes

This is your setup:

  • Nginx redirecting HTTP to HTTPS
  • Unicorn/puma/(etc) being reverse proxied to via nginx

What you may not realize:

When you’re submitting a form and using the redirect_to or respond_with @object, location: [] methods rails may not pick up that you want to use HTTPS as the protocol, adding this to your application_controller.rb ensures every URL generated by rails will go to a secure protocol:

1def default_url_options(options={})
2    options.merge!(protocol: 'https') unless Rails.env.development? || Rails.env.test?
3    options
4end

I ran into issues with rails redirecting to plain HTTP when doing some uploading utilizing iframes (and Chrome rightfully blocking the redirect back to http). Hope this helps someone else! No need to use any special gems.

Note: This only applies if you want all actions to be HTTPS though.

I was working on the agent for SSH Pot and ran into something interesting last night. A lot of the brute force attempts attempt to run a command like this:

1ssh [email protected] 'uname'

This is different than:

1ssh [email protected]
2$ uname

The first command is executing a command then exiting, the second is actually logging in and giving the user a shell. The first requests a exec subsystem and the second requests a shell subsystem - so there are two ways to handle it.

 1func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
 2	for req := range in {
 3		ok := true
 4		logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
 5		switch req.Type {
 6		case "shell":
 7			req.Reply(ok, nil)
 8		case "exec":
 9			if string(req.Payload) == string("uname") {
10				channel.Write([]byte("\n\rLinux\n\r"))
11			}
12
13			channel.Close()
14		}
15	}
16}

When logging in my logfile it would show something like:

1[request exec]: uname

And even when comparing the two side by side with something like this:

1logfile.Println("["+string(req.Payload)+"]:["+"uname"+"]")

I would get this output:

1[uname]:[uname]

Yet the comparison on line 9 would not get hit. After sitting and thinking about it for a while I decided to print the bytes out:

1INFO: 2014/07/07 23:15:18 sshd.go:157: [0 0 0 5 117 110 97 109 101]
2INFO: 2014/07/07 23:15:18 sshd.go:158: [117 110 97 109 101]

Aha! So for some reason req.Payload is padded with 3 null bytes and a ENQ byte (hex 5).

Here is the corrected version removing the correct bytes - now the string comparison works:

 1func HandleShellRequest(channel ssh.Channel, in <-chan *ssh.Request) {
 2	for req := range in {
 3		ok := true
 4		logfile.Println("[request " + req.Type + "]: " + string(req.Payload))
 5		switch req.Type {
 6		case "shell":
 7			req.Reply(ok, nil)
 8		case "exec":
 9			if string(req.Payload[4:]) == string("uname") {
10				channel.Write([]byte("\n\rLinux\n\r"))
11			}
12
13			channel.Close()
14		}
15	}
16}

This is a short example showing how to use an interface to ease testing, and how to use an interface with running shell commands / other programs and providing mock output.

Source on Github

Here is our main file that actually runs the commands and prints out “hello”.

 1package main
 2
 3import (
 4	"fmt"
 5	"os/exec"
 6)
 7
 8// first argument is the command, like cat or echo,
 9// the second is the list of args to pass to it
10type Runner interface {
11	Run(string, ...string) ([]byte, error)
12}
13
14type RealRunner struct{}
15
16var runner Runner
17
18// the real runner for the actual program, actually execs the command
19func (r RealRunner) Run(command string, args ...string) ([]byte, error) {
20	out, err := exec.Command(command, args...).CombinedOutput()
21	return out, err
22}
23
24func Hello() string {
25	out, err := runner.Run("echo", "hello")
26	if err != nil {
27		panic(err)
28	}
29	return string(out)
30}
31
32func main() {
33	runner = RealRunner{}
34	fmt.Println(Hello())
35}

Here is our test file. We start by defining our TestRunner type and implementing the Run(...) interface for it.

This function builds up a command to run the current test file and run the TestHelperProcess function passing along all the args you originally sent. This lets you do things like return different output for different commands you want to run.

The TestHelperProcess function exits when run in the context of the test file, but runs when specified in the files arguments.

 1package main
 2
 3import (
 4	"fmt"
 5	"os"
 6	"os/exec"
 7	"testing"
 8)
 9
10type TestRunner struct{}
11
12func (r TestRunner) Run(command string, args ...string) ([]byte, error) {
13	cs := []string{"-test.run=TestHelperProcess", "--"}
14	cs = append(cs, args...)
15	cmd := exec.Command(os.Args[0], cs...)
16	cmd.Env = []string{"GO_WANT_HELPER_PROCESS=1"}
17	out, err := cmd.CombinedOutput()
18	return out, err
19}
20
21func TestHello(t *testing.T) {
22	runner = TestRunner{}
23	out := Hello()
24	if out == "testing helper process" {
25		t.Logf("out was eq to %s", string(out))
26	}
27}
28
29func TestHelperProcess(*testing.T) {
30	if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
31		return
32	}
33	defer os.Exit(0)
34	fmt.Println("testing helper process")
35}

Hopefully this helps someone else! I had a hard time finding some good, short examples on the internet that combined both interfaces and mocking like this.

More examples from os/exec/exec_test.go

A useful logger in Go

Jun 9, 2014 - 1 minutes

Small function that will print out useful information when invoked:

 1func logMsg(msg string) {
 2	pc, _, _, _ := runtime.Caller(1)
 3	caller := runtime.FuncForPC(pc).Name()
 4	_, file, line, _ := runtime.Caller(0)
 5	sp := strings.Split(file, "/")
 6	short_path := sp[len(sp)-2 : len(sp)]
 7	path_line := fmt.Sprintf("[%s/%s:%d]", short_path[0], short_path[1], line)
 8	log_string := fmt.Sprintf("[%s]%s %s:: %s", time.Now(), path_line, caller, msg)
 9	fmt.Println(log_string)
10}

Sample output:

1[2014-06-10 01:38:45.812215998 +0000 UTC][src/trex-client.go:15]{main.main}:: checking jobs - finish
2[2014-06-10 01:38:47.329650331 +0000 UTC][src/trex-client.go:15]{main.main}:: building package list - start