Posts on Jack's home on the web

Tracing large job failures to serial console bottlenecks from OOM events

Mon, 06 Oct 2025 00:00:00 +0000

The investigation timeline: From false DNS timeout leads to discovering the real culprit—a serial console bottleneck causing 3+ minute network outages.

Abstract Link to heading

Often for machine learning workloads, a single node of a large job crashing can stall the entire job. We run a very large AWS EKS cluster with large distributed workloads. This post is about investigating one issue where nodes would seemingly go NotReady for several minutes without a clear cause, causing job failures.

My process to debug DNS timeouts in a large EKS cluster

Fri, 01 Aug 2025 17:01:02 +0000

Abstract Link to heading

We run a very large AWS EKS cluster with lots of interesting challenges. This post is about a recent investigation into DNS resolution failures that we were able to root cause to an Elastic Network Interface (ENI) packets per second (PPS) limit and a further root cause of the combination of sudo defaults and ndots in our cluster DNS.

This post talks about the investigation process and the resolution.

Permanent infrastructure as code state drift from gzip cloud-init

Sun, 15 Sep 2024 13:59:23 -0800

Setup Link to heading

Most cloud providers allow you to set commands to run when a host comes online. The general format for this is called cloud-init or user-data. There’s full documentation for cloud-init here. In my particular case, I had extra files I wanted to write to the host using pulumi. To make it all fit, I was gzip compressing the files then base64 encoding the result.

I advise against changing the kubelet root directory

Sun, 21 Jul 2024 13:59:23 -0800

Fighting with hard drives

TL;DR Link to heading

Changing kubelet’s root directory can break CSI drivers and other components. You can work around this, but the effort isn’t worth it: stick with the default /var/lib/kubelet.

Motivation Link to heading

I was setting up Kubernetes kubelet on a host with numerous NVMe drives . I thought “Let’s use this for ephemeral-storage in Kubernetes”. The host had no need for all these drives and Kubernetes could allow pods to claim them for ephemeral storage.

Debugging a kubernetes node

Thu, 25 Apr 2024 13:59:23 -0800

Do you ever need to debug a node on kubernetes? Maybe the networking is broken, or you need to check something else out? The netshoot docker container contains a large number of helpful network tools like tcpdump, iperf, and more. Use a script like this to kubectl directly to a node with escalated privileges. Once in, try tcpdump -i any to see all network traffic on the node.

#!/bin/bash
set -exuo pipefail
if [[ "${1-}" == "" ]]; then
 kubectl run "netshoot-$(whoami)" --rm -i --tty --image nicolaka/netshoot -- /bin/bash
else
 kubectl run "netshoot-$(whoami)" --rm -i --tty --image nicolaka/netshoot \
 --overrides='{ 
 "spec": {
 "hostNetwork": true,
 "tolerations": [{
 "key": "",
 "operator": "Exists"
 }],
 "nodeSelector": {
 "kubernetes.io/hostname": "'${1}'"
 },
 "containers": [{
 "name": "netshoot",
 "image": "nicolaka/netshoot",
 "securityContext": {
 "privileged": true
 },
 "command": ["/bin/bash"],
 "stdin": true,
 "tty": true
 }]
 }
 }'
fi

For example, I can run this to get a shell on a node:

How I do sprint planning meetings for a software team

Tue, 27 Feb 2024 07:00:08 -0800

Image from UnSplash

Preface Link to heading

This post describes how I’ve been conducting sprint planning on my current team. It deviates from standard agile in a few ways. I document how and why towards the bottom. I’ve found this format works well for us, but there is no process that works for everyone. I script out the sprint planning process to keep it focused and make sure we don’t forget to cover anything important.

(Almost) Every infrastructure decision I endorse or regret after 4 years running infrastructure at a startup

Thu, 01 Feb 2024 13:59:47 -0800

Image from UnSplash

I’ve led infrastructure at a startup for the past 4 years that has had to scale quickly. From the beginning I made some core decisions that the company has had to stick to, for better or worse, these past four years. This post will list some of the major decisions made and if I endorse them for your startup, or if I regret them and advise you to pick something else.

Using Docker and Terraform for hermetic AWS Lambda CI/CD

Thu, 30 Apr 2020 17:01:00 +0000

Context Link to heading

We are using AWS Auora to manager our databases and I want to setup automatic rotation of database passwords. I start by storing them in AWS Secrets Manager. AWS actually releases publicly usable rotations for PostgreSQL, but I want to add the extra fields connstring and host_ro so clients can directly pull both the read only Auora DNS and a full PostgreSQL connection string.

After using both, I regretted switching from Terraform to CloudFormation

Thu, 10 Oct 2019 17:01:01 +0000

An unambiguous best practice for systems is to codify your infrastructure in a repeatable text format, rather than clicks in a UI. The common name for this is Infrastructure as Code. By far, the two most popular ways to do this, especially in AWS, are Terraform and CloudFormation.

I used Terraform for about 3 years at a startup before working for Twitch(AKA Amazon Jr) where I used Terraform very heavily before the company pushed hard to switch to all things Amazon, including CloudFormation. I tried very hard to develop best practices around both and used each for very complicated operational setups that spanned organizations. After thinking hard about the switch from Terraform to CloudFormation, I feel strongly that Terraform is probably the right choice for your company.

Closure driven tests: an alternative style to table driven tests in go

Tue, 17 Sep 2019 17:01:03 +0000

Table driven tests are a popular model for testing in Go and a clear improvement to traditional testing of one function per case. However, there’s an alternative style that can often be clearer than table driven tests that relies on function closures.

I’ll reference the example code on Dave Cheney’s post Prefer table driven tests for my examples.

The original way Link to heading

The code we want to test is a function that splits strings.

Using Go 1.13’s new ReportMetric API to benchmark t-digest implementations

Wed, 11 Sep 2019 16:01:01 +0000

Histograms are an important part of metrics and monitoring. Unfortunately, it is computationally expensive to accurately calculate a histogram or quantile value. A workaround is to use an approximation and a great one to use is t-digest. Unfortunately again, there’s no such thing as an accurate approximation. There are various t-digest implementations in Go, but they do things slightly differently and there’s no way to track how accurate their approximations are over time and no great way for consumers of quantile approximation libraries to benchmark them against each other.

Interface Contracts Matter

Wed, 07 Aug 2019 16:31:02 +0000

A recent vulnerability in the go-redis library allowed correctly configured clients to return the results of other redis commands in the event of redis failures. While all code is subject to bugs, this one could have been prevented if proper care was taken when creating and using interfaces.

Bug behavior Link to heading

Most clients reuse TCP connections between RPC calls to eliminate the delay of TCP connection overhead. TCP connections have both an input and output stream. Redis operates on a request/response pattern where commands are shared on the same socket by sending multiple requests and expecting multiple responses. Most of the time responses are read after the request is sent. For example, there could be two requests to get the key “money:*” for two different users with jack having $200 and john having $100.

Go’s wrapped return pattern to clean up objects with background goroutines

Tue, 02 Jul 2019 17:01:01 +0000

Problem Link to heading

You have a constructor function that creates an object with a background goroutine. You want the object to garbage collect itself even if people don’t explicitly close it. This isn’t possible since since the background goroutine is always running and pointing to your created object.

Solution Link to heading

Wrap the returned object and use a finalizer on the returned object to shut down the background goroutine.

Design better software abstractions using bipartite composition

Wed, 29 May 2019 17:04:00 +0000

Problem Link to heading

You’ve written an object or class. It’s somewhat complex and you want to iterate ways to better abstract your code but are unsure how to start. There exists documentation about what a good abstraction looks like, but no concrete iterative steps.

Being deep inside your own code can make you blind to obvious alternatives.

Solution Link to heading

Create a bipartite graph of instance methods and member variables. Draw edges between methods and the variables they use. Then move isolated components into separate classes.

Go’s append is not always thread safe

Tue, 12 Jun 2018 16:01:01 +0000

Problem Example Link to heading

I commonly see bugs appending to slices in Go in a way that is not thread safe. A simple example is the unit tests below. This test has two goroutines append to the same slice. If you run this test with the -race flag, it works just fine.

package main

import (
 "sync"
 "testing"
)

func TestAppend(t *testing.T) {
 x := []string{"start"}
 
 wg := sync.WaitGroup{}
 wg.Add(2)
 go func() {
 defer wg.Done()
 y := append(x, "hello", "world")
 t.Log(cap(y), len(y))
 }()
 go func() {
 defer wg.Done()
 z := append(x, "goodbye", "bob")
 t.Log(cap(z), len(z))
 }()
 wg.Wait()
}

Now, let’s change the code just slightly to create the x slice with more spare capacity. The only thing that changed was line 9.

Using Go 1.10 new trace features to debug an integration test

Fri, 26 Jan 2018 20:08:28 +0000

I had an integration test that was very complex (most are, sadly) and took too long to start. I was able to diagnose why using new features of Go’s 1.10 trace tool. This post describes my workflow and what led me there.

The example code I’m using for this post is at https://github.com/cep21/trace-110. My integration test was a bit more complex, but generally had startup that was slow due to a single goroutine out of many that did lots of different things. In this example, takeIO is similar to the “startup” part of my integration test.

Aspects of a good Go library

Mon, 18 Dec 2017 19:32:56 +0000

A short checklist of what I wish for in a good Go library, in no particular order. This is a companion to the effective go list, Go code review comments list, and Go proverbs list.

In general, when given two reasonable ways to do something, defer to the option that does not violate these rules. Bend these rules only with a strong explanation why.

Dependencies Link to heading

Tagged library versions Link to heading

Use git tags to version your library. Semantic versioning is a reasonable system. If you care enough to disagree with semantic versioning, then you’re not the target here. :)

Go Experience report: Go’s type system

Thu, 07 Sep 2017 16:01:03 +0000

What I wanted to do Link to heading

Today’s pkg/errors Link to heading

Package errors is a popular go error wrapping package. The StackTrace type represents the stack trace with an []Frame. Errors wrapped by package errors implicitly implement the stackTracer interface, which allows users to extract stack information.

type StackTrace []Frame 
type Frame uintptr

type stackTracer interface { 
 StackTrace() errors.StackTrace 
}

Simultaneously to this, there exist libraries that would like to extract stack frame information from errors, if it’s present. One example is the rollbar logging library, which allows logging errors with the stack trace that created them. The only way for a rollbar logging library to support pkg/errors is to import the library directly and type assert for a StackTrace function.

Interface wrapping method erasure

Wed, 26 Jul 2017 14:02:39 +0000

Problem Link to heading

The go language spec has no way to simultaneously add or augment behavior to an interface while maintaining commonly accepted best programming practices around minimal interface design. While not unique to Go itself, a combination of Go’s feature set and evolved best practices have caused Go library authors to suffer disproportionately to other language developers.

Abstract example Link to heading

Given an interface I of a concrete type S, there is no way to add behavior to I while returning a type of I that also implements the functions of S, without knowing S’s functions a priori. Specifically the below example will panic.

DynamoDB trivia: reads throttled but under capacity

Mon, 23 Jan 2017 21:02:13 +0000

I get woken up to an alert about read capacity on one of my DynamoDB tables. Here is the error being logged.

ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.

When I load my CloudWatch graphs, and it appears I’m well below capacity.

Cloudwatch graph of DynamoDB metrics.

Creating custom Go profiles with pprof

Mon, 19 Dec 2016 18:01:02 +0000

Go’s pprof package is frequently used for heap and CPU profiles, but a little used feature of the package is being able to create custom profiles. Custom profiles are useful for tracking down leaked resources and troublesome stack traces that are abusing a library’s API.

Profile Link to heading

From the godoc for pprof itself:

A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation. Packages can create and maintain their own profiles; the most common use is for tracking resources that must be explicitly closed, such as files or network connections.

What “accept interfaces, return structs” means in Go

Mon, 05 Dec 2016 18:01:01 +0000

Not lacking nuance Link to heading

I’ve mentioned a general guideline of accept interfaces, return structs in a previous post and multiple times on code reviews to coworkers, but often get the question “Why”. Especially since this isn’t a hard rule. The crux of the idea, and understanding when to bend it, is in the balance of avoiding preemptive abstractions while maintaining flexibility.

The abstract gopher

Go race condition trivia

Tue, 15 Nov 2016 17:01:02 +0000

I recently ran into a -race error message that stumped me for a while, so I’ll retell it here in phases to see if you can catch it.

The code Link to heading

The intent of the code was a service abstraction that ran a goroutine in the background to send runtime stats to Stastd. A small example of the code follows:

type Stats struct {
 prev runtime.MemStats
}

func (p *Stats) stats() {
 ms := runtime.MemStats{}
 runtime.ReadMemStats(&ms)
 mallocCount := ms.Mallocs - p.prev.Mallocs
 fmt.Println("malloc change is", mallocCount)
 p.prev = ms
}

func (p *Stats) Loop(ctx context.Context) {
 runtime.ReadMemStats(&p.prev)
 for {
 select {
 case <-time.After(time.Millisecond * 100):
 p.stats()
 case <-ctx.Done():
 return
 }
 }
}

You can be 100% sure that I never run Loop() for the same Stats struct. And you’re 100% sure I only make a single Stats{} struct in my program. Before reading forward, inspect the above code and see if you can find the race condition I hit while using Stats.

For Go builtins, nil and empty read the same

Wed, 21 Sep 2016 16:44:29 +0000

An interesting property of Go’s built-in data structures is that read operations work even when they are nil! Not only that, but read operations on nil built-in structures behave just as if they are non-nil but empty. This is different from other languages and allows some interesting properties of structures in Go. Here is a full playground link.

Why nil matters for built-in structures Link to heading

Go does not have a way to force constructors for structs. The closest we can get is making the struct private and documenting users to get an instance from a function named New. This means there’s no way to initialize a member variable of a struct before people use it. Any maps, channels, or slices in your struct will start as nil.

Go 1.7 httptrace and context debug patterns

Mon, 29 Aug 2016 17:01:02 +0000

Package httptrace is a new addition in Go 1.7 that allows users to trace the steps of a HTTP request. This package also models an interesting design pattern around adding optional debugging to a library. However, it’s not the only way to accomplish this goal. I’ll briefly overview the package and provide alternative implementations for httptrace.

httptrace Link to heading

The basic idea of httptrace is that a laundry list of callbacks are optionally attached to a Context object that is added to a HTTP request. During the inner workings of the request, if any of these callbacks are set, they are called during different stages of the request.

How to correctly use context.Context in Go 1.7

Tue, 19 Jul 2016 15:56:02 +0000

This post will talk about a new library in Go 1.7, the context library, and when or how to correctly use it. Required reading to start is the introductory post that talks a bit about the library and generally how it is used. You can read documentation for the context library on tip.golang.org.

How to integrate Context into your API Link to heading

The most important thing to remember when integrating Context into your API is that it is intended to be request scoped. For example, it would make sense to exist along a single database query, but wouldn’t make sense to exist along a database object.

Go client library best practices

Tue, 05 Jul 2016 17:52:53 +0000

This post describes what I feel are best practices when creating a client library for a service. The initial setup is that you’re writing a library with an API to talk to a RESTful service over HTTPS, and all the library needs to do is return JSON unmarshalled objects. I’ll use some examples from a client I wrote to talk to Smite’s API.

Directly use the struct Link to heading

While constructor functions are sometimes needed for Go libraries, it’s strongly preferred to support users directly using your struct{} object. For example, the Go standard library’s http client is used by directly instantiating it. The advantages of direct struct initialization vs constructor functions is worthy of its own post, but the primary reasons are local clarity and simplicity of code.

Repository organization tips in Go

Mon, 27 Jun 2016 22:39:42 +0000

This post will describe common approaches to organizing Go code into packages and what I feel is the ideal directory structure for Go repositories. Before starting with this post, it may be useful to also read https://blog.golang.org/package-names first for context.

A setup with package proliferation Link to heading

Let’s take an example application called gatekeeper. It may have a public client that other programs use to connect to it and may have parts around authentication and payment, used by gatekeeper internally. One structure is like the following.

Preemptive Interface Anti-Pattern in Go

Thu, 23 Jun 2016 22:47:58 +0000

In this post I will describe the Preemptive Interface pattern often used in code and why I think it is generally an incorrect pattern to follow in Go.

What is a Preemptive Interface Link to heading

Interfaces are a way to describe behavior and exist in most typed languages. Preemptive interfaces are when a developer codes to an interface before an actual need arrises. An example may look like the following.