In this blog post, we will follow up the "Linux Namespaces and Go Don't Mix" post, and we will show how the problem mentioned in the previous post was resolved in the recent release of Go 1.10.
To recap, the main problem was that the Go runtime did not allow us to safely change a local state of an OS thread ("M" in the Go notation) scheduling a goroutine, even if the thread had been locked with runtime.LockOSThread.
The runtime could use the thread with the modified state to create a new thread which would inherit the state, and the new thread potentially would run any other goroutine resulting in an unexpected behaviour.
So, in our case, we could not safely change a network namespace from a goroutine. This was essential when configuring or listing container network interfaces via netlink in Weave Net.
Luckily, the post triggered some discussions which eventually led to a few relevant fixes to the Go runtime:
* #20676: No new thread will be created from a thread which is currently locked with runtime.LockOSThread.
* #20395: A locked thread will not be re-used to schedule other goroutines if a goroutine did not unlock it before exiting.
* #20458: If a thread was locked multiple times, runtime.UnlockOSThread has to be called equal number of times in order to unlock the thread.
All the fixes have been released with Go 1.10, and you can use the // +build go1.10 constraint in your code to require the minimal version of Go to compile it.
Finally, we can get rid of the ugly hack in Weave Net which used to create a separate OS process just to execute a Go function in the given network namespace.
All this looks great. However, there is one gotcha. runtime.LockOSThread will not necessarily run a new goroutine spawned by a locked one on the same thread.
To illustrate this behaviour, consider the following example:
// +build go1.10
package main
import (
"fmt"
"runtime"
"github.com/vishvananda/netns"
)
func main() {
ns, err := netns.New()
panicOnErr(err)
runtime.LockOSThread()
defer runtime.UnlockOSThread()
err = netns.Set(ns)
panicOnErr(err)
parentNetNS, err := netns.Get()
panicOnErr(err)
fmt.Println("parent:", parentNetNS)
wait := make(chan struct{})
go func() {
childNetNS, err := netns.Get()
panicOnErr(err)
fmt.Println("child:", childNetNS)
wait <- struct{}{}
}()
<-wait
}
func panicOnErr(err error) {
if err != nil {
panic(err)
}
}
The output of the example program:
parent: NS(4: 3, 4026532486) child: NS(5: 3, 4026531993)
As you can see, the child goroutine ended up running in a different network namespace than the parent (the last number of each line in the output is the inode of the network namespace).
Therefore, do not spawn a new goroutine from a locked one if the new goroutine expected to be run on the same thread or a thread with the same modified state. This might become an issue when calling some library code from locked goroutines, as a developer might be unaware of whether the library internally spawns a goroutine. Detecting such unsafe cases could be a subject for a static analysis (hint: a relatively low hanging fruit).
Another thing to consider is possible performance penalties when exiting a locked goroutine without unlocking it. In such case, the runtime might create a new thread for scheduling goroutines which steals some CPU cycles from doing a useful job. Thus, if you strive for performance, do not forget to undo changes to a thread state and to unlock it, so that the runtime could re-use it.
Go 1.10 introduced the fixes which make it more suitable for programming systems which are aware of an underlying OS. However, spawning a goroutine from a locked one requires great care.
Fewer eyebrows are raised.