Monday, October 5, 2015

Fun with terminals, character sets, Unicode, and Go

As part of my recent work on Tcell, I've recently added some pretty cool functionality for folks who want to have applications that can work reasonably in many different locales.

For example, if your terminal is running in UTF-8, you can access a huge repertoire of glyphs / characters.

But if you're running in a non-UTF-8 terminal, such as an older ISO 8859-1 (Latin1) or KOI8-R (Russian) locale, you might have problems.  Your terminal won't be able to display UTF-8, and your key strokes will probably be reported to the application using some 8-bit variant that is incompatible with UTF-8.  (For ASCII characters, everything will work, but if you want to enter a different character, like Я (Russian for "ya"), you're going to have difficulties.

If you work on the console of your operating system, you probably have somewhere around 220 characters to play with.  You're going to miss some of those glyphs.

Go of course works with UTF-8 natively.  Which is just awesome.

Until you have to work in one of these legacy environments.   And some of the environments are not precisely "legacy".  (GB18030 has the same repertoire as UTF-8, but uses a different encoding scheme and is legally mandatory within China.)

If you use Tcell for your application's user interface, this is now "fixed".

Tcell will attempt to convert to characters that the user's terminal understands on output, provided the user's environment variables are set properly ($LANG, $LC_ALL, $LC_CTYPE, per POSIX).  It will also convert the user's key strokes from your native locale to UTF-8.  This means that YOU, the application developer, can just worry about UTF-8, and skip the rest.  (Unless you want to add new Encodings, which is entirely possible.)

Tcell even goes further.

It will use the alternate character set (ACS) to convert Unicode drawing characters to the characters supported by the terminal, if they exist -- or to reasonable ASCII fallbacks if they don't.  (Just like ncurses!)

It will also cope with both East Asian full-width (or ambiguous width) characters, and even properly handles combining characters.  (If your terminal supports it, these are rendered properly on the terminal.  If it doesn't, Tcell makes a concerted effort to make a best attempt at rendering -- preserving layout and presenting the primary character even if the combining character cannot be rendered.)

The Unicode (and non-Unicode translation) handling capabilities in Tcell far exceed any other terminal handling package I'm aware of.

Here are some interesting screen caps, taken on a Mac using the provided unicode.go test program.

First the UTF-8.  Note the Arabic, the correct spacing of the Chinese glyphs, and the correct rendering of Combining characters.  (Note also that emoji are reported as width one, instead of two, and so take up more space than they should.  This is a font bug on my system -- Unicode says these are Narrow characters.)
Then we run in ISO8859-1 (Latin 1).  Here you can see the accented character available in the Icelandic word, and some terminal specific replacements have been made for the drawing glyphs.  ISO 8859-1 lacks most of the unusual or Asian glyphs, and so those are rendered as "?".  This is done by Tcell -- the terminal never sees the raw Unicode/UTF-8.  That's important, since sending the raw UTF-8 could cause my terminal to do bad things.

Note also that the widths are properly handled, so that even though we cannot display the combining characters, nor the full-width Chinese characters, the widths are correct -- 1 cell is taken for the combining character combinations, and 2 cells are taken by the full width Chinese characters.

Then we show off legacy Russian (KOI8-R):   Here you can see Cyrillic is rendered properly, as well as the basic ASCII and the alternate (ACS) drawing characters (mostly), while the rest are filled with place holder ?'s.

And, for those of you in mainland China, here's GB18030:  Its somewhat amusing that the system font seems to not be able to cope with the combining enclosure here.  Again, this is a font deficiency in the system.

As you can see, we have a lot of rendering options.  Input is filtered and converted too.  Unfortunately, the mouse test program I use to verify this doesn't really show this (since you can't see what I typed), but the Right Thing happens on input too.

Btw, those of you looking for mouse usage in your terminal should be very very happy with Tcell.  As far as I can tell, Tcell offers improved mouse handling on stock XTerm over every other terminal package.  This includes live mouse reporting, click-drag reporting, etc.   Here's what the test program looks like on my system, after I've click-dragged to create a few boxes:

I'm super tempted to put all this together to write an old DOS-style game.  I think Tcell has everything necessary here to be used as the basis for some really cool terminal hacks.

Give it a whirl if you like, and let me know what you think.

Friday, October 2, 2015

Tcell - Terminal functionality for Pure Go apps

Introducing Tcell  - Terminal functionality for Pure Go apps

As part of the work I've done on govisor, I had a desire for rich terminal functionality so that I could build a portable curses-style management application.

This turned out to be both easier and harder than I thought.

Easier, because there was an implementation to start from -- termbox-go, but harder because that version wasn't portable to the OS I cared most about, and fell short in some ways that I felt were important.  (Specifically, reduced functionality for mice, colors, and Unicode.)

This led me to create my own library; I've made some very very different design choices.  These design choices have let me both support more platforms (pretty much all POSIX platforms and Windows are included), increase support for a much richer set of terminals and terminal capabilities, etc.

The work is called "Tcell" (for terminal cells, which is the unit we operate on -- if you don't like the name ... well I'm open to suggestions.  I freely admit that I suck at naming -- and its widely acknowledged that naming is one of the "hard" problems in computer science.)

As part of this work, I've implemented a full Terminfo string parser/expander in Go.  This isn't quite as trivial as you might think, since the parameterized strings actually have a kind of stack based "minilanguage", including conditionals, in them.

Furthermore, I've wound up exploring lots more about mice, and terminals, and the most common emulators.  As a result, I think I have created a framework that can support very rich mouse reporting, key events, colors, and Unicode.  Of course, the functionality will vary somewhat by platform and terminal (your vt52 isn't going to support rich colors, Unicode, or even a mouse, after all.  But Xterm can, as can most modern emulators.)

This work would be a nice basis for portable readline implementations, getpassphrase, etc.  Or a full up curses implementation.  It does support a compatibility mode for termbox, and most termbox applications work with only changing the import line.   I still need to change govisor, and the topsl panels library, to use tcell, but part of that work will be including much more capable mouse reporting, which tcell now facilitates.

Admittedly, if you cannot tolerate cgo in any of your work -- this isn't for you -- yet.  I will probably start adding non-cgo implementations for particular ports such as Linux, Darwin, and FreeBSD.  But for other platforms there will always be cgo as a fallback.  Sadly, unless the standard Go library evolves to add sufficient richness that all of the termios functionality are available natively on all platforms (which is a real possibility), cgo will remain necessary.  (Note that Windows platform support does not use CGO.)

Tuesday, September 22, 2015

On Go, Portability, and System Interfaces

I've been noticing more and more lately that we have a plethora of libraries and programs written for Go, which don't work on one platform or another.  The root cause of these is often the use of direct system call coding to system calls such as ioctl().  On some platforms (illumos/solaris!) there is no such system call.

The Problems

But this underscores a far far worse problem, that has become common (mal)-practice in the Go community.  That is, the coding of system calls directly into high level libraries and even application programs.  For example, it isn't uncommon to see something like this (taken from termbox-go):
func tcsetattr(fd uintptr, termios *syscall_Termios) error {
        r, _, e := syscall.Syscall(syscall.SYS_IOCTL,
                fd, uintptr(syscall_TCSETS), uintptr(unsafe.Pointer(termios)))
        if r != 0 {
                return os.NewSyscallError("SYS_IOCTL", e)
        return nil

This has quite a few problems with it.
  1. It's not platform portable.  This function depends on a specific implementation of tcsetattr() that is done in terms of specific ioctl()s.  For example, TCSETS may be used on one platform, but on others TIOCSETA might be used.
  2. It's not Go portable, since SYS_IOCTL isn't implemented on platforms like illumos, even though as a POSIX system we do have a working tcsetattr(). 
  3. The code is actually pretty unreadable, and somewhat challenging to write the first time correctly.
  4. The code uses unsafe.Pointer(), which is clearly something we ought to avoid.
  5. On some platforms, the details of the ioctls are subject to change, so that the coding above is actually fragile.  (In illumos & Solaris system call interfaces are "undocumented", and one must use the C library to access system services.  This is our "stable API boundary".  This is somewhat different from Linux practice; the reasons for this difference is both historical and related to the fact that Linux delivers only a kernel, while illumos delivers a system that includes both the kernel and core libraries.)
How did we wind up in this ugly situation?

The problem I believe stems from some misconceptions, and some historical precedents in the Go community.   First the Go community has long touted static linking as one of its significant advantages.  However, I believe this has been taken too far.

Why is static linking beneficial?  The obvious (to me at any rate) reason is to avoid the dependency nightmares and breakage that occurs with other systems where many dynamic libraries are brought together.  For example, if A depends directly on both B and C, and B depends on C, but some future version of B depends on a newer version of C that is incompatible with the version of C that A was using, then we cannot update A to use the new B.  And when the system components are shared across the entire system, the web of dependencies gets to be so challenging that managing these dependencies in real environments can become a full time job, consuming an entire engineering salary.

You can get into surprising results where upgrading one library can cause unexpected failures in some other application.  So the desire to avoid this kind of breakage is to encode the entire binary together, in a single stand-alone executable, so that we need never have a fear as to whether our application will work in the future or not.  As I will show, we've not really achieved this with 100% statically linked executables in Go, though I'll grant that we have greatly reduced the risk.

This is truly necessary because much of the open source ecosystem has no idea about interface stability nor versioning interfaces.  This is gradually changing, such that we now have ideas like semver coming around as if they are somehow new and great ideas.  The reality is that commercial operating system vendors have understood the importance of stable API boundaries for a very very long time.  Some, like Sun, even made legally binding promises around the stability of their interfaces.  However, in each of these cases, the boundary has to a greater or lesser extent been at the discretion of the vendor.

Until we consider standards such as POSIX 1003.1.  Some mistakenly believe that POSIX defines system calls.  It does not.  It defines a C function call interface.  The expectation is that many of these interfaces have 1:1 mappings with system calls, but the details of those system calls are completely unspecified by POSIX.

Basically, the Go folks want to minimize external dependencies and the web of failure that can lead to.  Fixing that is a goal I heartily agree with.  However, we cannot eliminate our dependency on the platform.  And using system calls directly is actually worse, because it moves our dependency from something that is stable and defined by standards bodies, to an interface that is undocumented, not portable, and may change at any time.

If you're not willing to have a dynamic link dependency on the C library, why would you be willing to have a dependency on the operating system kernel?  In fact, the former is far safer than the latter! (And on Solaris, you don't have a choice -- the Go compiler always links against the system libraries.)

Harmful results that occur with static linking

If the application depends on a library that has a critical security update, it becomes necessary to recompile the application.  If you have a low level library such as a TLS or HTTP client, and a security fix for a TLS bug is necessary (and we've never ever ever had any bugs in TLS or SSL implementation, right?), this could mean recompiling a very large body of software to be sure you've closed the gaps.
With statically linked programs, even knowing which applications need to be updated can be difficult or impossible.  They defy the most easy kinds of inspection, using tools like ldd or otool to see what they are built on top of.

What is also tragic, is that static executables wind up encoding the details of the kernel system call interface into the binary.  On some systems this isn't a big deal because they have a stable system call interface.  (Linux mostly has this -- although glibc still has to cope with quite a few differences here by handling ENOSYS, and don't even get me started on systemd related changes.)  But on systems like Solaris and illumos, we've historically considered those details a private implementation detail between libc and kernel.  And to prevent applications from abusing this, we don't even deliver a static libc.  This gives us the freedom to change the kernel/userland interface fairly freely, without affecting applications.

When you consider standards specifications like POSIX or X/OPEN, this approach makes a lot of sense.  They standardize the C function call interface, and leave the kernel implementation up to the implementor.

But statically linked Go programs break this, badly.  If that kernel interface changes, we can wind up breaking all of the Go programs that use it, although "correct" programs that only use libc will continue to work fine.

The elephant in the room (licensing)

The other problem with static linking is that it can create a license condition that is very undesirable.  For example, glibc is LGPL.  That means that per the terms of the LGPL it must be possible to relink against a different glibc, if you link statically.

Go programs avoid this by not including any of the C library statically. Even when cgo is used, the system libraries are linked dynamically.  (This is usually the C library, but can include things like a pthreads library or other base system libraries.)

In terms of the system, the primary practice for Go programmers has been to use licenses like MIT, BSD, or Apache, that are permissive enough that static linking of 3rd party Go libraries is usually not a problem.  I suppose that this is a great benefit in that it will serve to help prevent GPL and LGPL code from infecting the bulk of the corpus of Go software.

The Solutions

The solution here is rather straightforward.

First, we should not eschew use of the C library, or other libraries that are part of the standard system image.  I'm talking about things like libm, libc, and for those that have them, libpthread, libnsl, libsocket.  Basically the standard libraries that every non-trivial program has to include.  On most platforms this is just libc.  If recoded to use the system's tcsetattr (which is defined to exist by POSIX), the above function looks like this:
// include <termios.h>
import "C"
import "os"
func tcsetattr(f *os.File, termios *C.struct_termios) error {
        _, e := C.tcsetattr(, C.TCSANOW, termios)
        return e
The above implementation will cause your library or program to dynamically link against and use the standard C library on the platform.  And it works on all POSIX systems everywhere and because it uses a stable documented standard API, it is pretty much immune to breakage from changes elsewhere in the system.  (At least any change that broke this implementation would also break so many other things that the platform would be unusable.  Generally we can usually trust people who make the operating system kernel and C library to not screw things up too badly.)

What would be even better, and cleaner, would be to abstract that interface above behind some Go code, converting between a Go struct and the C struct as needed, just as is done in much of the rest of the Go runtime.  The logical place to do this would be in the standard Go system libraries.  I'd argue rather strongly that core services like termio handling ought to be made available to Go developers in the standard system libraries that are part of Go, or perhaps more appropriately, with the repository.

In any event, if you're a Go programmer, please consider NOT directly calling syscall interfaces, but instead using higher level interfaces, and when those aren't already provided in Go, don't be afraid to use cgo to access standard functions in the C library.  Its far far better for everyone that you do this, than that you code to low level system calls.

Sunday, September 20, 2015

Announcing govisor 1.0

I'm happy to announce that I feel I've wrapped up Govisor to a point where its ready for public consumption.

Govisor is a service similar to supervisord, in that it can be used to manage a bunch of processes.  However, it is much richer in that it understands process dependencies, conflicts, and also offers capabilities for self-healing, and consolidated log management.

It runs as an ordinary user process, and while it has some things in common with programs like init, upstart, and Solaris SMF, it is not a replacement for any of those things.  Instead think of this is a portable way to manage a group of processes without requiring root.  In my case I wanted something that could manage a tree of microservices that was deployable by normal users.  Govisor is my answer to that problem.

Govisor is also written entirely in Go, and is embeddable in other projects.  The REST server interface uses a stock http.ServeHTTP interface, so it can be used with various middleware or frameworks like the Gorilla toolkit.

Services can be implemented as processes, or in native Go.

Govisor also comes with a nice terminal oriented user interface (I'd welcome a JavaScript based UI, but I don't write JS myself).  Obligatory screen shots below.

Which actually brings up the topic of "tops'l"  (which is a contraction of top-sail, often used in sailing).  Govisor depends the package tops -- "terminal oriented panels support library"), which provides a mid-level API for interacting with terminals.   I created topsl specifically to help me with creating the govisor client application.  I do consider topsl ready for use by Govisor, but I advise caution if you want to use it your own programs -- its really young and there are probably breaking changes looming in its future.

The documentation is a bit thin on the ground, but if you want to help or have questions, just let me know!  In the meantime, enjoy!

Saturday, May 2, 2015

MacOS X 10.10.3 Update is *TOXIC*

As a PSA (public service announcement), I'm reporting here that updating your Yosemite system to 10.10.3 is incredibly toxic if you use WiFi.

I've seen other reports of this, and I've experienced it myself.  What happened is that the update for 10.10.3 seems to have done something tragically bad to the WiFi drivers, such that it completely hammers the network to the point of making it unusable for everyone else on the network.

I have late 2013 iMac 27", and after I updated, I found that other systems started badly badly misbehaving.  I blamed my ISP, and the router, because I was seeing ping times of tens of seconds!
(No, not milliseconds, seconds!!!  In one case I saw responses over 64 seconds.)  This was on other systems that were not upgraded.  Needless to say, that basically left the network unusable.

(The behavior was cyclical -- I'd get a few tens of seconds where pings to would be in the 20 msec range, and then it would start to jump up very quickly until maxing up around a minute or so.  It would stay there for a minute or two, then rest or drop back to sane times.   But only very briefly.)

This was most severe when using a 5GHz network.  Switching down to 2.4GHz reduced some of the symptoms -- although still over 10 seconds to get traffic through and thoroughly unusable for a wide variety of applications.

There are reports that disabling Bluetooth may alleviate this, and also some people reported some success with clearing certain settings files.  I've not tried either of these yet.  Google around for the answer if you want to.  For now, my iMac 27" is powered off, until I can take the chance to disrupt the network again to try these "fixes".

Apple, I'm seriously seriously disappointed here.  I'm not sure at all how this testing got past you, but you need to fix this.  Its absolutely criminal that applying a recommended update with security critical fixes in it should turn my computer into a DoS device for my local network.  I'm shocked that several days later I've not seen a release update from Apple to fix this critical problem.

Anyway, my advice is, if possible, hold off for the update to 10.10.3.  Its tragically, horribly toxic not just to the upgraded device, but probably to the entire network it sits on.  I'm a little astounded that a bug in the code could hose an entire WiFi network as badly as this does -- I would have previously thought this impossible (and this was part of the reason why it took a while to diagnose this down to the computer -- I thought the ridiculous ping responses had to be a problem with my upstream provider!)

I'll post an update here if one becomes available.

Tuesday, April 14, 2015

vtprint - blast from the past

I recently decided to have a look back at some old stuff I wrote.  Blast from the past stuff.  One of the things I decided to look at was the very first open source program I wrote -- something called vtprint.

vtprint is a tool that borrowed ideas stolen from the pine mail program.  This comes from the days of serial dialups, and legacy (Kermit or ProComm for those who remember such things) serial connectivity over 14.4K modems.  Printing a file on a remote server was hard back then; you could transfer the file using xmodem using rx(1), or its "newer" variants, rb(1) or rz(1), but this was awkward.  It turns out that most physical terminals had support for an escape sequence that would start hardcopy to a locally attached printer, and another that would stop it.  And indeed, many terminal emulators have the same support.  (I added support for this myself to the rxvt(1) terminal emulator back in the mid 90's, so any emulators derived from rxvt inherit this support as well.)

So vtprint, which in retrospect could have been written in a few lines of shell code, is a C program.  It supports configuration via a custom file called "vtprintcap", that can provide information about different $TERM values and the escape sequences they use.  So for example, TVI925 terminals use a different escape sequence than VT100 or VT220 terminals.

The first version of it, 1.0, is lost in time, and was written back in 1991 or 1992 while I was working as a "Computer Consultant" (fancy name for BOFH, though I also did a lot of programming for the school's Windows 3.1 network and Novell NetWare 3.12 server) at the Haas School of Business at UC Berkeley.  (I was a Chemical Engineering student there, and I flunked out in thermodynamics -- I still blame Gibbs Free Energy ΔG as a concept I never truly could get my mind around.  It took a couple of years before I finally gave up fighting "hard engineering" and accepted my true calling as a software engineer.)

Anyway, I transferred to SDSU into the Computer Science department.  While there, back in 1993, I updated vtprint significantly, and that is the release that lives on at SourceForge.

Today I imported that project to github.  (And let me tell you, it took some hoops to do this.  Apparently all the old CVS projects that were going to have converted to CVS are supposed to already have done so, because at this point even the conversion tools are mostly crufty and broken.  Maybe I'll post what I did there.)  Most of the files indicate an age of 11 years ago.  That's because 11 years ago I imported the source into CVS for the benefit of SourceForge.  The actual files date back to 1994, and you can even see my old email address -- which hasn't worked for a couple of decades, in the files.

So, I gave vtprint a whirl today for the first time in years:

garrett@hipster{4}> ./vtprint README
*** vtprint (v2.0.2) ***
Copyright 1993-1994, Garrett D'AmoreLast revised October 25, 1994.
NO WARRANTY!  Use "vtprint -w" for info.Freely redistributable.  Use "vtprint -l" for info.
vtprint: Can't open /usr/local/lib/vtprint/vtprintcap, using builtin codes.vtprint: Using <stdout> for output.vtprint: Output flags: formfeedvtprint: Printed README.vtprint: Successfully printed 1 file (1 specified).

Sadly, MacOS X does not appear to emulate the escape sequences properly.  But iTerm2 does so very nicely.  It even generates a nice print preview using the standard MacOS X print dialog.  Sadly, it does not pass through PostScript unmolested, but for printing ASCII files it works beautifully.

I think that I'm going to start using vtprint again, when I need to print a file located on a virtual machine, or over a connection that only offers SSH (such as a secured access machine).  With firewalled networks, I suspect that this program has new usefulness.  I'm also switching from (as many people have suggested) to iTerm2.

Sunday, February 22, 2015

IPv6 and IPv4 name resolution with Go

As part of a work-related project, I'm writing code that needs to resolve DNS names using Go, on illumos.

While doing this work, I noticed a very surprising thing.  When a host has both IPv6 and IPv4 addresses associated with a name (such as localhost), Go prefers to resolve to the IPv4 version of the name, unless one has asked specifically for v6 names.

This flies in the fact of existing practice on illumos & Solaris systems, where resolving a name tends to give an IPv6 result, assuming that any IPv6 address is plumbed on the system.  (And on modern installations, that is the default -- at least the loopback interface of ::1 is always plumbed by default.  And not only that, but services listening on that address will automatically serve up both v6 and v4 clients that connect on either ::1 or

The rationale for this logic is buried in the Go net/ipsock.go file, in comments for the function firstFavoriteAddr ():
    76			// We'll take any IP address, but since the dialing
    77			// code does not yet try multiple addresses
    78			// effectively, prefer to use an IPv4 address if
    79			// possible. This is especially relevant if localhost
    80			// resolves to [ipv6-localhost, ipv4-localhost]. Too
    81			// much code assumes localhost == ipv4-localhost.
This is a really surprising result.  If you want to get IPv6 names by default, with Go, you could use the net.ResolveIPAddr() (or ResolveTCPAddr() or ResolveUDPAddr()) functions with the network type of "ip6", "tcp6", or "udp6" first.  Then if that resolution fails, you can try the standard versions, or the v4 specific versions (doing the latter is probably slightly more efficient.)  Here's what that code looks like:
        name := "localhost"

        // First we try IPv6.  Note that we *hope* this fails if the host
        // stack does not actually support IPv6.
        err, ip := net.ResolveIP("ip6", name)
        if err != nil {
                // IPv6 not found, maybe IPv4?
                err, ip = net.ResolveIP("ip4", name)
However, this choice turns out to also be evil, because while ::1 often works locally as an IPv6 address and is functional, other addresses, for example, will resolve to IPv6 addresses which will not work unless you have a clean IPv6 path all the way through.  For example, the above gives me this for 2607:f8b0:4007:804::1010, but if I try to telnet to it, it won't work -- no route to host (of course, because I don't have an IPv6 path to the Internet, both my home gateway and my ISP are IPv4 only.)

Its kind of a sad that the Go people felt that they had to make this choice -- at some level it robs the choice from the administrator, and encourages the existing broken code to remain so.  I'm not sure what the other systems use, but at least on illumos, we have a stack that understands the situation, and resolves optimally for the given the situation of the user.  Sadly, Go shoves that intelligence aside, and uses its own overrides.

One moral of the story here is -- always use either explicit v4 or v6 addresses if you care, or ensure that your names resolve properly.