New implementation of printf

So I finally got tired of waiting for someone else to do a printf(1) replacement in illumos for the closed binary from Oracle. I had thought this would be a trivial thing to do via ksh93/libcmd using a symbolic link ala /usr/bin/alias.

Lo and behold, it wasn't! Why? Because ksh93 printf insists (like all ksh93 builtins) on having -- and - getopt style processing. This is fundamentally incompatible with legacy printf. (Why does it do this? So it can dump its builtin man page, e.g. printf --man, to the console. A feature I've railed against in the past.)

Here's what should happen:

% printf -v
-v% 

Here's what ksh93 does:

garrett@thinkpad:~$ printf -v                                                   
ksh93: printf: -v: unknown option
Usage: printf [ options ] format [string ...]

Now there is an argument to be made that a script which relies on the legacy behavior is fundamentally broken. But it doesn't matter -- the scripts are in the field (there are real examples of them), and the legacy behavior must be preserved. Breaking these legacy scripts just so that we can dump printf --version output is... silly. This is case where pragmatism wins over purity.

Rather than try to rip this out and fight with the ksh93 about "deviation from the upstream" (apparently the ksh93 folks view any changes we make in illumos or OpenSolaris as automatically toxic unless they originate from David Korn or Glenn Fowler), I've just gone ahead and implemented my own printf(1) on top of FreeBSD's. This will be the implementation in illumos.

I've added significantly to FreeBSD's code though. Specifically, I added handling of %n$ processing to get parameterized position handling. This is needed for internationalization -- it allows you to change the order of output as part of the output from something like gettext(1). (This is needed when you have to change word order to accommodate different natural language grammars.)

So my implementation is superior to FreeBSD's, and its superior to the legacy closed binary version. Why? Because rather than a half-hearted attempt at processing positional parameters, my version really handles these, including full support for the usual format specifiers. For example:

New open code:

garrett@thinkpad{4}> printf '%2$1d %1$s\n' one 2 three 4
2 one
4 three

Old closed code:

garrett@master{22}> printf '%2$1d %1$s\n' one 2 three 4
134511600 one

Clearly the old behavior is just plain wrong. For the record, ksh93 does the right thing here too. (Although somewhat older versions of ksh93 would dump core on this command line.)

My diffs (which also include style and lint fixes required for illumos) relative to FreeBSD are online. You can also review a webrev of the changes that I hope to integrate into illumos. The license remains BSD, so the various BSD operating systems (or even Oracle) are free to incorporate these improvements if they like.

illumos gets global

I just pushed a major set of changes:

8 libc locale work needs updated license files
223 libc needs multibyte locale support for collation
225 libc locale binary files should be in native byte order
309 populate initial locales for illumos

As a result, illumos has gained base support for some 157 different locales, spanning 67 languages and 116 different territories. This includes nearly all the major languages of the world -- missing are Serbian, Javanese, Farsi, Malaysian, Burmese, and some languages spoken in central and west Africa. (Some of these will be very easy for someone else to add... let me know if you want one of these and are willing to do the work.)

The support for these locales includes full POSIX compliant collating support, which was completely absent in illumos before this integration.

Also, included, is a new open source implementation of localedef(1). To my knowledge, this new implementation is the only non-GNU version of localedef that is fully open, and this version is more fully functional than the GNU version. (The GNU localedef lacks full support for collation data.)

Other notes: this is only the base support for these locales. This will for example give localized output from "date". There is quite a lot of additional effort required to fully localize an illumos system, including support for input methods, fonts, and message catalogs for all the various applications. However, with this base support, it makes doing that other work much more practical.

This integration adds nearly 2 million lines to illumos, although far and away the vast majority of it is in the form of data from Unicode and the CLDR (common locale data repository). The ability to import data directly from these sources is the new code that I've written, including a major overhaul of the underlying ctype and collation support in libc to properly support multibyte locales.

Its my belief that with this integration, one of the biggest feature gaps between illumos and Solaris is closed.

Emacs & Gnome Terminal Co-existence Resolved

For many years, I've been stuck with old xterm, because it was the only one that honored my Meta keys in the same way that GNU emacs did. I could never figure out how to make gnome-terminal work, which always bothered me somewhat. (Notably GNOME terminal has better Unicode support which has lately become important to me.)

I finally found a reference that helped me out. I understood that the problem was conflicting ideas about modifier keys; gnome-terminal uses Mod1, but Emacs uses Mod4. What I didn't know was something I found out here, namely that Emacs only uses Mod4 if it exists. So a better solution for me is to simply clear Mod4 altogether, and both programs happily honor Mod1. (This leaves xterm hosed, but if gnome-terminal works, then I don't need xterm anymore.)

My resulting .xmodmap looks like this:

remove Lock = Caps_Lock
keysym Caps_Lock = Control_L
add Control = Control_L
clear Mod4

This makes my PC keyboard behave sensibly. Alt is Meta. And Caps Lock is consigned to oblivion and the large key that used to have that function is now much more usefully assigned to Control.

I'm posting this here in case anyone else has struggled with this particular annoyance in the past. The clear Mod4 trick was the surprise ticket. (What I'd really like is a way to tell programs which Modifier is "really" the Meta key, given that the programs can't seem to agree on this. And with just one preference -- redefining the numerous bindings in emacs for each sequence, while possible, is not my idea of a fun thing to do.)

The other thing I'd like is a standard way in illumos/opensolaris to integrate .xmodmap. Linux/Ubuntu seems to detect my .xmodmap and handles it nicely.

Another ZFS departure

Jeff Bonwick is leaving Oracle.

This is a huge event, because Jeff has been one of the main innovators in operating system technology during his tenure at Sun. While you may know him best for ZFS, he's also the inventor of the slab allocator, which revolutionized memory management when it was created. (And now, pretty much every modern system uses some variation of the slab allocator.)

And he's not just an Oracle VP. Jeff has made integrations into Solaris' ZFS code base on an ongoing basis. This is a guy that has led with actual actions and innovation, backed by code, rather than some boffin who's risen to management and no longer contributes. At some level, he's the model for the kind of technologist I aspire to be.

With so many innovators leaving (and yes, there are other key players in flight), its going to be very interesting to see how Oracle is able to continue to be a thought leader in the OS technology that they've acquired.

One the one hand, its really a shame to see to much of the heart and soul of the Solaris engineer core slowly disintegrating.

On the other hand, I think illumos may be the place where Solaris innovation happens, more so than at Oracle, even sooner than I previously expected.

South/Central American opportunity

I just learned that a peer of mine is looking to add some escalation engineers in Latin America. Job requirements include excellent English, and the ability to deep dive into customer problems including kernel crash dump analysis and C coding ability. If this sounds interesting to you, please let me know.

Oracle/NetApp ZFS lawsuit dismissed

Others have no doubt already picked upon this, but here it is anyway:

http://www.h-online.com/open/news/item/NetApp-and-Oracle-lift-ZFS-patent-cloud-1076313.html

Hopefully this is good news for downstream ZFS consumers.

We're Hiring!

In case you didn't know, a number of companies are hiring illumos talent.

I know of an opening for a USB kernel engineer at one company.

I'm told Joyent is growing like crazy.

And Nexenta is hiring! In fact, here are some of the opportunities we have open at Nexenta:

• QA leads. We have two positions for folks with skills and knowledge to design and build, and run, automated testing of the operating system, with a particular focus on storage and networking. Expertise in NFS, CIFS, iSCSI, ZFS, and the surrounding areas would be highly useful. Good communication skills, shell scripting or perl skills, and an ability to work in the office in Mountain View, are all required. Previous QA leadership preferred.
• Support engineers. We need support engineers across the globe. People who can answer the phone, and triage problems. Solaris or UNIX experience, ZFS clue, good troubleshooting and triage skills, and excellent communication skills are necessary.
• Kernel software engineers. I need people with deep TCP/IP, SCSI, Storage, and Filesystems expertise. Solaris expertise highly preferred, but can substitute FreeBSD or Linux kernel expertise. Highly motivated self-driven super-stars only.
• Sustaining software engineers. Excellent troubleshooting and kernel expertise is required. Expertise in one or more of TCP/IP, SCSI, storage, and filesystems is preferable. Solaris expertise highly preferred.
• IT staff. We have one opening for a mid-level IT engineer. Must be able to deal with Solaris, Linux, Windows, phones, and cantankerous development staff.

I expect even more growth will occur here over time. A jobs board for illumos will be coming soon.

Squash-proof?

So everyone has heard me talk about the 800 lb. gorilla with respect to illumos.

One question I keep getting asked is, can the illumos project be "squashed" by this 800 lb. gorilla?

My stock answer had been "no". But I realized something today; I've been wrong.

The way illumos can be "killed" is if the corporate owner of Solaris were to do something to make illumos irrelevant. Like, say, opening Solaris back up (and in this case, I think they would probably need to go further open than they were before).

I'm not worried though. Even if that happens, illumos will have been a major success. But I really don't think it is going to happen.

illumos Interest Groups

So, I've been asked by several people who are involved with OpenSolaris User Groups around the world about illumos.

Given the clear demise of OpenSolaris, it seems to me at least, to be kind of silly to continue to meet using that name.

Some groups have reverted to pure Solaris usage. Which is fine for those groups that want to focus on Oracle products and want to come under the Oracle umbrella that it has for user groups.

For groups that are more interested in open technology, perhaps it is time to start up some "illumos interest groups" (IIGs)? (Calling them "User Groups" at this point seems rather premature... I think there are only a very few of us that are actually "using" illumos at this point.. but I hope that number to grow very much very soon. :-)

Btw, are there any folks interested in illumos in either Riverside County or North San Diego County? (California) I'd be interested in participating in an interest group if there was one that didn't require me to drive over an hour to get to.

OpenSolaris ARC is Dead

I had tried to dial in to ARC today, but no luck. But then someone else pointed out that we have not seen any ARC cases since the tap was turned off.

In fact, I posted a query about this to the opensolaris-arc mailing list today, and I got back an interesting automated reply:

This mailing list is no longer active and accepting posts. Mailing
list archives can be found at
http://mail.opensolaris.org/pipermail/opensolaris-arc/. You can check
http://mail.opensolaris.org/mailman/listinfo to find another list to
which to send your email.

So, OpenSolaris ARC is dead. This has ramifications that go beyond just ON. Because there are other consolidations that we were promised were going to continue to be developed in the open: JDS, X11, and the pkg-gate. If the decisions for these technologies are no longer being made openly, or even the opinions being made available, then this makes Oracle's promise to continue to work with the community on them seem hollow.

So, what's left for "OpenSolaris" as so named? There are some code drops still being made. How long will that keep up? Are they continuing to take contribution from external parties? (I don't work on those gates, so I don't really know.) I'd like to know if the other consolidations have shut down too. At least the key decisions relating to those consolidations seem to have moved behind closed doors.