Thursday, December 30, 2010

beadm provided in "C"

I know this is somewhat controversial for some people, but....

A member of my staff (Alex Stetsenko) has pushed an implementation of "beadm" that is in C. This is actually derived from an earlier C implementation we already had in tree, called "tbeadm", which we already had. So at some level, this consolidation of two different implementations into a single one. As part of this work, the tbeadm version was modernized and improved to provide i18n capabilities and to behave truly as a drop-in replacement for the python version.

As a result of this change, python is no longer needed at runtime by illumos for anything except IPS packaging. Sites and distributions which do not use IPS packaging (most distros don't, actually) no longer need to install python.

Tuesday, December 21, 2010

Any illumos fans near Corinth, MS?

I drove across country last weekend, and am in Corinth, MS. I'd be happy to go out for a beer and some chat if there are any illumos fans or OSUGs nearby this week.

Wednesday, December 15, 2010

I sed(1) so!

I just integrated a new sed(1) port for illumos. This is derived from FreeBSD, but it includes a fix for a race condition, and support for translated messages. (FreeBSD friends, please feel free to include these changes back -- I've not changed the original BSD license.)

The legacy sed is gone.

This new sed should work on all the old sed scripts, but there were a few tricky parts that changed -- if you relied on parsing the output of the "l" command, or on the fact that legacy sed only treated content as a byte stream rather than multibyte characters, you might be affected.

Also, I've run into at least one sed script which was malformed, but mistakenly accepted by old sed, but which the new version doesn't accept (but instead gives you a meaningful error message.)

The biggest features in the new sed code:

  1. support for -i and -I
  2. support for -E to enable EREs
  3. much more helpful error messages ("command garbled" was just not very specific)

Enjoy, and please report any problems in the illumos bug tracking system at Thanks!

Update: Note that sed -i requires an argument (the extension) unlike GNU sed where the argument is optional. We can fix that, although this would make us less compatible with FreeBSD sed. (Specifically, it would make it nigh impossible to specify an "extension" starting with a dash.) If someone cares passionately about this, they should file a bug and bring it up on the developer list -- I am happy either way.

Wednesday, December 8, 2010

Update on SATA Expanders

So we've done some more research, largely following up on work done by Richard Elling, and I have an update on the SAS/SATA expander problem. There is at least some good news here.

The problems that we've had in the past with these have centered around "reset storms", where a single reset expands into a great number of resets, and I/O throughput quickly diminishes to zero.

The problem is that when a reset occurs on an expander, it aborts any in-flight operations, and they fail. Unfortunately, the *way* in which they fail is to generate a generic "hardware error". The problem is that the sd(7d) driver's response to this is to ... issue another reset, in a futile effort to hopefully correct things.

Now the problem is that this behavior is also performed, by default, for media errors as well. E.g. if you have a disk that has a bad sector on it. Of course, if your disk is mostly idle, it won't be a problem. But if you have a lot of I/O going on, its going to result mostly in a melt-down.

There is good news though, because of the way LSI's drivers are designed.

The LSI mptsas driver at least (and I suspect mpt as well, though I don't have code to look at it) treats "bus-level" resets and "target-level" resets as the same. Both of them do a reset, which will of course reset the expander.

But we can disable the most pernicious reset in sd with the following line in sd.conf:


This will allow bus-wide resets to occur, but it will most specifically disable the reset in response to generic hardware and media errors. The relevant section of code in sd.c is this:

if ((un->un_reset_retry_count != 0) &&
(xp->xb_retry_count == un->un_reset_retry_count)) {
/* Do NOT do a RESET_ALL here: too intrusive. (4112858) */
if (un->un_f_allow_bus_device_reset == TRUE) {

boolean_t try_resetting_target = B_TRUE;

* We need to be able to handle specific ASC when we are
* handling a KEY_HARDWARE_ERROR. In particular
* taking the default action of resetting the target may
* not be the appropriate way to attempt recovery.
* Resetting a target because of a single LUN failure
* victimizes all LUNs on that target.
* This is true for the LSI arrays, if an LSI
* array controller returns an ASC of 0x84 (LUN Dead) we
* should trust it.

if (sense_key == KEY_HARDWARE_ERROR) {
switch (asc) {
case 0x84:
if (SD_IS_LSI(un)) {
try_resetting_target = B_FALSE;

if (try_resetting_target == B_TRUE) {
int reset_retval = 0;
if (un->un_f_lun_reset_enabled == TRUE) {
"error: issuing RESET_LUN\n");
reset_retval =
if (reset_retval == 0) {
"error: issuing RESET_TARGET\n");
(void) scsi_reset(SD_ADDRESS(un),

The savy folks here might notice that this is a wide setting, which is true. You can set it on a specific instance of sd, which requires more effort. There is also a better way to do this, by setting the reset_retry_count property to zero. However, setting the sd.conf property for that properly is considerably more complex, because of the byzantine syntax that sd uses to set up target-specific property values.

So, I still recommend avoiding these SATA expanders. But if you have no choice, then using this sd.conf tunable may be a reasonable workaround.

At the same time, I'm investigating the possibility of having this disabled by default for all of Nexenta's customers -- and possibly even in illumos. If you're a SCSI expert and have opinions on the matter, please let me know.

Sunday, December 5, 2010

Status Update on the illumos Foundation

I sent this out earlier today:

We are working with the Software Freedom Law Center & Eben Moglen on the creation of the illumos legal entity. Nexenta have enlisted the help of Damien Eastwood (one of the more prominent former Sun lawyers) to help drive this. Jason Yoho at Nexenta is driving this fairly hard as well, so there are now more people than just me pushing this forward as quickly as we can.

We have set a goal that the legal entity (an illumos foundation) should exist with legal presence before the year's end. I'm told that this is an achievable goal.

I'll have more updates on this soon, I expect, but the process is moving forward.

New Advocate - Albert Lee

I'm pleased to announce the addition of Albert Lee (, aka Triskelios on IRC) to the list of Advocates who can approve integrations into illumos. Albert has been doing a lot of excellent work on both illumos and OpenIndiana, and I'm happy to expand the set of advocates we have available to include such a diligent and talented individual.

The current list of Advocates for illumos-gate are:

  • Garrett D'Amore
  • Albert Lee
  • Rich Lowe
  • Gordon Ross

I'm hoping to expand the list to include more non-Nexenta-employees as well. If you're a contributor and would like to help out in this way, let me know. Typically becoming an advocate means you have earned the trust of the rest of the advocates by making several "good" integrations into illumos-gate (4-5 at least usually, although some credit is given for previous integration experience with ON at Sun/Oracle), and have a demonstrated level of thoroughness to help us ensure quality integrations.

Thanks, and again, congratulations and thank you to Albert.

Wednesday, December 1, 2010

New open source iprb(7D) driver

For a variety of byzantine reasons, the iprb driver has never been open sourced, even though everyone who's ever actually had anything to do with it agrees that it should be. (I blame the lawyers on this one...)

So I went ahead and reimplemented -- from scratch -- a new iprb driver. I'd certainly appreciate feedback on the code, which you can read in the webrev. I'm hoping to integrate this into illumos later this week.

zfs should not depend on python... and doesn't anymore

As of the recent integration of a colleague of mine, illumos now has a zfs command that does not depend on python at all.

changeset: 13246:fe5d6e0b0bce
tag: tip
user: Alexander Stetsenko
date: Wed Dec 01 02:30:25 2010 +0300
278 get rid zfs of python and pyzfs dependencies
Reviewed by:
Reviewed by:
Reviewed by:
Reviewed by:
Approved by:

The zfs command is now entirely a C program. This may make it more friendly for use in other environments or platforms. FreeBSD folks, you might want to incorporate this into your tree. If you do, I'd sure like to know.

Thursday, November 11, 2010

Job Opp @ Nexenta: Director of Sustaining/Certifications

We're looking for a Director of Engineering, to own sustaining (aka bug fixing) and hardware platform certifications (where partners provide a hardware platform and ask us to certify it for use with NexentaStor.)

The job qualifications:

  1. Must be local to the SF bay area (because the hardware lab is here)
  2. Must have strong communication skills
  3. Must be able to deal with stressful situations, and able to "manage" strong personalities
  4. Must have Solaris/OpenSolaris expertise... hands-on kernel work (crash dump analysis, coding, etc.)
  5. Desire experience with Storage protocols and products
  6. Desire expertise with x86 hardware
  7. Desire perl and/or python skills

To be clear, this will start out as a hands-on position, with a fast-paced startup environment. But the growth opportunities here are enormous. If you think you're up to this, please let me know.

Friday, November 5, 2010

New desktop image

Here's a sample of the new logo as a desktop image. I've not made this available publicly yet (mostly because I don't know how to capture this in a form that will include the gradient and post it where people can find it.) If someone with some gnome expertise on how to share this for others contacts me, I can work to make it available.

Wednesday, October 27, 2010

New illumos logo

Today at the OpenStorage Summit 2010, I unveiled the new illumos logo. We will be updating our branding, which also includes a new font, and other elements, over the next few weeks.

There were other updates on illumos that I covered in this talk. I think this was recorded, but I'm not sure right now where it was recorded and how to acces it. I'll be sure to share that when I find out.

Wednesday, October 13, 2010

CFV: web/HTML/graphics people

I have an urgent need to rennovate the illumos website. If you'd like to help the project out, and you have got both time and talent, please let me know. A major overhaul of the site is in order, and we need someone willing to dedicate some time on it. There may be some funds available for the right person, but to be clear, illumos can't afford the services of a professional design bureau.

New implementation of printf

So I finally got tired of waiting for someone else to do a printf(1) replacement in illumos for the closed binary from Oracle. I had thought this would be a trivial thing to do via ksh93/libcmd using a symbolic link ala /usr/bin/alias.

Lo and behold, it wasn't! Why? Because ksh93 printf insists (like all ksh93 builtins) on having -- and - getopt style processing. This is fundamentally incompatible with legacy printf. (Why does it do this? So it can dump its builtin man page, e.g. printf --man, to the console. A feature I've railed against in the past.)

Here's what should happen:

% printf -v

Here's what ksh93 does:

garrett@thinkpad:~$ printf -v
ksh93: printf: -v: unknown option
Usage: printf [ options ] format [string ...]

Now there is an argument to be made that a script which relies on the legacy behavior is fundamentally broken. But it doesn't matter -- the scripts are in the field (there are real examples of them), and the legacy behavior must be preserved. Breaking these legacy scripts just so that we can dump printf --version output is... silly. This is case where pragmatism wins over purity.

Rather than try to rip this out and fight with the ksh93 about "deviation from the upstream" (apparently the ksh93 folks view any changes we make in illumos or OpenSolaris as automatically toxic unless they originate from David Korn or Glenn Fowler), I've just gone ahead and implemented my own printf(1) on top of FreeBSD's. This will be the implementation in illumos.

I've added significantly to FreeBSD's code though. Specifically, I added handling of %n$ processing to get parameterized position handling. This is needed for internationalization -- it allows you to change the order of output as part of the output from something like gettext(1). (This is needed when you have to change word order to accommodate different natural language grammars.)

So my implementation is superior to FreeBSD's, and its superior to the legacy closed binary version. Why? Because rather than a half-hearted attempt at processing positional parameters, my version really handles these, including full support for the usual format specifiers. For example:

New open code:

garrett@thinkpad{4}> printf '%2$1d %1$s\n' one 2 three 4
2 one
4 three

Old closed code:

garrett@master{22}> printf '%2$1d %1$s\n' one 2 three 4
134511600 one

Clearly the old behavior is just plain wrong. For the record, ksh93 does the right thing here too. (Although somewhat older versions of ksh93 would dump core on this command line.)

My diffs (which also include style and lint fixes required for illumos) relative to FreeBSD are online. You can also review a webrev of the changes that I hope to integrate into illumos. The license remains BSD, so the various BSD operating systems (or even Oracle) are free to incorporate these improvements if they like.

Friday, October 8, 2010

illumos gets global

I just pushed a major set of changes:

8 libc locale work needs updated license files
223 libc needs multibyte locale support for collation
225 libc locale binary files should be in native byte order
309 populate initial locales for illumos

As a result, illumos has gained base support for some 157 different locales, spanning 67 languages and 116 different territories. This includes nearly all the major languages of the world -- missing are Serbian, Javanese, Farsi, Malaysian, Burmese, and some languages spoken in central and west Africa. (Some of these will be very easy for someone else to add... let me know if you want one of these and are willing to do the work.)

The support for these locales includes full POSIX compliant collating support, which was completely absent in illumos before this integration.

Also, included, is a new open source implementation of localedef(1). To my knowledge, this new implementation is the only non-GNU version of localedef that is fully open, and this version is more fully functional than the GNU version. (The GNU localedef lacks full support for collation data.)

Other notes: this is only the base support for these locales. This will for example give localized output from "date". There is quite a lot of additional effort required to fully localize an illumos system, including support for input methods, fonts, and message catalogs for all the various applications. However, with this base support, it makes doing that other work much more practical.

This integration adds nearly 2 million lines to illumos, although far and away the vast majority of it is in the form of data from Unicode and the CLDR (common locale data repository). The ability to import data directly from these sources is the new code that I've written, including a major overhaul of the underlying ctype and collation support in libc to properly support multibyte locales.

Its my belief that with this integration, one of the biggest feature gaps between illumos and Solaris is closed.

Sunday, October 3, 2010

Emacs & Gnome Terminal Co-existence Resolved

For many years, I've been stuck with old xterm, because it was the only one that honored my Meta keys in the same way that GNU emacs did. I could never figure out how to make gnome-terminal work, which always bothered me somewhat. (Notably GNOME terminal has better Unicode support which has lately become important to me.)

I finally found a reference that helped me out. I understood that the problem was conflicting ideas about modifier keys; gnome-terminal uses Mod1, but Emacs uses Mod4. What I didn't know was something I found out here, namely that Emacs only uses Mod4 if it exists. So a better solution for me is to simply clear Mod4 altogether, and both programs happily honor Mod1. (This leaves xterm hosed, but if gnome-terminal works, then I don't need xterm anymore.)

My resulting .xmodmap looks like this:

remove Lock = Caps_Lock
keysym Caps_Lock = Control_L
add Control = Control_L
clear Mod4

This makes my PC keyboard behave sensibly. Alt is Meta. And Caps Lock is consigned to oblivion and the large key that used to have that function is now much more usefully assigned to Control.

I'm posting this here in case anyone else has struggled with this particular annoyance in the past. The clear Mod4 trick was the surprise ticket. (What I'd really like is a way to tell programs which Modifier is "really" the Meta key, given that the programs can't seem to agree on this. And with just one preference -- redefining the numerous bindings in emacs for each sequence, while possible, is not my idea of a fun thing to do.)

The other thing I'd like is a standard way in illumos/opensolaris to integrate .xmodmap. Linux/Ubuntu seems to detect my .xmodmap and handles it nicely.

Tuesday, September 28, 2010

Another ZFS departure

Jeff Bonwick is leaving Oracle.

This is a huge event, because Jeff has been one of the main innovators in operating system technology during his tenure at Sun. While you may know him best for ZFS, he's also the inventor of the slab allocator, which revolutionized memory management when it was created. (And now, pretty much every modern system uses some variation of the slab allocator.)

And he's not just an Oracle VP. Jeff has made integrations into Solaris' ZFS code base on an ongoing basis. This is a guy that has led with actual actions and innovation, backed by code, rather than some boffin who's risen to management and no longer contributes. At some level, he's the model for the kind of technologist I aspire to be.

With so many innovators leaving (and yes, there are other key players in flight), its going to be very interesting to see how Oracle is able to continue to be a thought leader in the OS technology that they've acquired.

One the one hand, its really a shame to see to much of the heart and soul of the Solaris engineer core slowly disintegrating.

On the other hand, I think illumos may be the place where Solaris innovation happens, more so than at Oracle, even sooner than I previously expected.

Saturday, September 25, 2010

South/Central American opportunity

I just learned that a peer of mine is looking to add some escalation engineers in Latin America. Job requirements include excellent English, and the ability to deep dive into customer problems including kernel crash dump analysis and C coding ability. If this sounds interesting to you, please let me know.

Thursday, September 9, 2010

Tuesday, September 7, 2010

We're Hiring!

In case you didn't know, a number of companies are hiring illumos talent.

I know of an opening for a USB kernel engineer at one company.

I'm told Joyent is growing like crazy.

And Nexenta is hiring! In fact, here are some of the opportunities we have open at Nexenta:

  • QA leads. We have two positions for folks with skills and knowledge to design and build, and run, automated testing of the operating system, with a particular focus on storage and networking. Expertise in NFS, CIFS, iSCSI, ZFS, and the surrounding areas would be highly useful. Good communication skills, shell scripting or perl skills, and an ability to work in the office in Mountain View, are all required. Previous QA leadership preferred.
  • Support engineers. We need support engineers across the globe. People who can answer the phone, and triage problems. Solaris or UNIX experience, ZFS clue, good troubleshooting and triage skills, and excellent communication skills are necessary.
  • Kernel software engineers. I need people with deep TCP/IP, SCSI, Storage, and Filesystems expertise. Solaris expertise highly preferred, but can substitute FreeBSD or Linux kernel expertise. Highly motivated self-driven super-stars only.
  • Sustaining software engineers. Excellent troubleshooting and kernel expertise is required. Expertise in one or more of TCP/IP, SCSI, storage, and filesystems is preferable. Solaris expertise highly preferred.
  • IT staff. We have one opening for a mid-level IT engineer. Must be able to deal with Solaris, Linux, Windows, phones, and cantankerous development staff.
I expect even more growth will occur here over time. A jobs board for illumos will be coming soon.

Friday, September 3, 2010


So everyone has heard me talk about the 800 lb. gorilla with respect to illumos.

One question I keep getting asked is, can the illumos project be "squashed" by this 800 lb. gorilla?

My stock answer had been "no". But I realized something today; I've been wrong.

The way illumos can be "killed" is if the corporate owner of Solaris were to do something to make illumos irrelevant. Like, say, opening Solaris back up (and in this case, I think they would probably need to go further open than they were before).

I'm not worried though. Even if that happens, illumos will have been a major success. But I really don't think it is going to happen.

Wednesday, September 1, 2010

illumos Interest Groups

So, I've been asked by several people who are involved with OpenSolaris User Groups around the world about illumos.

Given the clear demise of OpenSolaris, it seems to me at least, to be kind of silly to continue to meet using that name.

Some groups have reverted to pure Solaris usage. Which is fine for those groups that want to focus on Oracle products and want to come under the Oracle umbrella that it has for user groups.

For groups that are more interested in open technology, perhaps it is time to start up some "illumos interest groups" (IIGs)? (Calling them "User Groups" at this point seems rather premature... I think there are only a very few of us that are actually "using" illumos at this point.. but I hope that number to grow very much very soon. :-)

Btw, are there any folks interested in illumos in either Riverside County or North San Diego County? (California) I'd be interested in participating in an interest group if there was one that didn't require me to drive over an hour to get to.

OpenSolaris ARC is Dead

I had tried to dial in to ARC today, but no luck. But then someone else pointed out that we have not seen any ARC cases since the tap was turned off.

In fact, I posted a query about this to the opensolaris-arc mailing list today, and I got back an interesting automated reply:

This mailing list is no longer active and accepting posts. Mailing
list archives can be found at You can check to find another list to
which to send your email.

So, OpenSolaris ARC is dead. This has ramifications that go beyond just ON. Because there are other consolidations that we were promised were going to continue to be developed in the open: JDS, X11, and the pkg-gate. If the decisions for these technologies are no longer being made openly, or even the opinions being made available, then this makes Oracle's promise to continue to work with the community on them seem hollow.

So, what's left for "OpenSolaris" as so named? There are some code drops still being made. How long will that keep up? Are they continuing to take contribution from external parties? (I don't work on those gates, so I don't really know.) I'd like to know if the other consolidations have shut down too. At least the key decisions relating to those consolidations seem to have moved behind closed doors.

Monday, August 23, 2010

OGB has dissolved today

The old OpenSolaris Governing Board has dissolved unanimously today.

The OpenSolaris governance is now in default, and returns to Oracle's hands.

For folks upset by this, let me remind them of Illumos. Its a sad note for OpenSolaris, but I think the reborn Illumos community will be better than the OpenSolaris community ever could be.

I do want to thank the (former) OGB members for their efforts, even if they did prove to be in vain.

Sunday, August 22, 2010

Why SAS->SATA is not such a great idea

So, we've had some "issue" reports relating to the mpt driver. In almost all cases, the results are related to situations where people are using SATA drives, and hooking them into SAS configurations.

Although the technology is supposed to work, and sometimes it works well, its a bad idea.

Let me elaborate:

  • SAS drives are generally subjected to more rigorous quality controls. This is the main reason why they cost more. (That and the market will pay more.)
  • SAS to SATA conversion technologies involve a significant level of protocol conversion. While the electricals may be the same, the protocols are quite different.
  • Such conversion technology is generally done in hardware, where only the hardware manufacturer has a chance of debugging problems when they occur.
  • Some of these hardware implementations remove debugging information that would be present in the SATA packet, and just supply "generic" undebuggable data in the SCSI (SAS) error return.
  • The conversion technology creates another potential point of failure.
  • Some of these hardware implementations won't be upgradeable, or at least not easily upgradeable, with software.
  • SATA drives won't have a SCSI GUID (ATA specs don't require it), and so the fabricated GUID (created by the SAS converter) may be different when you move the drive to a different chassis, potentially breaking things that rely on having a stable GUID for the drive.

Don't get me wrong. For many uses, SATA drives are great. They're great when you need low cost storage, and when you are connecting to a system that is purely SATA (such as to an AHCI controller), there is no reason to be concerned.

But building a system that relies upon complex protocol conversion in hardware, just adds another level of complexity. And complexity is evil. (KISS).

So if you want enterprise SAS storage, then go ahead and spring for the extra cost of drives that are natively SAS. Goofing around with the hybrid SAS/SATA options is just penny wise, and pound foolish.

But hey, its your data. I just know that I won't be putting my trusted data in a configuration that is effectively undebuggable.

(Note: the above is my own personal opinion, and should not be construed as an official statement from Nexenta.)

Aug 30, 2010: Update: At a significant account, I can say that we (meaning Nexenta) have verified that SAS/SATA expanders combined with high loads of ZFS activity have proven conclusively to be highly toxic. So, if you're designing an enterprise storage solution, please consider using SAS all the way to the disk drives, and just skip those cheaper SATA options. You may think SATA looks like a bargain, but when your array goes offline during ZFS scrub or resilver operations because the expander is choking on cache sync commands, you'll really wish you had spent the extra cash up front. Really.


Look, I really, really wanted to avoid entering the packaging debate. I mean, its an emotional decision, right?

Well, its supposed to be.

Except that I've spent nearly an entire day trying to figure out how to onu the latest illumos gate (which includes Rich Lowe's b147 merged in). I have gate changes that I desperately need to test in the context of a full install. (Well, I could say "screw it", and just test the bits in place -- which I've already done, but that's hardly a complete test.) I can't test them. Because I can't figure out how to use the packaging system to install them. And neither can our resident IPS expert, Rich Lowe.

This is no longer an emotional decision for me. Yeah, there are a lot of "emotional" things not to like about IPS. (It forces a dependency upon Python; its still immature; it seems to fail if you are disconnected from the network; it doesn't seem possible to build and install "just" a single package; apparently there are a lot of magic incantations that nobody outside of the IPS developers really understands; etc.) I was willing to set aside all those "emotional" responses and use IPS, if it worked. If for no other reason than the fact that it did away with BFU I have been willing to give it my best effort. But the latest situation has left me dead in the water, and apparently NO ONE can help me.

Look, I'm not a complete moron. (Well, maybe you disagree with me, but this is my blog.)

I should be able to make this work. If I cannot, then what kind of barrier is this going to create for participation from other people? Is Rich Lowe going to hold the hands of everyone else to get past these issues?

What happens the next time the pkg folks introduce another flag day?

This is unacceptable.

I'd like to hear other solutions. At the moment, I'm very very seriously considering gutting the IPS build requirements and having illumos go back to building SVR4 packages natively, using a tool to convert IPS meta data. (So meta data would be IPS, but binary deliverable would be auto-generated SVR4 packages.)

The current situation reminds me of Linus' comments about CVS. I feel the same way about IPS right now. I'm very angry ... the tools that are supposed to facilitate development have caused it to cease for me. If the only way for me to move forward is to reinvent SVR4 build systems, then that's what I'll do.

IPS is a failed science experiment. I don't see how it is going to get widespread adoption from anyone (ISVs or otherwise) with it as it stands today.

Flames to /dev/null. Let me know if you have a solution though.

Update: Rich was finally able to get me to the point of working. Although I can't ever downgrade. After what I just went through, I never want to. I'm really terrified that nobody really understands the steps it took to get me to a working state, and I am unwilling to force others to go through the same nightmare. So I'm still made at IPS, and I still think we need to unhitch the illumos cart from it.

Thursday, August 19, 2010

The Tap Is Turned Off

A little birdie told me that the last update to Oracles hg repository for ON was this one:

changeset: 13149:b23a4dab3d50
tag: tip
user: Sukumar Swaminathan
date: Wed Aug 18 15:52:48 2010 -0600
6973228 Cannot download firmware 2.103.x.x on Emulex FCoE HBAs
6960289 fiber side of emulex cna does not connect to the storage
6950462 Emulex HBA permanently DESTROYED, if the firmware upgrade is interrupted
6964513 COMSTAR - Emulex LP9002 fail to return a SCSI Inquiry correctly to a VMware 4 Initiator

From here on out, Illumos and Oracle Solaris diverge. The funny thing is, based on the calls I've had today, I could hardly be more optimistic about the future of illumos and the code base that was formerly called Solaris. Even more talent is getting behind this effort every day.

I'm very very excited... frankly Oracle shutting down the tap just really opened up the opportunity for us to really start innovating, in ways that I would have been loathe to do if we were still trying to maintain a very closely aligned source tree.

I think its entirely possible that Oracle may wind up viewing Illumos as the upstream rather than the reverse!

More milestones...

Illumos milestones reached today.

a) I pushed a working tr, and was able to build illumos on a system running illumos. This is the first time this has been possible.

b) Richlowe pushed a merge to build 147. There are probably consequences for developers (more updates required for bits that are not part of ON) -- stay tuned for updates about that.

All in all, things are moving quickly.

Tuesday, August 17, 2010

Presenting Illumos at SVOSUG

I'm pleased to announce that I'll be giving a brief talk at this month's SVOSUG meeting, Thursday Aug 26, at 6:45 pm in Mountain View. It will cover Illumos, and I will be joined by a colleague who will talk a bit more about Nexenta as well. If you're in the Bay Area at that time, it would be great to have a chance to meet.

I expect there will be some (probably significant) consumption of alcoholic beverages after the meeting, at an as yet undetermined location.

Monday, August 16, 2010

More new stuff...

I've been pretty busy with Illumos lately, but last week I took a few days off for family time.

One of the things I did was take my son (9 years old) out to the Kern River to try some whitewater kayaking. This was his first time on moving water, and it amazed me how quickly he picked up basic concepts. He was doing ferries, peel outs, and eddy turns like a champ after about 20-30 minutes. Amazing. He didn't even swim his first day -- he elected to stay in his boat (actually trying to do a roll) until I could give him an Eskimo rescue. (His only swim that day was when he got flipped by one of the holes in Riverside Park.)

He did get a good swim on the second day, when we were working on ferries though the much faster swift water running at the bottom of Ewings rapid. His first ferry was quite high into the rapid itself, and clean, but the second time he went for a swim. Came up happy and smiling, ready to try again if we had had the time.

I wish I had some pictures.

Guess I'm gonna have to get the kid a boat soon. He wants to try kayak surfing with me, and he really wants to learn to roll. Too bad there are no vendors that offer whitewater boats small enough for kids in southern California. We probably won't make it to Kernville again until next season. :-(

Sunday, August 15, 2010

Milestone Commit for Illumos

Richard Lowe has just made a milestone change to the Illumos repository.

Its a milestone for two reasons:

a) It is the first commit from another developer other than me. (Other developers have code in progress, but not yet ready to commit, but soon!) This also makes it truly a community project, since Rich has no affiliation with me other than as a participant in the Illumos project.

b) It eliminates the dependency on the Oracle "extra" repository, which required folks to get a certificate to access non-redistributable code in order to build illumos.

Thank you very much Rich. I'm looking forward to more integrations from developers soon!

Friday, August 13, 2010

The Hand May Be Forced

Well, as you may have read, Oracle has decided that at some point very soon, we're going to lose normal regular access to the source code for OS/Net. (I.e. the Solaris kernel and supporting programs.)

While I would have vastly preferred for Illumos to have a cooperative and collaborative relationship with Oracle, it appears that Oracle doesn't value this. In fact, the exact words were from the management at Oracle were as follows:

Solaris is not something we outsource to others, it is not the assembly of someone else’s technology, and it is not a sustaining-only product.

While I understand the need to own the technology, there are few things that could be stated that show a stronger NIH attitude than this. Its unlikely that there will ever be a way for Oracle and the greater community to have a collaborative relationship.

This is a dark day for OpenSolaris -- its effectively dead now. (Its parent, Solaris, lives on however.)

How unfortunate.

For Oracle that is.

Because from the fertile ashes of the dead springs forth new life bringing hope and light in the form of Illumos.

Illumos has garnered the support of some of the top minds in the industry; already the list of names of Solaris contributors and potential contributors that have already publicly committed to supporting this project is extensive. Many of the names are famous, people like Bryan Cantrill. Oracle's actions and inaction have actually made this possible.

I can also say, the list goes even further -- considerably so. I have had private conversations with quite a few other people who have quietly committed to involvement. Some of the names are very surprising, and I hope that they will soon be in a position to announce their involvement for themselves. These are people that are big name contributors; folks who have made very large numbers of code commits to Solaris -- some of the deepest and most "challenging" parts of Solaris, too.

The upshot of this is that the future for Illumos is surprisingly bright. Rather than a dependency on the good will of one corporate sponsor with dubious intentions, the project will have the diverse backing of some of the most well-known innovators (and their employers) from the OpenSolaris -- nay, Open Source -- community.

So, by their actions here, Oracle may be forcing Illumos to "fork", which was always a prospect, even if not one I cherished. But with the backing of the innovators I know who are with us, I think we have a chance to actually be the premiere foundation for SunOS derived technology. Oracle may be investing more into Solaris, but if the best and brightest have left for greener pastures and are contributing to Illumos, then I think we'll have the "best" investments in the base. Following Oracle's lead when the brightest minds have already left looks less and less desirable by the moment. (And to be fair, there are still many bright folks within the Solaris organization at Oracle. But the balance is changing, and changing in favor of Illumos and the open development community.)

Oracle Solaris will not be the only source for this technology, and now it appears it may not even be the best source for this technology.

I once said I never intended for Illumos to compete with Solaris. That was true, but if Oracle forces the issue, then even despite their vast economic resources, I say, "Bring it!"

Tuesday, August 3, 2010

Illumos Announcement

Today we announced the Illumos Project. I think the call I gave on it had a lot more information than I want to write here, and there are now quite a number of blog postings from other more recognizable names than my own. I'm thrilled by the excitement here!

Friday, July 30, 2010


A number of the community leaders from the OpenSolaris community have been working quietly together on a new effort called Illumos, and we're just about ready to fully disclose our work to, and invite the general participation of, the general public.

We believe that everyone who is interested in OpenSolaris should be interested in what we have to say, and so we invite the entire OpenSolaris community to join us for a presentation on at 1PM EDT on August 3, 2010.

You can find out the full details of how to listen in to our conference, or attend in person (we will be announcing from New York City) by visiting (The final details shall be posted there not later than 1PM EDT Aug 1, 2010.)

We look forward to seeing you there!

- Garrett D'Amore & the rest of the Illumos Cast

Thursday, July 15, 2010

Please Be Patient

With all the ruckus surrounding Oracle's apparent abandonment of the community, and OGB's stated intention to suicide, the community uproar has been crazy.

Without giving any details, let me say that a few of us are quietly but diligently working on solutions to the critical problems, and I expect we'll be able to talk much more freely about the solutions we will be offering in early August, which is coming up very soon now.

So, I'm going to humbly ask folks to be patient -- hold your comments, complaints, and flames about Oracle and OpenSolaris and OGB in check please. If you can wait just a little bit longer, then I believe we'll be able to offer a more constructive outlet for your frustration and energies.


Wednesday, July 14, 2010

In NYC for DebConf10

I'll be attending DebConf10 (the Debian developer's conference) in NYC this year. Nexenta will be presenting information about our distribution. Its my hope that we can use this to generate more interest in OpenSolaris technology. If you're in NYC, and want to meet during the first week of August, let me know!

Wednesday, July 7, 2010

ZFS disk monitoring...

So I've posted this on zfs-discuss at opensolaris dot org, but its been suggested I mention it here too.

It turns out that the ZFS/FMA integration doesn't pick up on drive removals for most disk devices until the filesystem attempts to perform some I/O to the drive. This is rather unfortunate, because if a file system is not busy, you might suffer a loss of redundancy and not find out about it until too late.

It also means that you won't know about failures of hot spare devices until you need to put them into service, since by definition they are idle. (Note: as an exception running periodic scrubs should detect this too, although scrubs are highly intrusive to the overall I/O load on the system and probably should not be performed too often as a result.)

I'm told the Oracle 7000 series appliances have a solution for this problem, but of course the source for that is not in OpenSolaris. (Apparently there are quite a few differences in the core OS between the 7000 series and vanilla OpenSolaris -- unfortunately we can't know because -- unlike with NexentaStor -- we don't have access to the kernel source tree!)

This is not good for folks who use ZFS with ordinary Solaris 10 or OpenSolaris... or with derivatives such as NexentaStor.

To address that problem, I've developed a some code called "zfs-monitor" that periodically monitors the health of any physical vdev (disk) that is part of a ZFS pool (hot spare, log, or real device). This code is implemented as an FMA module. When a disk goes offline, zfs-monitor detects it, and triggers an FMA event, which allows ZFS to do the right thing. This means if a disk goes away, even if it isn't in use, whatever action is appropriate will be performed. (Logged in FMA fault logs, and if appropriate, a hot spare will be recruited to replace the failed or offline device.)

This code is part of NexentaStor 3.0.3. As there are some semantic differences of opinion (what constitutes device failure versus intentional removal by an administrator), the code is unlikely to be pushed into ON without further change. (At the same time, I've fixed a different problem in the ZFS FMRI parsing code, and I've submitted a request to get that fix integrated -- but I've not heard back from anyone at Oracle who is willing to sponsor the change yet.)

I'm happy to share the code for zfs-monitor to anyone who requests it. (In fact, you can examine the code in our open Mercurial repository directly!) Note that for it to work properly, you also will need the fix for the ZFS FMRI parsing bug just mentioned.

At Nexenta, we're committed to innovating and improving upon the great foundation of ZFS and OpenSolaris, and to the reasonable extent possible, we want to share those innovations with the greater OpenSolaris community. Hopefully changes like this demonstrate this commitment in a tangible fashion.

Monday, June 28, 2010

Looking for CIFS/AD expertise

(I know its probably questionable using my blog for this, but I thought I'd post it here anyway. My apologies if anyone finds this offensive. I'll keep it brief in any case.)

I'm looking for a high-caliber developer, preferably with some kernel and/or OpenSolaris expertise, who's also got extensive knowledge of ActiveDirectory and CIFS. If that's you, or you know someone who fits that description, please contact me -- garrett at nexenta dot com. (No recruiters or agents please.)

Tuesday, June 15, 2010

skype for Solaris

So I'm irked, really irked.

If we had Skype support for Solaris, I could probably ditch this half baked mess of Linux hosts running VMware guests with OpenSolaris and Nexenta. I want just a single host OS for my development box.

Right now the single biggest barrier to running OpenSolaris on my desktop for my job at Nexenta is Skype. But this is silly, because Skype works in Linux, and the APIs should basically be compatible. Especially with the OSS layer that we already have in OpenSolaris these days via Boomer.

If someone at Skype sees this (good luck trying to find a contact on their web site!), and wants to work with me on it, I'd be happy to help them work through the issues of getting a native Skype port.

If anyone who has an "in" at Skype reads this post, please forward it to your in at Skype.

If any folks are paying for business services from Skype, feel free to let them know you want a Solaris client, and there is an expert on the Solaris audio stack waiting to help.


(On a side note, I'd also like to have VMware on Solaris as well. Yeah, I know about VirtualBox, but I need support VMware for clients, and it would be a heck of a lot easier if I could just host VMware guests on my development head.)

LDRS 29, very cool

So, this past weekend my son and I went to LDRS 29, which is the event for the national high-powered rocketry club, Tripoli. We were there only one day and one night, but here were some cool highlights from Saturday:
  • Mass squat launch -- Timothy's Squat with an Aerotech G-67 redline motor flew very nicely, if a bit late off the pad. 28 other rocketeers had their rockets launch at roughly the same time.
  • Many wild squats. With the $29 specials from WhatsUpHobbies, lots of people were flying very unstable Squat rockets with I-140 skidmark engines. This configuration needs nose weight, as we found out for ourselves when we flew Timothy's with the same engine.
  • Four half-scale Patriots launched in 3 second intervals from a "box" launch vehicle -- much like a real Patriot. Very, very cool.
  • Drag race of six or seven N-impulse rockets. These are big rockets, lots of power.
  • Drag race between a number of very detailed rockets. There was a CATO about 20 feet off the pad, and unfortunately several other rockets were destroyed on ascent by the CATO. Cool to watch, but glad it wasn't one of my rockets.
  • Full scale O-impulse Patriot launch (and unfortunate catastrophic failure near the end of boost, flaming bits falling down all over the range.)
  • My own J-350W powered LOC-IV (with significant modifications in nose weight and fiberglass reinforced fins.) This was my Level 2 cert flight and it went brilliantly. (So I'm certified to fly level 2 high powered rockets - impulse up to and including L power.)
  • Our Big Daddy Estes rocket (typically D or E power) launch with an F-32 Blue Thunder engine -- believe it or not this was one of the highlights for me of the day. As the launch control officer proclaimed -- "a little too much power for the rocket, but we like that!"
  • Flying a drag race between two D-21 powered 18mm rockets. Both were lost, and later found damaged.
  • A number of very cool night launches -- lots of creativity here on the part of the rocketeers.
  • Discover channel. This was a mixed bag... it was cool that they were there, but they did interfere with launch schedules quite a bit. Still, I think we'll be part of the ultimate show, which is supposed to air July 5. Looking forward to watching that.
There was one significant accident, involving an extremely high powered rocket on the far pads. A couple of people were unfortunately badly burned, and had to be medevaced, and our wishes go with them for their recovery.

As for Timothy and I, we're hooked. We'll be going to the November RocStock event as well, provided we can make the schedule work out.

Press release

Noticed this press release got posted to the Nexenta web site. /me preens. :-)

Friday, June 4, 2010

O_SYNC behavior not honored

UPDATE (6/21/2010): This problem is apparently solved in b142. Probably other builds as well. But I was unable to reproduce this problem with real hardware on b142.

Note that VMware does not honor cache flushing, so VMware (and possibly other v12n users) will potentially still see this issue.

So, it turns out that ZFS in recent (somewhere after build 134 apparently) builds has a critical bug ... O_SYNC writes are not really synchronous. This leads to potential data loss.

I've not yet figured out which change introduced the bug, but I hope to work on it next week.

In the meantime, I would strongly discourage use of post-134 binaries for anything where data integrity is important.

I've filed a P1 bug with Oracle for this issue. I'll be trying to nail it down further next week; if I'm able to fix it before Oracle can, I'll offer up my fix.

I'll post the CR number when I receive the number back.

I imagine that this bug, which is trivially reproducible, will be getting top priority from the ZFS engineers next week.

UPDATE: CR number is 6958848

The link to access it isn't available yet.

Great Falcon-9 Launch!

SpaceX, one of our greatest hopes for a commercial manned space program, has achieved a huge milestone with the successful maiden launch of Falcon-9 with a Dragon capsule today. This is the craft that may one day soon be used for ISS resupply, and perhaps even crew transport.

Even as Obama shuts down the US governments manned space program, the commercial sector is picking it up. This is a momentous day.

Congratulations to Elon Musk and the rest of the team at SpaceX!

Wednesday, June 2, 2010

audioens in VMware...

So, we have not had audio in OpenSolaris within VMware since... well, ever.

I've been doing some investigation. I'm seeing a situation where the VMware emulated audioens device behaves rather differently from the real hardware.

For one, it seems to insist on using real interrupts. In particular, the sample count registers do not appear to be updated unless one receives and acknowledges an interrupt. (By toggling the interrupt enable bit.) This means that this virtualized device will never be able to run "interrupt free" like the other audio devices (or real audio hardware).

For another, it appears that the audio device has some weird dependency on the relationship between the size of the audio buffer, and the interrupt rate (the number of samples at which to interrupt). Using different values gives, strange results.

Finally, I cannot, for the life of me, figure out how to cause the device to actually trigger an interrupt. I've been able to make some progress by simulating a soft interrupt at 100Hz, which is how the interrupt free framework works anyway, but from what I can tell, nothing is causing a real interrupt to be delivered. This is really strange. (Without this functionality, I am able to process audio at a reasonable rate, but it still stutters, and is not really suitable for real-world use.)

My guess is that the virtual device has some weird dependencies that we don't know about. For example, while the hardware spec identifies registers as being 8, 16, or 32 bits wide, and we use those at the right bit widths, other FOSS drivers all seem to just blithely use 32-bit wide accesses. Is there a hidden dependency here?

If any reader from VMware is seeing this, and can help me understand the behavior of the simulated device, I'd appreciate it. I'd like to make audio work in this environment, if possible. I'm pretty close, I think.

Actually, it seems kind of crazy that these environments emulate such complex audio hardware. (For example sophisticated sample rate conversion hardware.) Much better, I think, would be a simple paravirtualization driver that just exposes a simple buffer and some control functions. If someone at VMware wants to work on that as a solution, I'd be happy to help with Solaris support for it. Since these things run isochronously, and chew up a fair bit of cpu when they run, such a solution would probably be quite useful. (For example, its silly to perform multiple sample rate conversions in software... instead we could express native sample rates via a PV driver to the guest, ensuring only a single SRC operation is performed appropriately in the guest operating system.)

Tuesday, June 1, 2010

Well *That* Didn't Work Out So Well

You may recall my recent blog post about Windows 7 being surprisingly usable.

Well, I have to recant here.

I used Windows 7 for about a week and half. While it *worked*, it was a pleasure to use. But after three BSODs in just that week and half, I have abandoned it. I'm now running Ubuntu. (Why not OpenSolaris? Because I need the ability to host VMware and Skype, and I can't do that natively on OpenSolaris -- yet.)

Sure, I could have called up support -- but Microsoft support is provided by my computer manufacturer, and I didn't feel like spending 3 hours on the phone dealing with tech support while they tried to triage my problem. In the end, it was simply faster and easier for me to reinstall with Linux, even allowing for the time it took to download the media.

Sure, the problem might have been my virtualization software, or maybe it was a shoddy audio driver, or maybe it was brokenness in my graphics driver, or maybe it was the 3rd party antivirus software (which begs the question-- why doesn't Microsoft ship with builtin malware protection -- you'd think given all the heat that they've taken over this that they would have figured out that they *need* a solution here that doesn't involve 3rd parties...)

The "automatic solution finder" that Windows 7 ships was completely unhelpful, it didn't find any links. Google was not much help either... with everything to buggy hardware, drivers, and even overtemp problems being cited as root causes.

I'm sure that tech support would have had me running around in circles trying to solve the IRQL_NOT_LESS_OR_EQUAL blue screen. (I'm guessing, from my kernel expertise, that this is probably an assertion fault somewhere that an IRQ level is set unexpectedly high or low -- exactly the kind of problem I know how to fix in Solaris.) Probably plugging and unplugging hardware, unloading and reinstalling drivers and maybe other software, and generally burning an unmentionable amount of my precious time. Especially given that the hardware tech support I'd have been routed to was unlikely to have any real software clue (which is where I think this problem was most likely located.)

Again, faster and simpler to just dump the busted OS, and load something else.

And, with Linux (or any other FOSS), I have at least a fighting chance of trying to debug the problem myself. Sure, my kernel-fu is substantially higher than average joe home user, so my leanings are more towards something I can troubleshoot myself. But, I will say this, so far I've not had a panic ("oops" in Linux parlance) yet in the past four days that I've been running Ubuntu LTS 10.04 (even though I'm running the "not recommended for general desktop use" 64-bit edition.)

Microsoft, if you see this post, I hope you'll learn something from this.

Friday, May 21, 2010

Last Day

So, today's my last day as a Sun^WOracle employee.

While I'm excited to be starting at Nexenta, I want to reiterate what I've already said, which is that I've really enjoyed working with the great folks at S'noracle, and that they made this decision to leave quite difficult. Its been quite a ride over the years, and its been fun and exciting. Thanks everyone!

Of course, my old e-mail address(es) at Oracle won't reach me after about noon today.

To reach me for matters pertaining to OpenSolaris, will continue to work. For matters pertaining to my new employer, Nexenta, you can use My personal e-mail address of remains unchanged. Now please standby while I go reinforce the spam filters...

New Computer

As part of the process of changing employers, I needed to get a new computer for the new job (and return the old desktop to Oracle.)

I wound up picking this one... I didn't seem to be able to build it any cheaper (as of the date of this post) myself. And guess what... someone goofed! Instead of the 3 GHz Core i7 950, it came with a 3.2 GHz Core i7 960. Bonus! (Other goofs relative to the ad: the system has 9 GB -- but that's spelled out in the details, comes with a black aluminum chassis, and ships with a cheap logitech keyboard.)

I'm still using the stock load of Windows 7, and I'm both surprised (and maybe a bit embarrassed) to admit that the Windows environment (especially when replacing IE with Chrome) is actually quite nice -- fast and usable. Maybe running this environment (and running OpenSolaris in a VM) might not be so bad after all! (Ok, I'll go find some soap to wash my mouth out for blaspheming....) If I do this, besides being able to use Skype for work, I'll be able to use my Phoenix RC flight simulator without having to resort to borrowing the wife's computer...

Engines arrived for Squat yesterday

The Squat is a 4" diameter short high power rocket with a 54mm engine mount. My engines, 54mm hardware (including the higher end Aeropack retainer), and 38 mm adapter arrived from What's Up Hobbies yesterday. Timothy's going to fly it at the LDRS mass launch on a G67 redline -- this will be his first reloadable engine. Later that day I'll fly it on an I140 skidmark, which represents both my first 54mm engine, and my first Caeseroni engine.

Timothy and I put the rocket together last night; I must say, the higher end metal hardware and thicker fins on this rocket are definitely a step up even from the LOC IV I flew previously on my Level 1 flight (go to about 1:30 in the video link -- I haven't figured out how to edit the video file yet).

I also received the propellant for the J350W, which I'll be flying in my LOC IV as part of my Level 2 certification attempt. OpenRocket says the LOC IV will be approaching 700 mph with this particular engine! Guess I will be glassing the fins on it to help strengthen them for transonic speeds. (I'm open to alternative suggestions from the experts, as well.)

LDRS is going to be fun, indeed!

Monday, May 10, 2010

Greener pastures

I've recently made a major decision -- I'll be leaving Oracle. My last day as an Oracle employee will be on May 21, 2010. Leaving such a great group of people at Sun will be difficult indeed.

However, I hope to be able to continue as a significant contributor to the OpenSolaris community, as I'll be joining the team at Nexenta. At Nexenta, my responsibility will be to lead a group of engineers working on the OpenSolaris kernel. As such, I'm excited that I'll be able to continue to work on finest operating system kernel on the planet, and I look forward to further collaboration with some of the best software engineers on the planet.

My first day at Nexenta will be on May 24, 2010.

Wednesday, April 14, 2010

Going up to SF bay area

Its been a while since I've been to the Bay Area. I'm going up for two days, which is a shade longer than I usually go for. Part of the reason is to make sure I meet with folks in the Bay Area that I otherwise don't see. I'll be up Thursday and Friday April 29 and 30 -- and I expect I'll be at MPK most of that time. Anyone who wants to chat, please let me know.

Wednesday, March 17, 2010

audiocmihd driver (Asus Xonar cards)

Some people have been asking me about this driver. (Asus Xonar cards are fairly high-end high definition cards using the CMI 8788 chip.)

I've finally gotten the code reasonably cleaned up, and converted to my interrupt free audio framework.

I'll probably start a case to get this integrated into late b137, or b138. Mostly its just running a bunch of tests at this point.

One problem I have is that I only have Xonar DX1 cards. (PCI.)

If someone is able to help me qualify the driver with build 137 (or a nightly build) of ON, please let me know. The more I can get this driver tested, the sooner I can get it integrated into OpenSolaris.

Tuesday, March 16, 2010

Interrupt Free Audio

Today I integrated "interrupt free audio". This set of changes, including some other changes, represents a substantial simplification in the DDI for audio drivers.

The typical audio driver no longer needs to worry about interrupt handlers. On average, about 300 lines of code (or about 10-20% of complexity for typical drivers) was removed from each audio driver.

Furthermore, many audio drivers (for example audio810) are able to run completely lock free, since the audio framework provides synchronization for certain operations. (Operations against each audio engine are synchronized, operations against audio controls are synchronized as a whole, and everything is synchronized against suspend/resume functions.)

Even better, these changes enable some new advanced features that will be used for Sun Ray, virtualization, and hotplug support in the future.

Oh yeah, and since the asynchronous processing now happens as part of the regular timer interrupt, it means that system CPUs can remain in deeper C states for longer, even while playing audio. So, we should have an improvement on system power consumption (admittedly I've not measured this.)

There will be more stuff related to audio in the future, stay tuned.

"Legislative Sleight of Hand"

I normally have avoided using my blog as a soapbox for my political beliefs. However, I simply cannot remain silent on recent events in the House of Representatives (United States for foreign readers.

No matter what your position is on the health care reforms under consideration, everyone should agree that the reforms are sweeping; perhaps some of the most significant legislation that will affect nearly every American we've seen in quite some time.

House Democratic leadership, knowing that the measure is unpopular with many voters (and hence House Democrats may be unlikely to "vote the party line" to avoid a backlash in their constituencies) are planning a move that is even more offensive than "reconciliation".

While I'm a Republican, and generally opposed to nationalization of 1/6th of our economy, I find far more offensive that the House leadership (particularly Ms. Pelosi) would consider a move that so boldly disenfranchises the people of this nation.

This is a crime, if not against the law, then certainly against the spirit of democracy upon which our country is founded. If health care reform is to be passed, then it should be done with a regular vote where the politicians who vote for it are required to be accountable for those votes (and vice versa, by the way).

If it passes without such a vote, then it will go down as one of the greatest failures of "representative democracy" in history.

Monday, March 15, 2010

Why We Need a Human Spaceflight

Space aficionados may be aware that President Obama has canceled the previous administration's "Vision for Space Exploration", which consisted of the Constellation program including Ares I, Ares V, and Orion. This has been fairly well covered in the mainstream media.

Critics of the Constellation program raise some significant and relevant objections to the Constellation program.

However, I strongly believe that as a nation, we need a national space program that includes human spaceflight beyond low Earth orbit. The cancellation of Constellation, while perhaps with good cause, has left our national space program with a vacuum -- the lack of a heavy lift vehicle, and lack of any vision, would effectively constrain human exploration to LEO for a generation. Furthermore, it significantly constrains the kinds of activities that we can perform in LEO.

Its my belief that this is short-sighted in the extreme.

We need a space program that includes vehicles with the ability to loft large payloads into orbit. Projects like the International Space Station, and further commercialization of space, are only possible with the ability to loft a significant payload into orbit.

We also need to plan for human exploration beyond our front porch. While many people argue that sending robotic explorers is less risky, and far cheaper, the idea that we can or should abdicate all future space endeavors to robotic missions is actually offensive.

Robots won't inspire a generation of students to continue to excel at math and science. Robots can't stand in as national heroes. And robots alone won't help develop the enthusiasm required for the general public to continue to want to invest in space and space technologies. Robotic exploration is mostly a solved problem -- many new technologies that are necessary for human space travel will simply not be invented or invested in, without the "problems" to solve that are involved in human space exploration.

I'm from a generation of kids who viewed astronauts as near personal heroes; I dreamed, and still dream, of being able to see our planet from space itself one day. I dream of the days when human kind steps beyond just Earth, and has outposts on the Moon, Mars, the asteroids, and perhaps other interesting places in the solar system. And someday beyond.

My own son dreams someday of being an astronaut and visiting Mars. Unfunded as it was, at least VSE allowed a glimmer of such a hope. Obama has killed that hope, and maybe the dreams and hopes of thousands or millions of other like minded kids.

Fortunately, there is a proposal that would revive these dreams, and allow us to retain a national heavy lift capability, retain a lot of the knowledge and expertise that we acquired with the successful STS (space shuttle) program (even reusing a significant amount of the materials and technology), and allow for a "way forward" that would allow us to get beyond LEO and go to interesting places elsewhere in the solar system. The DIRECT v3.0 proposal is IMO the best way forward; it allows us to have our cake and eat it too -- giving us all the heavy lift capabilities that we need, minimizing the significant impact on our economy that both the Constellation program, and the cancellation of the STS and Constellation programs, create.

I firmly believe that we are on the cusp of a major economic shift, where commercialization of space may play as important a role in the coming decade or two as the Internet has played in the previous two. The question is, will we as a nation continue to develop that potential, or will we let it slip away, to be picked up by India, China, or Russia?

Yes, I'm an American. And I believe that it is important for America to be a leader in the exploration and utilization of space. Ultimately, I believe that "planting flags" is much more important than the proponents of solely robotic exploration would have us believe. Someday people will visit Mars. Will America be there, or will we just be an observer while one of the Asian nations celebrates a major achievement?

Wednesday, March 3, 2010

ON IPS surprisingly easy

So I have an EOF RTI that was in queue when the ON IPS integration happened last night.

Of course, this totally whacked my packaging changes, and I had to modify them. Making the changes was quite easy. Here's the old, and the new version of the changes. Its actually less files to update under IPS.

I was dreading retesting. Dealing with distro construction sounded "painful".

I needn't have worried. In the tools directory there is this neat tool called "onu" (on-update I guess?)

I had to load a machine with b133 to set up a baseline, but we have a nice way to do that internally via our internal infrastructure and AI. It boils down to running one command on an install server than doing "boot net:dhcp - install" at the OBP prompt. (Yes, this is a SPARC system.)

It took a little bit for it to install, but less than an hour.

Then, after rebooting and getting the initial settings on the system, it was just a simple matter of "onu -d ${ws}/packages/sparc/nightly-nd" to update it. This took a while (20-30 minutes, I wasn't counting). Eventually the system was up and ready for business. Easier than bfu. Amazing.

Thanks to the IPS team! I can't wait for bfu to finally go away.

Sunday, February 21, 2010

Funny Ancient Software

I just found out that Ubuntu has been shipping (since version 6.06 -- Dapper Drake I think it was called?) and apparently included all the way into forthcoming 10.x LTS version) a program I wrote nearly two decades ago as a student -- vtprint -- and yes, that link points to manual text I wrote way back when.

(This program, "vtprint", was for use with printing from a UNIX shell prompt, when you don't have a better way to move files around. Back then we used commands like "kermit" to connect to a UNIX server from our PC over a 2400 or 9600 baud modem -- and well before PPP or even SLIP.)

I haven't used vtprint since about 1995, but its funny to still see it kicking around. Too bad the docs still have an old e-mail address for me at SDSU.... I guess nobody has needed a bug fix for it for some time.

Monday, February 15, 2010

Congratulations to BMW Oracle Racing

If you're involved in the sailing community, you'll already know that Larry Ellison, who's now ultimately my boss, had put together a team to challenge the America's Cup. They won this weekend, bringing the America's Cup back home to America, and I'm enormously proud of Ellison and his team, both as an American, as a sailor, and -- now -- as an Oracle employee.

Friday, February 12, 2010

Open Development

Note: I'm posting this on my personal blog, which as always is a reflection of my own thoughts and in no way represent any official policy from my employer (whoever that may be).

Now that we former Sun employees (for the most part) are now part of a larger company, there have been some questions about how much of the trend Sun had made towards open development will continue (particularly where Solaris/OpenSolaris is concerned.)

(I want to separate the concern of Open Source -- where source code is made available for products after they are released -- from Open Development -- where the product is developed in the open.)

Many of us who were part of that acquisition are wondering the same things. Officially, the word is "no changes in what we're doing", but unofficially there's an atmosphere that our new employer places a greater emphasis on commercial profitability and a lesser emphasis on things like "including the community."

Speaking abstractly, there are risks to any open development effort, particularly when the effort is intended to be supportive of a commercial endeavor. The risks range from enabling competitors with early information, to forestalling customer purchases of bits today as customers wait for the new feature that's being developed in the open, to simply diluting the impact that "surprise" delivery of a new product or feature can make.

Certainly there seems to be some evidence that Oracle may have a greater concern about the costs and risks associated with "early disclosure" than Sun did. No matter how passionately one may believe in Open Development, nobody can deny that there are real costs and risks associated with open development.

So the challenge for Oracle is to figure out what the balance is that makes commercial sense.

Ultimately profit is the primary responsibility of any publicly traded company.

The challenge for the community is to figure how to provide commercial justification to Oracle for the open development that the community likes to see.

If you want to retain truly Open Development (which includes public posting of webrevs for stuff developed internally, open ARC reviews, and public design discussions on mailing lists and similar fora) of Solaris and OpenSolaris, this is call out to you.

Have you or your company made a significant Sun purchase? Has open development (as opposed to open source) influenced your decision? How and why? Will open development influence future purchasing decisions? If you can, put a number to the value of open development, and provide that information to your sales reps or post it publicly.

The decision makers need to see value in the practice of open development if they're going to continue to support it.

Again, I'm only talking about open development, not about open source.

As an aside, I don't think statements coming from community contributors without the support of purchasing dollars are likely to carry much weight with Oracle decision makers. I believe that if you look at the contributions from the community-at-large in OpenSolaris, you'll find that the meaningful contributions have been fairly small and generally of little commercial interest, and have always required additional engineering expense from Sun. So "leveraging" the community for development has not, IMO, been a gamble that has yielded dividends -- at least not from the perspective of either Sun or Oracle.

Thursday, February 4, 2010

Scalability FUD

Yesterday I saw yet another argument about the Linux vs. Solaris scalability debate. The Linux fans were loudly proclaiming that the claim of Solaris' superior scalability is FUD in the presence of evidence like the Cray XT class of systems which utilize thousands of processors in a system, running Linux.

The problem with comparing (or even considering!) the systems in the Top500 supercomputers when talking about "scalability" is simply that those systems are irrelevant for the typical "scalability" debate -- at least as it pertains to operating system kernels.

Irrelevant?! Yes. Irrelevant. Let me explain.

First, one must consider the typical environment and problems that are dealt with in the HPC arena. In HPC (High Performance Computing), scientific problems are considered that are usually fully compute bound. That is to say, they spend a huge majority of their time in "user" and only a minuscule tiny amount of time in "sys". I'd expect to find very, very few calls to inter-thread synchronization (like mutex locking) in such applications.

Second, these systems are used by users who are willing, expect, and often need, to write custom software to deal with highly parallel architectures. The software deployed into these environments is tuned for use in situations where the synchronization cost between processors is expected to be "relatively" high. Granted the architectures still attempt to minimize such costs, using very highly optimized message passing busses and the like.

Third many of these systems (most? all?) are based on systems that don't actually run a single system image. There is not a single universal addressable memory space visible to all processors -- at least not without high NUMA costs requiring special programming to deliver good performance, and frequently not at all. In many ways, these systems can be considered "clusters" of compute nodes around a highly optimized network. Certainly, programming systems like the XT5 is likely to be similar in many respects to programming software for clusters using more traditional network interconnects. An extreme example of this kind of software is SETI@home, where the interconnect (the global Internet) can be extremely slow compared to compute power.

So why does any of this matter?

It matters because most traditional software is designed without NUMA-specific optimizations, or even cluster-specific optimizations. More traditional software used in commercial applications like databases, web servers, business logic systems, or even servers for MMORPGs spend a much larger percentage of their time in the kernel, either performing some fashion of I/O or inter-thread communication (including synchronization like mutex locks and such.)

Consider a massive non-clustered database. (Note that these days many databases are designed for clustered operation.) In this situation, there will be some kind of central coordinator for locking and table access, and such, plus a vast number of I/O operations to storage, and a vast number of hits against common memory. These kinds of systems spend a lot more time doing work in the operating system kernel. This situation is going to exercise the kernel a lot more fully, and give a much truer picture of "kernel scalability" -- at least as the arguments are made by the folks arguing for or against Solaris or Linux superiority.

Solaris aficionados claim it is more scalable in handling workloads of this nature -- that a single SMP system image supporting traditional programming approaches (e.g. a single monolithic process made up of many threads for example) will experience better scalability on a Solaris system than on a Linux system.

I've not measured it, so I can't say for sure. But having been in both kernels (and many others), I can say that the visual evidence from reading the code is that Solaris seems like it ought to scale better in this respect than any of the other commonly available free operating systems. If you don't believe me, measure it -- and post your results online. It would be wonderful to have some quantitative data here.

Linux supporters, please, please stop pointing at the Top500 as evidence for Linux claims of superior scalability though. If there are some more traditional commercial kinds of single-system deployments that can support your claim, then lets hear about them!

Missing audio packages

I have learned that at least two packages, SUNWaudioemu10k and SUNWaudiosolo, are not part of the "standard" ("entire?") install of OpenSolaris b131. If you're looking for either of these, you should do "pfexec pkg install SUNWaudiosolo" or "pfexec pkg install SUNWaudiosolo".

Hopefully we'll get this sorted out before the next official release.

Update: Apparently (according to the expert I talked to) this problem only affects systems updating with pkg image-update. If you install a fresh system, the audio packages should be installed.

Tuesday, February 2, 2010

System board for ZFS NAS

I'm thinking about creating a home storage server, like many, and I want it to be performant enough to host work spaces for compilation over NFS, and efficient enough to reduce my current power consumption somewhat.

I'm thinking of a new Intel D510 system board, and looking at several, I found a board from Supermicro that looks ideally suited to the task. Does anyone else have experience with this board? It looks like its all stock Intel parts, so it should Just Work.

I'm thinking that with 4 or more SATA drives combined with RAIDZ, and dual Intel 82574 gigabit Ethernet (which I could use in an Ethernet link aggregation), I should be able to get excellent performance. (I might even set up jumbo frames, to further bump NFS performance -- if they really are 82574's then they support up to 9K MTU).

Kindle Converts a Skeptic

Recently I bought my wife an Amazon Kindle (the new international unit), at her request. Personally I was rather skeptical -- trying to read book material on computers, even laptops or netbooks, has always felt very awkward to me. I always believed that there was something about holding a paperback (or even a hardback) which would never be replaceable by technology -- maybe for others, but at least not for me.

I have to recant. Debbie has read something like a dozen novels already on her unit. I decided to try it out... and I have to say, I was surprised. I was reading H.G. Wells' War of the Worlds (not for the first time of course), which was a free download, and wow, was I surprised. After 10 or 15 minutes of reading, I almost forgot I was holding something in my hand that isn't printed paper. (The form-factor, which is quite similar to a book, works quite well here. I don't think I'd like the larger DX, as it would destroy the "illusion" of reading a paper back book.)

Not only did the technology not "get in the way", the reading experience was actually more pleasurable, largely because I was able to bump up the font size up to a more comfortable reading level. Last night I read about 1/3 of the book, before I got too tired, but I'm sold on the concept -- and I was a die hard skeptic before.

I don't think I'd like to use it for other things ... but for the primary purpose of reading novels, it settles in quite nicely bringing some technological advantages without letting the technology get in the way of reading.

Will Apple's newer iPad compete here? I'm skeptical. The Apple product is a fancier device, with its backlit screen, and probably will feel more like a hybrid between a laptop and an iPhone (of course I still have an ancient model of phone that is used pretty exclusively for making phone calls -- call me a Luddite.) I suspect that the combination of screen glare, snazz, and lower battery life (iPad users will need to be lot more cognizant of their current battery status), means that its going to be a poor replacement for a Kindle, and an even poorer replacement for the printed materials that the Kindle is meant to replace.

When I go on my round-the-world sailing trip (not any time soon!), would I want a Kindle with me? Absolutely (or something similar) -- along with a solar or wind based charging system. (Product idea... a case for the kindle that integrates photovoltaic solar charging system, so your Kindle is always charging when its closed.)

An iPad? Not likely -- if I'm going to be working or sending e-mails, sure, but then one of the netbooks is probably a better option.. with a "real" keyboard.

Monday, February 1, 2010

Reprehensible behavior from a monopoly

Misbehavior stemming from lack of competition is apparently not unique to the IT industry.

I saw this post today, and couldn't believe it. And then a bit of additional research shows this is not unique -- a number of people complained about actions on the part of Greyhound that would never be tolerated in market where there is true competition.

Forcing a grandmother to wait out in cold, while there's still snow on the ground, may not be in violation of the letter of the law, but it is certainly in violation of the basic tenets of human decency, and the management at the Memphis location showed they have none.

Its been over ten years since I've ridden a Greyhound (or any other long-haul bus for that matter), and after reading this, I am unlikely to ride another Greyhound again. Instead I'll stick to air transport where lively competition means that even the worst airlines understand that they have to at least pretend to care about their customers.

If you're reading this and thinking about taking a Greyhound somewhere, don't.

While there may not be much competition for Greyhound for long-haul ground travel, there is at least some. And there is always air transport for those able to use it.

I'll be interested to hear if Greyhound corporate does anything to fix the problems they obviously have. A good start would be firing most or all of the staff at their Memphis location (especially the management and security guard in question) and refunding the tickets of each of the passengers who were stuck there.

Wednesday, January 27, 2010

Its Official

I'm no longer a Sun Microsystems Employee, since Sun no longer exists. Hopefully I'll get to keep my job at Oracle, but I've not seen any e-mail yet. I expect I will before day's end.

Monday, January 25, 2010

Auto Install Finally Working For Me!

Some of you may know, I've been struggling (and failing) to make auto install work for me. I've had challenges, because my network is not routable, and due to other issues (bugs!) in OpenSolaris Auto Install.

However, it seems that I've finally hit on a successful recipe. I want to record this here for others.

First, in order to use AI, you will need your installation server to be running a recent build of OpenSolaris. The release notes indicate b128 is sufficient. I just ran "pkg image-update" to update to build 131. If you fail to do this, there won't be a warning at all, its just your clients will simply not boot.

The next thing you'll need to do is download a full-repository and the AI image.

Unfortunately, there are not public versions of the full repo ISO file available that are "current". (No, I can't get you a copy, and no I don't know why they haven't posted a more recent update.) Hopefully this problem will be corrected soon.

Setting up the local repository can be done following these directions. (Note that you will have to change the paths to reflect your system. I stash ISO images in /data/isos, and install images under /data/install. These are separate ZFS filesystems.

# where do ISOs live, without leading /
# where does the repo live, without leading /
# AI service name to use
# parent directory for installations, without leading /
# port to use for install server

mount -F hsfs -r /${ISODIR}/osol-repo-131-full.iso /mnt
zfs create -o compression=on ${REPO}
svccfg -s application/pkg/server setprop pkg/inst_root=/${REPO}
svccfg -s application/pkg/server setprop pkg/readonly=true
svccfg -s application/pkg/server setprop pkg/port=${PORT}

You'll need to edit the ${REPO}/cfg_cache file, changing the origins entry to match your system. I used a value like this:

origins =

Then you'll want to use installadm to setup an initial boot service. Here's the recipe I used:

zfs create -o compression=on ${INSTDIR}/${NAME}
installadm create-service -n ${NAME} -s /${ISODIR}/osol-dev-131-ai-x86.iso /${INSTDIR}/${NAME}

Now you need to change the default manifest file. This is the tricky part, that IMO was not terribly well explained anywhere else.

cp /${NSTDIR}/${NAME}/auto_install/default.xml /tmp

Then edit default.xml file in /tmp, changing the value of "main_url" to point to your server. I used a value like this:

<main url="" publisher=""/>
Then apply this manifest to the default manifest:

installadm add -m /tmp/default.xml -n ${NAME}

Finally, I did some tweaking in my DHCP configuration. I have a macro for each service name, that provides the defaults. For example, my "os131_x86" macro looks likes this:

Include pepper
BootFile os131_x86
GrubMenu menu.lst.os131_x86

My "pepper" macro (pepper is the name of my server) sets some shared defaults, but most especially it sets BootSrvA to the IP address of the server ( in my case.)

Then I just configure individual addresses for which ever version of OpenSolaris (or SXCE) I want to install using the the correct configuration macro. (For SXCE there are very different DHCP options to use. Also the SPARC version of OpenSolaris uses different options as well.)

Tuesday, January 19, 2010

Six Years & Counting

Its hard for me to believe that six years ago today at this hour in the morning I was getting myself ready to meet my bride. We had a wonderful wedding on the beach in front of the Del Mar powerhouse in San Diego, with our friends and family in attendance.

Looking back, its been the best six years of my life. I've truly been blessed. I'm looking forward to spending the next sixty together with my beautiful bride Deborah.

Thursday, January 14, 2010

Interesting device driver work

So there are a couple of "closed" drivers that are not part of OpenSolaris, and might never be because of redistribution restrictions. However, this represents an opportunity for an enterprising software engineer to contribute. The drivers are

glm - Symbios 53x810 and similiar devices
qus - QLogic ISP 10160 and similar devices
adp - Adaptec AIC 7870P and similar devices
cadp - Adaptec AIC 7896 and similar devices

There are open source drivers for these from FreeBSD and NetBSD, which could be used as a starting point for a port. I'd probably be interested in trying one of these out myself, if time allowed -- but alas it does not, my plate is already quite full.

The best part of these drivers is that there are few, if any, "political" or "business" restrictions on integrating replacement drivers. Indeed, at one point recently each of these was considered for an EOF simply because they weren't considered strategic anymore. (The EOFs were rejected, but these will only be delivered via an extras repository or somesuch.)

So, what are you waiting for? This is a good opportunity to learn about SCSA, and provide us with superior replacement drivers. (The glm replacement looks like it could be done in as few as 2 or 3 KLOC; that is all the NetBSD version of the driver uses.)