illumos sysad/integrator position (NYC)

I have an illumos partner that is interested in finding a strong system administrator/system integrator candidate for work on an illumos-based product in New York. Candidates need to be strong with Solaris, security, scripting (perl, python and/or ruby, etc.) and should have cross platform experience. If this sounds interesting to you, please contact me directly.

need C programmers/interns in So. Cal.

I'm looking to hire a few really smart folks who can work in southwest Riverside County, CA (near Temecula, CA). I want people who are sharp C programmers, want to work on open source kernel software, and are eager to stretch themselves. This is a great learning opportunity. If you know anyone like that, let me know!

illumos update in LA

I'll be giving an update on illumos, as part of my talk in LA this evening. If you're around, please join us!

beadm provided in "C"

I know this is somewhat controversial for some people, but....

A member of my staff (Alex Stetsenko) has pushed an implementation of "beadm" that is in C. This is actually derived from an earlier C implementation we already had in tree, called "tbeadm", which we already had. So at some level, this consolidation of two different implementations into a single one. As part of this work, the tbeadm version was modernized and improved to provide i18n capabilities and to behave truly as a drop-in replacement for the python version.

As a result of this change, python is no longer needed at runtime by illumos for anything except IPS packaging. Sites and distributions which do not use IPS packaging (most distros don't, actually) no longer need to install python.

Any illumos fans near Corinth, MS?

I drove across country last weekend, and am in Corinth, MS. I'd be happy to go out for a beer and some chat if there are any illumos fans or OSUGs nearby this week.

I sed(1) so!

I just integrated a new sed(1) port for illumos. This is derived from FreeBSD, but it includes a fix for a race condition, and support for translated messages. (FreeBSD friends, please feel free to include these changes back -- I've not changed the original BSD license.)

The legacy sed is gone.

This new sed should work on all the old sed scripts, but there were a few tricky parts that changed -- if you relied on parsing the output of the "l" command, or on the fact that legacy sed only treated content as a byte stream rather than multibyte characters, you might be affected.

Also, I've run into at least one sed script which was malformed, but mistakenly accepted by old sed, but which the new version doesn't accept (but instead gives you a meaningful error message.)

The biggest features in the new sed code:

1. support for -i and -I
2. support for -E to enable EREs
3. much more helpful error messages ("command garbled" was just not very specific)

Enjoy, and please report any problems in the illumos bug tracking system at http://illumos.org/. Thanks!

Update: Note that sed -i requires an argument (the extension) unlike GNU sed where the argument is optional. We can fix that, although this would make us less compatible with FreeBSD sed. (Specifically, it would make it nigh impossible to specify an "extension" starting with a dash.) If someone cares passionately about this, they should file a bug and bring it up on the developer list -- I am happy either way.

Update on SATA Expanders

So we've done some more research, largely following up on work done by Richard Elling, and I have an update on the SAS/SATA expander problem. There is at least some good news here.

The problems that we've had in the past with these have centered around "reset storms", where a single reset expands into a great number of resets, and I/O throughput quickly diminishes to zero.

The problem is that when a reset occurs on an expander, it aborts any in-flight operations, and they fail. Unfortunately, the *way* in which they fail is to generate a generic "hardware error". The problem is that the sd(7d) driver's response to this is to ... issue another reset, in a futile effort to hopefully correct things.

Now the problem is that this behavior is also performed, by default, for media errors as well. E.g. if you have a disk that has a bad sector on it. Of course, if your disk is mostly idle, it won't be a problem. But if you have a lot of I/O going on, its going to result mostly in a melt-down.

There is good news though, because of the way LSI's drivers are designed.

The LSI mptsas driver at least (and I suspect mpt as well, though I don't have code to look at it) treats "bus-level" resets and "target-level" resets as the same. Both of them do a reset, which will of course reset the expander.

But we can disable the most pernicious reset in sd with the following line in sd.conf:

allow-bus-device-reset=0;

This will allow bus-wide resets to occur, but it will most specifically disable the reset in response to generic hardware and media errors. The relevant section of code in sd.c is this:

 if ((un->un_reset_retry_count != 0) &&
     (xp->xb_retry_count == un->un_reset_retry_count)) {
  mutex_exit(SD_MUTEX(un));
  /* Do NOT do a RESET_ALL here: too intrusive. (4112858) */
  if (un->un_f_allow_bus_device_reset == TRUE) {

   boolean_t try_resetting_target = B_TRUE;

   /*
    * We need to be able to handle specific ASC when we are
    * handling a KEY_HARDWARE_ERROR. In particular
    * taking the default action of resetting the target may
    * not be the appropriate way to attempt recovery.
    * Resetting a target because of a single LUN failure
    * victimizes all LUNs on that target.
    *
    * This is true for the LSI arrays, if an LSI
    * array controller returns an ASC of 0x84 (LUN Dead) we
    * should trust it.
    */

   if (sense_key == KEY_HARDWARE_ERROR) {
    switch (asc) {
    case 0x84:
     if (SD_IS_LSI(un)) {
      try_resetting_target = B_FALSE;
     }
     break;
    default:
     break;
    }
   }

   if (try_resetting_target == B_TRUE) {
    int reset_retval = 0;
    if (un->un_f_lun_reset_enabled == TRUE) {
     SD_TRACE(SD_LOG_IO_CORE, un,
         "sd_sense_key_medium_or_hardware_"
         "error: issuing RESET_LUN\n");
     reset_retval =
         scsi_reset(SD_ADDRESS(un),
         RESET_LUN);
    }
    if (reset_retval == 0) {
     SD_TRACE(SD_LOG_IO_CORE, un,
         "sd_sense_key_medium_or_hardware_"
         "error: issuing RESET_TARGET\n");
     (void) scsi_reset(SD_ADDRESS(un),
         RESET_TARGET);
    }
   }
  }

The savy folks here might notice that this is a wide setting, which is true. You can set it on a specific instance of sd, which requires more effort. There is also a better way to do this, by setting the reset_retry_count property to zero. However, setting the sd.conf property for that properly is considerably more complex, because of the byzantine syntax that sd uses to set up target-specific property values.

So, I still recommend avoiding these SATA expanders. But if you have no choice, then using this sd.conf tunable may be a reasonable workaround.

At the same time, I'm investigating the possibility of having this disabled by default for all of Nexenta's customers -- and possibly even in illumos. If you're a SCSI expert and have opinions on the matter, please let me know.