Thursday, December 30, 2010

beadm provided in "C"

I know this is somewhat controversial for some people, but....

A member of my staff (Alex Stetsenko) has pushed an implementation of "beadm" that is in C. This is actually derived from an earlier C implementation we already had in tree, called "tbeadm", which we already had. So at some level, this consolidation of two different implementations into a single one. As part of this work, the tbeadm version was modernized and improved to provide i18n capabilities and to behave truly as a drop-in replacement for the python version.

As a result of this change, python is no longer needed at runtime by illumos for anything except IPS packaging. Sites and distributions which do not use IPS packaging (most distros don't, actually) no longer need to install python.

Tuesday, December 21, 2010

Any illumos fans near Corinth, MS?

I drove across country last weekend, and am in Corinth, MS. I'd be happy to go out for a beer and some chat if there are any illumos fans or OSUGs nearby this week.

Wednesday, December 15, 2010

I sed(1) so!

I just integrated a new sed(1) port for illumos. This is derived from FreeBSD, but it includes a fix for a race condition, and support for translated messages. (FreeBSD friends, please feel free to include these changes back -- I've not changed the original BSD license.)

The legacy sed is gone.

This new sed should work on all the old sed scripts, but there were a few tricky parts that changed -- if you relied on parsing the output of the "l" command, or on the fact that legacy sed only treated content as a byte stream rather than multibyte characters, you might be affected.

Also, I've run into at least one sed script which was malformed, but mistakenly accepted by old sed, but which the new version doesn't accept (but instead gives you a meaningful error message.)

The biggest features in the new sed code:

  1. support for -i and -I
  2. support for -E to enable EREs
  3. much more helpful error messages ("command garbled" was just not very specific)

Enjoy, and please report any problems in the illumos bug tracking system at Thanks!

Update: Note that sed -i requires an argument (the extension) unlike GNU sed where the argument is optional. We can fix that, although this would make us less compatible with FreeBSD sed. (Specifically, it would make it nigh impossible to specify an "extension" starting with a dash.) If someone cares passionately about this, they should file a bug and bring it up on the developer list -- I am happy either way.

Wednesday, December 8, 2010

Update on SATA Expanders

So we've done some more research, largely following up on work done by Richard Elling, and I have an update on the SAS/SATA expander problem. There is at least some good news here.

The problems that we've had in the past with these have centered around "reset storms", where a single reset expands into a great number of resets, and I/O throughput quickly diminishes to zero.

The problem is that when a reset occurs on an expander, it aborts any in-flight operations, and they fail. Unfortunately, the *way* in which they fail is to generate a generic "hardware error". The problem is that the sd(7d) driver's response to this is to ... issue another reset, in a futile effort to hopefully correct things.

Now the problem is that this behavior is also performed, by default, for media errors as well. E.g. if you have a disk that has a bad sector on it. Of course, if your disk is mostly idle, it won't be a problem. But if you have a lot of I/O going on, its going to result mostly in a melt-down.

There is good news though, because of the way LSI's drivers are designed.

The LSI mptsas driver at least (and I suspect mpt as well, though I don't have code to look at it) treats "bus-level" resets and "target-level" resets as the same. Both of them do a reset, which will of course reset the expander.

But we can disable the most pernicious reset in sd with the following line in sd.conf:


This will allow bus-wide resets to occur, but it will most specifically disable the reset in response to generic hardware and media errors. The relevant section of code in sd.c is this:

if ((un->un_reset_retry_count != 0) &&
(xp->xb_retry_count == un->un_reset_retry_count)) {
/* Do NOT do a RESET_ALL here: too intrusive. (4112858) */
if (un->un_f_allow_bus_device_reset == TRUE) {

boolean_t try_resetting_target = B_TRUE;

* We need to be able to handle specific ASC when we are
* handling a KEY_HARDWARE_ERROR. In particular
* taking the default action of resetting the target may
* not be the appropriate way to attempt recovery.
* Resetting a target because of a single LUN failure
* victimizes all LUNs on that target.
* This is true for the LSI arrays, if an LSI
* array controller returns an ASC of 0x84 (LUN Dead) we
* should trust it.

if (sense_key == KEY_HARDWARE_ERROR) {
switch (asc) {
case 0x84:
if (SD_IS_LSI(un)) {
try_resetting_target = B_FALSE;

if (try_resetting_target == B_TRUE) {
int reset_retval = 0;
if (un->un_f_lun_reset_enabled == TRUE) {
"error: issuing RESET_LUN\n");
reset_retval =
if (reset_retval == 0) {
"error: issuing RESET_TARGET\n");
(void) scsi_reset(SD_ADDRESS(un),

The savy folks here might notice that this is a wide setting, which is true. You can set it on a specific instance of sd, which requires more effort. There is also a better way to do this, by setting the reset_retry_count property to zero. However, setting the sd.conf property for that properly is considerably more complex, because of the byzantine syntax that sd uses to set up target-specific property values.

So, I still recommend avoiding these SATA expanders. But if you have no choice, then using this sd.conf tunable may be a reasonable workaround.

At the same time, I'm investigating the possibility of having this disabled by default for all of Nexenta's customers -- and possibly even in illumos. If you're a SCSI expert and have opinions on the matter, please let me know.

Sunday, December 5, 2010

Status Update on the illumos Foundation

I sent this out earlier today:

We are working with the Software Freedom Law Center & Eben Moglen on the creation of the illumos legal entity. Nexenta have enlisted the help of Damien Eastwood (one of the more prominent former Sun lawyers) to help drive this. Jason Yoho at Nexenta is driving this fairly hard as well, so there are now more people than just me pushing this forward as quickly as we can.

We have set a goal that the legal entity (an illumos foundation) should exist with legal presence before the year's end. I'm told that this is an achievable goal.

I'll have more updates on this soon, I expect, but the process is moving forward.

New Advocate - Albert Lee

I'm pleased to announce the addition of Albert Lee (, aka Triskelios on IRC) to the list of Advocates who can approve integrations into illumos. Albert has been doing a lot of excellent work on both illumos and OpenIndiana, and I'm happy to expand the set of advocates we have available to include such a diligent and talented individual.

The current list of Advocates for illumos-gate are:

  • Garrett D'Amore
  • Albert Lee
  • Rich Lowe
  • Gordon Ross

I'm hoping to expand the list to include more non-Nexenta-employees as well. If you're a contributor and would like to help out in this way, let me know. Typically becoming an advocate means you have earned the trust of the rest of the advocates by making several "good" integrations into illumos-gate (4-5 at least usually, although some credit is given for previous integration experience with ON at Sun/Oracle), and have a demonstrated level of thoroughness to help us ensure quality integrations.

Thanks, and again, congratulations and thank you to Albert.

Wednesday, December 1, 2010

New open source iprb(7D) driver

For a variety of byzantine reasons, the iprb driver has never been open sourced, even though everyone who's ever actually had anything to do with it agrees that it should be. (I blame the lawyers on this one...)

So I went ahead and reimplemented -- from scratch -- a new iprb driver. I'd certainly appreciate feedback on the code, which you can read in the webrev. I'm hoping to integrate this into illumos later this week.

zfs should not depend on python... and doesn't anymore

As of the recent integration of a colleague of mine, illumos now has a zfs command that does not depend on python at all.

changeset: 13246:fe5d6e0b0bce
tag: tip
user: Alexander Stetsenko
date: Wed Dec 01 02:30:25 2010 +0300
278 get rid zfs of python and pyzfs dependencies
Reviewed by:
Reviewed by:
Reviewed by:
Reviewed by:
Approved by:

The zfs command is now entirely a C program. This may make it more friendly for use in other environments or platforms. FreeBSD folks, you might want to incorporate this into your tree. If you do, I'd sure like to know.