Response to Alasdair's Resignation

[ Recently, Alasdair Lumsden announced his retirement as leader of the OpenIndiana community.  This is a copy of my response to that. ]

Dear Alasdair,

Here at the illumos Foundation, we are very sad to hear of your resignation as lead of the OpenIndiana distribution. You have demonstrated great leadership and have been a wonderful friend of the community. We understand that such projects can be very time consuming, and we wish you all the best in your other endeavors. 

As a community, many of us share some of your frustrations. At times, progress does appear to be slow. At other times, there may appear to be a lack of interest from many of the former Sun employees. However, although you may feel that there is a lack of support or interest, many of us active in the community feel differently. While OpenIndiana served to carry on the banner of the OpenSolaris distribution, it was obviously a dead end because the desktop wars are over, and even the Linux community could not win. Meanwhile, the flourishing embedded and special purpose markets for illumos technology continue to grow. 

For example, use of illumos technology is growing in the areas of internet and cloud infrastructure, database and storage appliances, and embedded products. Many successful companies are driving the technology forward and the future has never looked brighter for open source developers looking to contribute to the community and find interesting jobs.

Although you may think we don’t see that the expanding free and open source software marketplace is not our total playground, what we do see is a talented, dedicated group of volunteers committed to seeing illumos become an important part of specialized, niche technology solutions, especially for solving critical problems facing the Internet. 

Honestly, we do not expect the illumos core itself to be economically viable. We do expect--and see--distributions built upon the illumos core becoming commercially successful and growing. This model is similar to the successful Linux model, where the distributions can achieve different results based on the needs of the constituents or the community; some commercial, some less so. 

The model of OpenSolaris is broken. The model of OpenIndiana following OpenSolaris is broken. The illumos model is following the successful Linux model. This is exemplified by distributions such as the commercially supported, general purpose OmniOS, Joyent’s SmartOS, Delphix OS, and others. 

In the future, we expect illumos to become a bigger part of the technological community, as many companies now emerging from stealth mode are incorporating our core platform functions. As our community grows, and word continues to spread of our stability, dependability and scalability, we hope to take illumos to even more specialized markets. 

Among the illumos community of volunteers are some of the smartest, most talented, most dedicated and hard-working people I’ve ever had the pleasure of being associated with. Their years of experience, combined with their insight into the serious problems facing the Internet, assure a rock-solid future for illumos. As a community, we have a shared set of values, and while sometimes we’ve had our disagreements, it’s always been those shared values that have come through and helped the community - and the technology we are building - grow and improve. We will continue to build an enterprise-grade, highly dependable operating system. 

Again, we are saddened by your leaving. On a personal note, I have enjoyed the collaboration that we shared in the formative days of illumos and OpenIndiana, and I hope we will be able to do so in the future.  We thank you for all your efforts on behalf of OpenIndiana and illumos, and know that your efforts have not been in vain; we will take all that we have learned under your leadership and build on it. As a natural evolution of the community, distributions come and go; we take the lessons and move on. Our community has never been one to back down from a challenge; we will handle this in stride and continue to move forward with our mission and our future goals. 

Sincerely,

Garrett D’Amore

illumos Founder

Women in Tech Adverstising - a rebuttal

So, there has been some that have argued vehemently against the advertising policies of GoDaddy, and its ilk.  (See https://twitter.com/bdha/status/211210555577466881 for example of the complaints.)

I have a different point of view.  What about sites that always picture a beautiful woman on a headset for their representation of tech support?  Should we also boycott them?  Would you prefer a site with an ugly person answering the phone?  After all, it's a phone or text chat, so it shouldn't matter at all what the person looks like, right?

Conversely, what about all the sites that use men in their advertising?

My hosting provider, bluehost.com, uses both.  Take a look.  Does the pretty blonde in glasses have anything to do with domain names or web hosting?  Of course not.  The guy wearing the headset for the tech support line isn't exactly hideous either, if you swing that way.  Are you likely to talk to this guy if you call?  Probably not -- he's probably just another model.

GoDaddy uses Danica Patrick as their spokesperson.  She's admittedly beautiful, a professional athlete (she drives race cars), and probably makes a fair bit of money endorsing GoDaddy.  Some of the ads she has been in have admittedly been a little tongue in cheek in their appeal to a mostly male clientele, but I think it would be a stretch to say that GoDaddy is using sex to sell domain services.

It would be an even further stretch to say Danica is being exploited by GoDaddy.

(Note that there may be other reasons to avoid doing business with GoDaddy or its CEO, but I am only referring to the use of Danica Patrick, a female spokesperson, and women in advertising in general.)

Would the people who claim that this is somehow evil objectification of women complain as loudly if the spokesperson were a good looking baseball star?  Is this objectification of men, or of male sports stars?  What if the ad somehow appeals more strongly to women, or to gay men?

What if the site featured an animal spokesperson prominently?  Say an orangutan?  (Perhaps with computer animation. :-)  Would this be considered animal exploitation?

I submit to you, dear reader, that each one of these cases are basically the same.  We use humans, animals, or whatever, to provide a human element and appeal in a product being sold.  Sure, it would be nice if the site instead featured some kind of graphic that demonstrated its technical superiority, but come on -- we're talking about a domain registrar, which is about as boring as tech gets.  I can hardly blame GoDaddy for trying to liven up their site and generate some enthusiasm.

In fact, even in the more extreme cases, such as the use of bikini-clad booth babes, I submit that folks who think objectification like this is somehow exploitative are misunderstanding.  These women are very rarely without many choices and options.  Usually by the time we see them, its when they are in a job that they have worked hard to get -- maintaining a perfect figure can be a full-time job, after all.

If folks like author of the aforementioned tweet had their way, they would deny these women the opportunity to earn a living showing off their own accomplishments.  Isn't that even more exploitative?  In fact, in the extreme case, you could consider the more repressive societies of the middle east?  Is this the social ideal to which we should aspire?  Where the only part of woman we see in advertising is a burqa?

(Note that in the case of the site using bikini-clad booth-babes, my gut response is to question what the company is not showing -- their product.  So while the babes may catch my eye, they have yet to draw me to a booth I would not otherwise have visited for the products featured.  Actually, at one former employer a female colleague -- also quite beautiful, but quite knowledgeable -- is aware that she is often mistaken as a booth babe, and regularly uses this to her advantage; customers get surprised somehow when the beautiful woman knows more about the technology than most other men do.)

As for me, I welcome the diversity of our society, and the appreciation of the human form.  If some people are able to use this to humanize their marketing approach, good for them!  Let's face it, I'd rather look at beautiful faces online than ugly ones.

Far better, IMO, for those who would fight against this spend their energies worrying about the women who are truly exploited or repressed -- in the fundamentalist muslim societies of the middle east, in the brothels, and human slave trade.  Don't feel sorry for the woman who is able to make a nice living for herself displaying the body she has worked hard to get and to maintain, and that she is proud of.

And yes, for the record, I usually like Carl's Jr. commercials too.

Ivy Bridge Motherboards -- Don't *Really* Exist!

So, if you're like me, and you've been shopping for a new system lately because you're last one is dying or dead, you might see some fancy looking Intel E3-1200v2 Xeon systems.  (Why Xeon?  Because ECC memory is a good thing.  I attribute some of my difficulty with my last system to using non-ECC memory.  Because its not ECC, I don't really know whether the problem was in RAM or CPU, but I digress...)

Great, so you pick out a E3-1240v2 Ivy Bridge CPU, and then you look for a System Board.

I found the Supermicro X9SCA-F-O.  Looks like a sweet board.  The spec claims support for the E3-1200v2 cpus, and 1600MHz RAM.  It has a separate IPMI LAN support, as well.  Sweet!  (Look closer though....)

So, package arrives from NewEgg, and you're ready to go.

Assemble the kit.... if you're a software guy like me this might take a couple of hours.

Power it on.

Beep!  Beep!  Beep!  Beep!

WTF does that mean!?  Google "4 beeps supermicro".

Oh, "Unsupported processor."  Further reading....

The system needs a BIOS update (to v2.0) in this case.  Seems reasonable.

Download the instructions....

"Step 1. First, boot the motherboard with an E3-1200 series (not E3-1200 V2)
processor..."

WTF!?!  A new E3-1200 (that's Sandy Bridge mind you, still fairly recent stuff) processor is going to cost me like 200 bucks, minimum.  And I can't return it after I buy it except to exchange it with a different CPU.  But I already have the E3-1240v2.

Anyone want to buy a very lightly used E3-1220 from me?  It looks like I'll be purchasing one tomorrow. Ultimately, I can't afford not to burn the $200, because I simply don't have the time to keep screwing around this stuff.  I've burned the better part of today trying to wrangle hardware.  (Hmmm.... where's a Systems Engineer when you need one?)  Conversely, anyone got one lying around they want to give, loan, or sell really cheap?

Shame on you Supermicro.  There are at least several things you could have done better here:

1. Make it much clearer that if buying a unit, it should include a legacy CPU first.  (I.e. that Ivy Bridge support is only for people upgrading CPUs and not building new systems.)

2. Have a separate SKU for systems that have the requisite BIOS changes.  Even if the silk-screen on the board is the same (and it shouldn't be, because the silk screening indicates only 1066 and 1333 MHz RAM supported, although the BIOS enables 1600 MHz), having the separate SKU lets buyers make sure they have the bits needed.

3. Work with retailers to ensure that existing stock is returned and upgraded as soon as possible.

4. Offer an expedited RMA process for people like me who are screwed in the meantime.  (To be fair, I've not called supermicro yet, but Googling tells me people have not had good luck with this option yet.  The only solution that has worked for people is to get their own 2nd processor.)

5. Technically, this board has an IPMI unit on it.  I think that means it has a separate microprocessor. Why can't I update the BIOS using that?  That would eliminate the need to buy a second CPU, at least for those of us with IPMI enabled gear.

Lessons for me:

1. Its probably worth a couple hundred bucks to let someone else put together the system for me.  I'm a driver guy -- futzing around with actual bits of silicon and cabling, its not my forte.

2. There's a reason it's called the bleeding edge.  I think I got cut with it today.

Why I *hate* external dependencies!

So, I've been trying to setup a system that can build illumos-gate.  Because my old system had a memory DIMM fault, and isn't usable anymore.

I thought, hmm... let me try something other than OpenIndiana.

Big mistake.

Only OpenIndiana seems able to build vanilla illumos-gate right now.

Why?  Because of external dependencies!  I tried building on OmniOS.  The stopper there -- Python 2.4!  On SmartOS, the dependency is IPS itself.  It seems only OpenIndiana is suitable for building stock illumos-gate.

The problem is that our build is inherently very sensitive to the build environment.  It makes life incredibly unpleasant if you try to build on any system that is not configured exactly as we specified.

This, IMO, is an unacceptable situation.

I will speak loudly against any attempt to make changes that introduce further external dependencies.  The fact that some already exist is no excuse.  Every external dependency makes working on the tree more painful.  If you're thinking of a change that would move something out of tree, but add a build-server dependency, think again!  

These dependencies are a serious barrier for contribution.

We (illumos) should take a page from NetBSD, who have successfully arranged that their environment is self contained to the point that they don't care about their build server at all -- all it needs to be able to do is bootstrap gcc and run automake/autoconf.  There is something very elegant about this.

Expect to see me on yet another crusade to kill dependencies on external runtimes, be it Python, Perl, or whatever.   Our tree should, IMO, be as "stand-alone" as possible.

If that means you have to write C, so be it.  Grow up and get a real compiler dude.  I know languages like Python and Perl are attractive because they have a lower barrier to learn -- but depending on them -- and far worse -- depending on *specific* versions of them -- is toxic for anyone else who wants to work on the code in a different environment.   (This has nothing to do with your language of your choice, but everything to do with the pain and suffering your language choice creates for anyone who just wants to do "make install".)

illumos hackathon a resounding success

Yesterday we had our first ever illumos hack-a-thon. About a dozen people from the community showed up, and worked on some very very cool things. At the end of the day, about a half-dozen different demos were done, to show what was worked on.

We'll be posting photos shortly. But here are some of the projects that got attention:

• tab completion for dcmds and data types in mdb
• ::print for DTrace
• expanded truss support for ZFS ioctls (nvlist expansion)
• time-ordered output for DTrace
• a comment field in the vdev label for ZFS pool devices
• the ability to change the GUID of a ZFS pool

Several other projects were in progress.

We selected these projects out of a much larger list of project proposals (which I'll post soon), based on the what people thought was most useful, and the ability to achieve results in a single day hack-a-thon. (And what people we willing to either work on, or mentor.)

Most importantly, people got to work on areas that they weren't intimately familiar with, and work with mentors who were much more familiar. For my own part, I had a blast working with George Wilson to implement the ability to alter the GUID of a pool (which is going to have some further uses we can talk about later). My webrev of these changes is now on-line, so I welcome review feedback.

We had domain experts like Adam Leventhal, Matt Ahrens, Eric Schrock, George Wilson, Gordon Ross, and Dan McDonald on hand, to help mentor projects if they needed it, and the amount of cross fertilization was amazing.

While none of the actual code changes are absolutely earth-shattering, they are still very very cool, useful things, that many of us will be happy to have available. I'm already imagining several useful use cases for the ability to reguid a pool, for example. And I can't wait until I have ::print in DTrace and tab completion in mdb. These will make my life significantly better.

Ultimately, this was the most enjoyable coding experience I've had in a long time. I can't wait until we do it again!

NexentaStor 3.1 available now

After a long, and arduous, release cycle, I am pleased to report that NexentaStor 3.1 is available now.

Customers running existing 3.0 installations may upgrade at no cost.

This release includes a number of key features, including some significant improvements for performance and manageability.

Folks using SCSI target mode will probably see the biggest performance boost relative to earlier editions of NexentaStor, especially those folks using NexentaStor to serve up storage to VMware guests -- thanks to the VAAI offload support that is part of this release. And for the record, yes, this release includes the fix the long standing problem with iSCSI timeouts.

For ZFS fans, this release also includes the updates for ZFS version 28. (This does mean that folks upgrading need to be cautious -- their pools will not be automatically updated to ZFS version 28, but if they are manually updated then there will be no way to move those pools back to an older release. That also means that pools should not be updated in HA cluster configurations unless both cluster partners are updated to NexentaStor 3.1 first.) This also means faster snapshot creation and deletion, and the ability to import pools in read only mode or that have encountered a failure of their SLOG device, making some disaster recovery scenarios much less challenging.

Nexenta Core Platform (our general purpose Debian based open core) will be updated shortly to 3.1 as well -- probably sometime in the next week or so. This will be the same core software as supplied with NexentaStor. (We do strongly encourage folks using Nexenta Core Platform for serving up storage to use our commercial NexentaStor product. There is even a free edition for users with up to 18TB of data.)

We are already (and have been for a while actually) hard at work on the next release, which will be based upon illumos, and include a number of other innovative features. Stay tuned for an update.

Bombproof taskqs

As part of fixing some recent bugs, I integrated the following into illumos:

734 taskq_dispatch_prealloc() desired

943 zio_interrupt ends up calling taskq_dispatch with TQ_SLEEP

The interesting one is the first of these. The interface is actually called taskq_dispatch_ent(), and is private to the "consolidation" (i.e. for use within bundled code only).

What this interface provides for, however, is a way to bomb-proof your taskq dispatches, if you can arrange for the dispatching state structure (taskq_ent_t) to be allocated in advance. This means you never have to worry about the possibility of a dispatch failing due to insufficient resources.

What's even cooler, is that the cost of the dispatching is much cheaper; taskq_dispatch() was the hottest piece of code on a very busy storage server. Now it goes much much faster, because it is just twiddling some linked list pointers and sending a signal to wake up the thread processing the taskq.

More importantly, the fact that we don't ever have to sleep, or have an expensive call into the kernel memory subsystem, has solved some frustrating "stalls" in the storage subsystem.

I encourage developers working with taskqs in illumos to have a look at this interface. If you can use it, you will simplify your code, and shorten your run-paths. Both of which are good things. The only limitation: this interface is not available for dynamic taskqs. (Which makes sense since dynamic taskqs might need to allocate whole threads.)

illumos podcast

Constantin Gonzalez recently interviewed me for his "HELDENFfunk" podcast series, while I was in Amsterdam for our European User's Conference. Also interviewed were OpenIndiana founders Alasdair Lumsden and Andrzej Szeszo.

illumos Panel Discussion in SF Bay Area

All:

There will be a panel discussion about illumos as part of the San Fransisco and Silicon Valley OpenSolaris User's Group meeting tomorrow, June 15. I will be joining Bryan Cantrill and Adam Leventhal to answer your questions about illumos.

The doors open at 6:45pm, and we will continue until about 8pm.

The location is 275 Middlefield #50, Menlo Park, California. Hope to see you there!

NexentaStor 3.0.5 available now

NexentaStor 3.0.5 is now available.

Apart from fixing some key bugs, the main thing that this release includes is a significant update to the CIFS stack, which addresses both performance concerns, and AD failover concerns.

Note that NS 3.1 is due out *any day now*, and will include all these changes, plus a boat load of others. I'll have a lot more to say about the 3.1 release soon.

GSoC Candidates Selected

You may be aware that we have selected two candidates for the slots allocated by Google to illumos -- the first is to replace some system utilities from code in perl to native C. The second of which is to bring GRUB2 to illumos.

What you may not know, is that Nexenta will be sponsoring three additional candidates to pursue projects of their own to benefit illumos. These candidates have been selected already, and we will have more to say about them and their work in the future. Stay tuned!

Thanks again, Joyent!

Joyent have continued to demonstrate their commitment to and support of illumos.

In addition to a string of recent source code integrations, they are now hosting some of our infrastructure in their cloud, with more to follow.

After moving the stuff there, we're now enjoying significantly better performance, and enhanced functionality. Try out the new OpenGrok instance yourself to see!

I'd also like to give a special thank you to Circonus, who are providing active monitoring services for our site now, as a gratuity to illumos. Apparently, they're going to be hosting their stuff on illumos based systems as well, so there's additional synergy here.

What is this OSUNIX thing anyway?

So there has been some things brewing in a sub-sect of the illumos community about a project to fork illumos, because of alleged problems with my leadership. You can read the thread here if you want.

I want to address this head on.

First the claim is that I've got omnipotent control over illumos. This is absolutely false. While I created the project, and serve as technical lead, I've offered to step down if the developer-council and admin-council would like to me to do so. Notably my employer (Nexenta) has minority representation on both councils, and I've tried to keep the groups as neutral as possible. I said when I created the illumos project, and I still maintain, illumos is a community project, not a Nexenta one.

I'm working on the process to make this more formal through non-profit governance. I should have more to say here before the end of week. (I've got a meeting about this today.)

I've also handed over determination of the Advocate list (the list of people who get to approve and integrate submissions) to developer-council. So far Nexenta has 75% of the advocate slots, but this can change at the request of developer-council. Since about 75% of the contributions to the illumos code have come from my team at Nexenta, this should hardly be surprising. In fact, I've flatly refused to add any more Nexenta advocates, even though there are meritorious candidates, until we get broader representation here. (Becoming an advocate requires making a number of good, well-formed, contributions. And it requires people willing to perform thorough review.)

There is a claim that I've somehow driven companies away from illumos. I hope not. As of now, I'm not aware of any companies that have requested to participate or contribute, who I've turned away. In fact, the only contributions that have been turned down have been Joerg Schilling's star project (he couldn't find people willing to review the code) and the ksh93 update (which has been unable to pass a technical code review -- ultimately we'll probably take in the ksh93 changes in more piecemeal fashion breaking them apart into reasonable and reviewable integrations instead of a 100KLOC+ set of code of varying quality.) As far as I know, everything else is vaporware.

I'd love to know what companies I've driven away, and what I did to do so. Honestly, if there is constructive criticsm here, then I want to hear it because I want to a better job -- and I want illumos to be as inclusive as possible. The fact that nobody has come forward (and nobody has approached me privately either!) makes me wonder how much this is really happening.

In fact, I have done all I can to encourage contribution, and to give credit for such contributions where it is due. And indeed, we have contributions from Joyent, Areca, and others. And a number of things queued up from names like Intel and LSI.

At the end of the day, if the project forks, so be it. Forks aren't necessarily a bad thing, and if a fork means we get more contributors to the ecosystem, then I welcome it. But I hope that the basis for such a fork is not just because one or two people don't like me.

(For the record, I am perfectly happy that I'm not everyone's favorite person... my job is to do the best I can for the future of the project, not to be the most universally well loved person. The open source world is filled with other personalities who people have strong feelings about -- Linus Torvalds, Richard Stallman, Theo De Raadt, Andrew Tridgell, and I don't think that the projects they lead have suffered any for it.)

Anyway, I hope that explains my position. If someone wants to have an open dialog with me about any of this, I'm happy to do so. I don't monitor the OSUNIX lists normally, but I'm reachable via email, IRC (gdamore), this blog, twitter (gedamore), and the developer list on illumos.org.

Thank you Joyent!

Joyent posted an update -- they've released a branch of illumos, on github, containing much of their illumos contributions.

Some of the stuff is probably fairly Joyent specific, but some of it is highly useful to almost everyone using illumos!

From their mail:

• ZFS I/O fair-share scheduling for zones
• the Joyent brand, which can be used as a template for other non-SysVR4 or IPS zone brands
• Reintroduction of sparse zone images
• Crossbow vnics on demand for zones & non-unique vnic naming (unique per zone, not per system)
• svcs enhancements ( svcs -Z/-z for interrogating zone services, -L for outputting log files directly (no more ls /var/svc/log | grep... ))
• vfsstat and iostat tweaks and ziostat, iostat(1M) for ZFS I/O
• more per-zone IO kstats
• the zonemon utility for zone kernel state troubleshooting
• DTrace enhancements such as llquantize

I just want to say again, thank you very much Joyent! Now, how quickly can we merge this stuff into illumos mainline?

CFV: illumos content authors

I'm looking for people interested in contributing content to the illumos website. Right now we have a test website but it needs help with producing content. First and foremost we need English content, but the new framework will support other localizations as well.

If you're interested in contributing here, drop me an email. I'll be setting up a mailing list for this soon.

Thank you Areca! 6Gbps 1880 support in illumos

A big thank you goes out to Areca.

Areca have provided the source code for their Solaris driver, including support for the newer 6G RAID adapters. As a result, I've integrated a (somewhat cleaned up) copy this code as an update to arcmsr(7d) in illumos, under generous open source licensing:

changeset:   13305:fb26af81b9b2
tag:         tip
user:        Garrett D'Amore 
date:        Fri Mar 25 22:14:56 2011 -0700
description:
      834 need support for Areca 1880 6Gbps
      Reviewed by: Dan McDonald <danmcd@nexenta.com>
      Reviewed by: Albert Lee <trisk@nexenta.com>
      Reviewed by: Richard Lowe <richlowe@richlowe.net>

This will make another HBA option available to folks. (Note these cards do support a JBOD mode, so you don't have to use hardware RAID -- indeed I would recommend that you don't when you have ZFS on the disks.

Another outlet..

So, at the recommendations of others, I'm on twitter now.. Don't know how often I'll keep it updated, but I'll try.

illumos has Serbian Family Language Support

I just integrated:

changeset: 13312:537259ad27f6

tag: tip

user: Garrett D'Amore

date: Wed Mar 23 08:35:14 2011 -0700

description:

324 need serbian locale support

Reviewed by: Rich Lowe

Approved by: Garrett D'Amore

This is a bit unusual relative to most of the locales, because Serbo-Croatian is a language fraught with some unique political considerations:

There is a common root language, that everyone speaks and understands. But speakers of it rarely agree on what to call it. In Serbia its Serbian. In Bosnia its Bosnian. And so on for Croatian and Montenegrin.

In illumos, we have followed the Unicode CLDR example, and we now have these locales:

hr_HR.UTF-8 - Croatian in Croatia

sr_BA.UTF-8 - Serbian in Bosnia and Herzegovina

sr_ME.UTF-8 - Serbian in Montenegro

sr_RS.UTF-8 - Serbian in Serbia

I want to apologize to anyone offended by this decision, but rather than make a contentious decision on our own, I decided it was best to simply follow the decisions of an international standards body. I believe that there is no fundamental difference in the languages, although some national variances appear to be present in the data files. If someone has more accurate names for these, or believes that some aliased locales will assist with compatibility with other operating systems, then I would be happy to hear suggestions. Ideally from someone familiar with accepted practice in these locations.

There is another wrinkle in all this too. This language -- thanks largely to occupation by Soviet forces as part of the SFR Yugoslavia, is commonly represented using two different alphabets -- Cyrillic and Latin. Generally most locations use Latin, but within Serbia, Cyrillic is mandated by law. So sr_RS uses Cyrillic, while the others use Latin.

Here are the two alphabets:

А Б В Г Д Ђ Е Ж З И Ј К Л Љ М Н Њ О П Р С Т Ћ У Ф Х Ц Ч Џ Ш

A B C Č Ć D Dž Đ E F G H I J K L Lj M N Nj O P R S Š T U V Z Ž

Anyway, if someone sees room for corrections or improvements here, especially if they are familiar with the language(s) and/or region(s), I would appreciate hearing back from you.

Planet OpenSolaris *isn't*

It would appear that the old Planet OpenSolaris is no longer a community site.

At least the only blog posts that seem to be there anymore are those that are hosted on blogs.sun.com.

Certainly my posts, which used to show up there until quite recently, no longer do so.

Its possible that this is just a technical snafu, but the recent burst of posts there from Oracle employees suggest a shuffling of things internally in how Oracle handles blogs, and I suspect that eradication of community posts is just one more step along the way.

Of course, if I'm wrong, this post will show up there, and I'll have egg all over my face. :-)

Update: illumos is accepted as GSoC Mentoring Org

Great news! We (illumos) have been accepted as a Google Summer of Code mentoring organization.

If you are a student and want to get paid this summer to work on an enterprise grade operating system, please have a gander at our ideas page, and then go ahead and start an application.

You can find our application template on our organization information page on the GSoC 2011 site.

Good luck to all the applicants!

illumos gets documentation!

With this integration:

changeset:   13304:b54231762cfa
tag:         tip
user:        Richard Lowe 
date:        Mon Mar 14 14:05:30 2011 -0400

description:
        243 system manual pages should live with the software
        Reviewed by: garrett@nexenta.com
        Reviewed by: gwr@nexenta.com
        Reviewed by: trisk@opensolaris.org
        Approved by: gwr@nexenta.com

We now have manual pages in illumos. (Only the English pages -- POSIX locale -- are kept in the illumos code repository.)

This is key because it means that code and documentation can be maintained together, which is how some other projects (but not Solaris) manage it.

So, got a problem with the man(1) pages on illumos? File a bug! There is a category called "manpage"... please let us know, or even better, contribute a fix!

Google Summer of Code & illumos

Got a pet project for illumos you would like someone to take up, and would do yourself if you had time? Like working with bright up and coming stars?

Are you a student looking to get involved with a nascent community of world class engineers, and have some free time on your hands this summer?

Maybe you can participate in Google's Summer of Code. We hope illumos will be selected to participate this. Some ideas are posted on our wiki already, but I'd love to hear other proposals. We have a very short window of time before we have to submit our mentoring org application, so let us know!

COMSTAR and SCSI UNMAP

I just pushed this for Dan McDonald into illumos:

changeset:   13297:4b9dc4ca8e9f
tag:         tip
user:        Dan McDonald <danmcd@nexenta.com>
date:        Fri Mar 04 13:57:09 2011 -0800
description:
701 UNMAP support for COMSTAR
Reviewed by: Garrett D'Amore <garrett@nexenta.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Approved by: Garrett D'Amore <garrett@nexenta.com>

This change represents a significant new feature in COMSTAR and ZFS, which will greatly benefit people use SCSI target mode functionality in situations involving over-provisioning. More on that in a minute...

The feature itself was developed by Sumit Gupta for Nexenta, and is part of our upcoming 3.1 release of NexentaStor.

Subsequently, Dan took ownership of that code, and working with Eric and George (who are well established ZFS gurus and had significant and useful feedback) improved it still further, and got the code into illumos proper. I believe this represents the first significant ZFS feature to go into the tree since the illumos fork, and also amply demonstrates the collaboration in the illumos community.

I'm looking forward to more collaboration like this in the community.

Now, what does this feature give you? Well, if you're using SCSI Target Mode (either via iSCSI or Fibre Channel or FCoE) to serve up storage to systems running NTFS or ext4, you will be able to make better use of your storage.

Traditionally, when a file was deleted from a filesystem, it was mostly a matter of book-keeping in the meta data in the filesystem. There was nothing to note this in the underlying storage.

With newer SSDs, and with COMSTAR, the ability to get back this notification is incredibly useful. SSDs want it to do garbage collection or other optimizations thereby improving performance.

COMSTAR wants it because now when your thinly-provisioned zvol gets the notification, we can return the storage back to the pool. Prior to this change, the zvol could only grow, it could never shrink. Now, we can give storage back to the pool when you delete a file on the initiator. This is huge in environments running with a lot of VMs using thinly provisioned storage with overallocation.

Anyway, this is now in illumos thanks to Nexenta, and notably, Oracle doesn't have it. Of course, they are welcome to pick up the code for it, but they will need to follow the terms of the CDDL if they choose to do so, the same as everyone else.

SCALE illumos Photos

As promised, I'd send illumos photos from SCALE. Here's the illumos booth staff, from left to right there is Roland, Garrett (your humble author), Delya, Albert, and Rocky. Rocky was there representing Area Data Systems, who are both Nexenta partners and illumos sponsors.

We also have a facebook photo gallery up, which seems somehow apropos since we were right next to the facebook both at SCALE.

I'm also pleased to report that a number of other Nexenta partners were present as well, showing off Nexenta based products. Next year, we hope they'll be back show casing technology based on illumos and NexentaStor 4.0.

Open Source Opportunities near Boston!

Its now fairly locked into stone that we will be opening an engineering office somewhere not far from Boston (probably just north of it) sometime during 1H2011.

As a result, we've started an aggressive recruiting campaign in the area. I am interested in talking to people with backgrounds in kernel software or device drivers -- especially if the background is on Solaris, but Linux and BSD backgrounds are fine too.

The culture here is startup, and while I'd love to find a few more architect-level candidates, I'm also keen to find folks just beginning their career with a high level of enthusiasm who are talented and driven to become the industry's next generation of storage and networking gurus.

Nexenta itself is still a small company, and so its still a great opportunity to get into the ground floor of what may well prove to be the fastest growing storage company ever. And even better, you can feel good knowing that your contributions will contribute to the greater good -- we are a company committed to Open Source and .. as our motto says, "Enterprise Storage for Everyone."

We'll also be hiring Quality Engineers in the same office, so if breaking stuff is more up your alley than building stuff, then there may also be a place for you.

If this sounds interesting to you, please let me know.

Note that we do not work with recruiters -- individual candidates only please.

illumos at SCALE

So, I've spent the past day or so here at the Hilton LAX with Albert, Roland, and Delya manning the illumos booth at SCALE. We're at booth #72. We've been handing out disks with OpenIndiana, Nexenta Core Platform (4.0 alpha release based on illumos), and NexentaStor 3.1 (alpha release) as well as the first ever illumos T-shirts for those folks who install one of these on their system (either physical or virtual.) I'll have photos to post soon -- sorry they're not ready yet.

The show has been amazing, and we've made a lot of new connections, both with other members of the open source community, and with new potential users and contributors as well as other potential collaborators.

If you're around tomorrow, please drop by and say hello!

(Oh yeah, and come watch Richard Elling and I mix it up with the BTRFS guys at the open source filesystems panel tomorrow!)

And if you're looking for an open source job, drop by and chat. We are hiring in a number of areas, and I'd love to have a chance to chat with candidates.

See you tomorrow!

New audio driver

I've just pushed

changeset:   13278:dabee83e3bb7
tag:         tip
user:        Garrett D'Amore 
date:        Mon Jan 31 17:40:15 2011 -0800
description:
    519 RFE audiocmihd
    Reviewed by: gwr@nexenta.com
    Reviewed by: dev@opensound.com
    Reviewed by: trisk@nexenta.com
    Reviewed by: ams@nexenta.com
    Approved by: trisk@nexenta.com

This represents the first significant contribution to illumos by a third party other than Nexenta, and is also the first hardware driver illumos has support for which Solaris does not. This is for ASUS Xonar cards.

You can tell you have a card that could benefit from this if prtconf -vp | grep pci13f6,8788 shows a result.

I expect this will be introduced soon into a forthcoming OpenIndiana build. Enjoy, and a big thanks again to 4Front Technologies for making this contribution!

Welcome to new Nexentians

I'd like to take a minute to publicly welcome the following engineers, who are well known in the OpenSolaris community, to my team within Nexenta.

Dan McDonald - Dan just started today, and joins us from Oracle (and previously Sun), where he was one of the lead engineers on IPsec and networking security in general. He's also famous as the creator of the internal "punchin" tool used by Sun engineers for many years. At Nexenta he'll be doing some different things, but also will be our go-to man for issues involving the TCP/IP stack within NexentaStor and we expect to see contributions coming from him back into illumos.

Andrew Stormont - Andy started on Jan 1, and is our first software engineer in the UK. He's known for creating the StormOS desktop distribution based on XFCE on top of Nexenta Core Platform. We're looking forward to capitalizing on the synergy of StormOS and Nexenta Core Platform forward.

Roland Mainz - Roland is known throughout the community as "Mr. Ksh93", and previously was the contributor of the single largest integration from the open community into OpenSolaris. We expect he'll be continuing some of the work on improving our userland within illumos, fixing ksh93 bugs, and doing some other interesting things which should result in a direct improvement in measurable quality for both illumos and NexentaStor.

So, please join me in welcoming these three engineers to the Nexenta team, and also recognizing their past and future contributions into the illumos open source community.

Changes to illumos Contribution Process

First off, let me state that the following changes are aimed at both easing the challenging of contributing changes to illumos, while increasing our level of "confidence" in what changes are being integrated into our source code tree.

Up until now, illumos has used a contribution model that is primarily derived from the model used within Sun and Oracle for Solaris development.

This development model is based on the notion that all contributors have (or had at least) the direct ability to "push" code to the repository, after a certain number of review steps had been followed.

This model works well with a small team, or where all contributors are reasonably well trusted. This is also not typical at all of the way most FOSS projects work. (Indeed, with OpenSolaris, this model was not used for external contribution.)

Going forward, we want to enable a much wider group of developers, some of whom may not hang around long enough in our community to get a high level of "trust".

We also want to enable a contribution process that is more similar to what other FOSS projects use.

So, to this end, we are going to move from "developer push" to "advocate pull". "Advocates" are just our version of "maintainers" or "gatekeepers". (The Linux equivalent of this is Linus' "lieutenants".)

So now, rather than developers pushing changes directly to our mercurial tree, going forward Advocates will take patches from Contributors (either via hg export or patch file), verify that the content of the patch is what was reviewed, and will then be responsible for integrating those changes into our shared master.

Note that as part of this process, the Advocate will be ensuring that the original Contributor is still credited in the SCM change history. So Contributors still get credit for their work.

Also, we will still be insisting on other parts of the contribution process that we already have, such as code review, testing, and verification of legal right to receive the contribution.

The main implication for Contributors is that they can supply changes in the form of regular patches, which frees them from having to deal directly with one SCM or the other (more on that below). The other

implication is that if a merge conflict occurs that the Advocate can reject a change and ask the contributor to resolve the conflict and resubmit.

Note that this whole process is much much more similar to the process used by other big name open source projects, such as Linux.

At this point, I'd like to point out that we have a "clone" of illumos-gate on github. So you can use git if you want to. We also have an hg clone at bitbucket.

For now, Advocates use hg as our "master" repository, but we are also talking about a conversion to git to make life better. That's a more detailed topic of conversation, but mostly the concern about whether we are using git or hg should be irrelevant to contributors, as they can use either and are not directly exposed to the integration step (hg push or git push for example.)

The final tidbit here is that we need to set up a public page with a list of Advocates, but for now the list of illumos-gate Advocates is:

• Garrett D'Amore
• Albert Lee
• Rich Lowe
• Gordon Ross

As more people contribute and demonstrate a level of throughness and trustworthiness, I hope the above list will expand somewhat.

illumos sysad/integrator position (NYC)

I have an illumos partner that is interested in finding a strong system administrator/system integrator candidate for work on an illumos-based product in New York. Candidates need to be strong with Solaris, security, scripting (perl, python and/or ruby, etc.) and should have cross platform experience. If this sounds interesting to you, please contact me directly.

need C programmers/interns in So. Cal.

I'm looking to hire a few really smart folks who can work in southwest Riverside County, CA (near Temecula, CA). I want people who are sharp C programmers, want to work on open source kernel software, and are eager to stretch themselves. This is a great learning opportunity. If you know anyone like that, let me know!

illumos update in LA

I'll be giving an update on illumos, as part of my talk in LA this evening. If you're around, please join us!

beadm provided in "C"

I know this is somewhat controversial for some people, but....

A member of my staff (Alex Stetsenko) has pushed an implementation of "beadm" that is in C. This is actually derived from an earlier C implementation we already had in tree, called "tbeadm", which we already had. So at some level, this consolidation of two different implementations into a single one. As part of this work, the tbeadm version was modernized and improved to provide i18n capabilities and to behave truly as a drop-in replacement for the python version.

As a result of this change, python is no longer needed at runtime by illumos for anything except IPS packaging. Sites and distributions which do not use IPS packaging (most distros don't, actually) no longer need to install python.

Any illumos fans near Corinth, MS?

I drove across country last weekend, and am in Corinth, MS. I'd be happy to go out for a beer and some chat if there are any illumos fans or OSUGs nearby this week.

I sed(1) so!

I just integrated a new sed(1) port for illumos. This is derived from FreeBSD, but it includes a fix for a race condition, and support for translated messages. (FreeBSD friends, please feel free to include these changes back -- I've not changed the original BSD license.)

The legacy sed is gone.

This new sed should work on all the old sed scripts, but there were a few tricky parts that changed -- if you relied on parsing the output of the "l" command, or on the fact that legacy sed only treated content as a byte stream rather than multibyte characters, you might be affected.

Also, I've run into at least one sed script which was malformed, but mistakenly accepted by old sed, but which the new version doesn't accept (but instead gives you a meaningful error message.)

The biggest features in the new sed code:

1. support for -i and -I
2. support for -E to enable EREs
3. much more helpful error messages ("command garbled" was just not very specific)

Enjoy, and please report any problems in the illumos bug tracking system at http://illumos.org/. Thanks!

Update: Note that sed -i requires an argument (the extension) unlike GNU sed where the argument is optional. We can fix that, although this would make us less compatible with FreeBSD sed. (Specifically, it would make it nigh impossible to specify an "extension" starting with a dash.) If someone cares passionately about this, they should file a bug and bring it up on the developer list -- I am happy either way.

Update on SATA Expanders

So we've done some more research, largely following up on work done by Richard Elling, and I have an update on the SAS/SATA expander problem. There is at least some good news here.

The problems that we've had in the past with these have centered around "reset storms", where a single reset expands into a great number of resets, and I/O throughput quickly diminishes to zero.

The problem is that when a reset occurs on an expander, it aborts any in-flight operations, and they fail. Unfortunately, the *way* in which they fail is to generate a generic "hardware error". The problem is that the sd(7d) driver's response to this is to ... issue another reset, in a futile effort to hopefully correct things.

Now the problem is that this behavior is also performed, by default, for media errors as well. E.g. if you have a disk that has a bad sector on it. Of course, if your disk is mostly idle, it won't be a problem. But if you have a lot of I/O going on, its going to result mostly in a melt-down.

There is good news though, because of the way LSI's drivers are designed.

The LSI mptsas driver at least (and I suspect mpt as well, though I don't have code to look at it) treats "bus-level" resets and "target-level" resets as the same. Both of them do a reset, which will of course reset the expander.

But we can disable the most pernicious reset in sd with the following line in sd.conf:

allow-bus-device-reset=0;

This will allow bus-wide resets to occur, but it will most specifically disable the reset in response to generic hardware and media errors. The relevant section of code in sd.c is this:

 if ((un->un_reset_retry_count != 0) &&
     (xp->xb_retry_count == un->un_reset_retry_count)) {
  mutex_exit(SD_MUTEX(un));
  /* Do NOT do a RESET_ALL here: too intrusive. (4112858) */
  if (un->un_f_allow_bus_device_reset == TRUE) {

   boolean_t try_resetting_target = B_TRUE;

   /*
    * We need to be able to handle specific ASC when we are
    * handling a KEY_HARDWARE_ERROR. In particular
    * taking the default action of resetting the target may
    * not be the appropriate way to attempt recovery.
    * Resetting a target because of a single LUN failure
    * victimizes all LUNs on that target.
    *
    * This is true for the LSI arrays, if an LSI
    * array controller returns an ASC of 0x84 (LUN Dead) we
    * should trust it.
    */

   if (sense_key == KEY_HARDWARE_ERROR) {
    switch (asc) {
    case 0x84:
     if (SD_IS_LSI(un)) {
      try_resetting_target = B_FALSE;
     }
     break;
    default:
     break;
    }
   }

   if (try_resetting_target == B_TRUE) {
    int reset_retval = 0;
    if (un->un_f_lun_reset_enabled == TRUE) {
     SD_TRACE(SD_LOG_IO_CORE, un,
         "sd_sense_key_medium_or_hardware_"
         "error: issuing RESET_LUN\n");
     reset_retval =
         scsi_reset(SD_ADDRESS(un),
         RESET_LUN);
    }
    if (reset_retval == 0) {
     SD_TRACE(SD_LOG_IO_CORE, un,
         "sd_sense_key_medium_or_hardware_"
         "error: issuing RESET_TARGET\n");
     (void) scsi_reset(SD_ADDRESS(un),
         RESET_TARGET);
    }
   }
  }

The savy folks here might notice that this is a wide setting, which is true. You can set it on a specific instance of sd, which requires more effort. There is also a better way to do this, by setting the reset_retry_count property to zero. However, setting the sd.conf property for that properly is considerably more complex, because of the byzantine syntax that sd uses to set up target-specific property values.

So, I still recommend avoiding these SATA expanders. But if you have no choice, then using this sd.conf tunable may be a reasonable workaround.

At the same time, I'm investigating the possibility of having this disabled by default for all of Nexenta's customers -- and possibly even in illumos. If you're a SCSI expert and have opinions on the matter, please let me know.

Status Update on the illumos Foundation

I sent this out earlier today:

We are working with the Software Freedom Law Center & Eben Moglen on the creation of the illumos legal entity. Nexenta have enlisted the help of Damien Eastwood (one of the more prominent former Sun lawyers) to help drive this. Jason Yoho at Nexenta is driving this fairly hard as well, so there are now more people than just me pushing this forward as quickly as we can.

We have set a goal that the legal entity (an illumos foundation) should exist with legal presence before the year's end. I'm told that this is an achievable goal.

I'll have more updates on this soon, I expect, but the process is moving forward.

New Advocate - Albert Lee

I'm pleased to announce the addition of Albert Lee (trisk@nexenta.com, aka Triskelios on IRC) to the list of Advocates who can approve integrations into illumos. Albert has been doing a lot of excellent work on both illumos and OpenIndiana, and I'm happy to expand the set of advocates we have available to include such a diligent and talented individual.

The current list of Advocates for illumos-gate are:

• Garrett D'Amore
• Albert Lee
• Rich Lowe
• Gordon Ross

I'm hoping to expand the list to include more non-Nexenta-employees as well. If you're a contributor and would like to help out in this way, let me know. Typically becoming an advocate means you have earned the trust of the rest of the advocates by making several "good" integrations into illumos-gate (4-5 at least usually, although some credit is given for previous integration experience with ON at Sun/Oracle), and have a demonstrated level of thoroughness to help us ensure quality integrations.

Thanks, and again, congratulations and thank you to Albert.

New open source iprb(7D) driver

For a variety of byzantine reasons, the iprb driver has never been open sourced, even though everyone who's ever actually had anything to do with it agrees that it should be. (I blame the lawyers on this one...)

So I went ahead and reimplemented -- from scratch -- a new iprb driver. I'd certainly appreciate feedback on the code, which you can read in the webrev. I'm hoping to integrate this into illumos later this week.

zfs should not depend on python... and doesn't anymore

As of the recent integration of a colleague of mine, illumos now has a zfs command that does not depend on python at all.

changeset: 13246:fe5d6e0b0bce
tag: tip
user: Alexander Stetsenko
date: Wed Dec 01 02:30:25 2010 +0300
description:
278 get rid zfs of python and pyzfs dependencies
Reviewed by: gordon.w.ross@gmail.com
Reviewed by: trisk@opensolaris.org
Reviewed by: alexander.r.eremin@gmail.com
Reviewed by: jerry.jelinek@joyent.com
Approved by: garrett@nexenta.com

The zfs command is now entirely a C program. This may make it more friendly for use in other environments or platforms. FreeBSD folks, you might want to incorporate this into your tree. If you do, I'd sure like to know.

Job Opp @ Nexenta: Director of Sustaining/Certifications

We're looking for a Director of Engineering, to own sustaining (aka bug fixing) and hardware platform certifications (where partners provide a hardware platform and ask us to certify it for use with NexentaStor.)

The job qualifications:

1. Must be local to the SF bay area (because the hardware lab is here)
2. Must have strong communication skills
3. Must be able to deal with stressful situations, and able to "manage" strong personalities
4. Must have Solaris/OpenSolaris expertise... hands-on kernel work (crash dump analysis, coding, etc.)
5. Desire experience with Storage protocols and products
6. Desire expertise with x86 hardware
7. Desire perl and/or python skills

To be clear, this will start out as a hands-on position, with a fast-paced startup environment. But the growth opportunities here are enormous. If you think you're up to this, please let me know.

New desktop image

Here's a sample of the new logo as a desktop image. I've not made this available publicly yet (mostly because I don't know how to capture this in a form that will include the gradient and post it where people can find it.) If someone with some gnome expertise on how to share this for others contacts me, I can work to make it available.

New illumos logo

Today at the OpenStorage Summit 2010, I unveiled the new illumos logo. We will be updating our branding, which also includes a new font, and other elements, over the next few weeks.

There were other updates on illumos that I covered in this talk. I think this was recorded, but I'm not sure right now where it was recorded and how to acces it. I'll be sure to share that when I find out.

CFV: web/HTML/graphics people

I have an urgent need to rennovate the illumos website. If you'd like to help the project out, and you have got both time and talent, please let me know. A major overhaul of the site is in order, and we need someone willing to dedicate some time on it. There may be some funds available for the right person, but to be clear, illumos can't afford the services of a professional design bureau.

New implementation of printf

So I finally got tired of waiting for someone else to do a printf(1) replacement in illumos for the closed binary from Oracle. I had thought this would be a trivial thing to do via ksh93/libcmd using a symbolic link ala /usr/bin/alias.

Lo and behold, it wasn't! Why? Because ksh93 printf insists (like all ksh93 builtins) on having -- and - getopt style processing. This is fundamentally incompatible with legacy printf. (Why does it do this? So it can dump its builtin man page, e.g. printf --man, to the console. A feature I've railed against in the past.)

Here's what should happen:

% printf -v
-v% 

Here's what ksh93 does:

garrett@thinkpad:~$ printf -v                                                   
ksh93: printf: -v: unknown option
Usage: printf [ options ] format [string ...]

Now there is an argument to be made that a script which relies on the legacy behavior is fundamentally broken. But it doesn't matter -- the scripts are in the field (there are real examples of them), and the legacy behavior must be preserved. Breaking these legacy scripts just so that we can dump printf --version output is... silly. This is case where pragmatism wins over purity.

Rather than try to rip this out and fight with the ksh93 about "deviation from the upstream" (apparently the ksh93 folks view any changes we make in illumos or OpenSolaris as automatically toxic unless they originate from David Korn or Glenn Fowler), I've just gone ahead and implemented my own printf(1) on top of FreeBSD's. This will be the implementation in illumos.

I've added significantly to FreeBSD's code though. Specifically, I added handling of %n$ processing to get parameterized position handling. This is needed for internationalization -- it allows you to change the order of output as part of the output from something like gettext(1). (This is needed when you have to change word order to accommodate different natural language grammars.)

So my implementation is superior to FreeBSD's, and its superior to the legacy closed binary version. Why? Because rather than a half-hearted attempt at processing positional parameters, my version really handles these, including full support for the usual format specifiers. For example:

New open code:

garrett@thinkpad{4}> printf '%2$1d %1$s\n' one 2 three 4
2 one
4 three

Old closed code:

garrett@master{22}> printf '%2$1d %1$s\n' one 2 three 4
134511600 one

Clearly the old behavior is just plain wrong. For the record, ksh93 does the right thing here too. (Although somewhat older versions of ksh93 would dump core on this command line.)

My diffs (which also include style and lint fixes required for illumos) relative to FreeBSD are online. You can also review a webrev of the changes that I hope to integrate into illumos. The license remains BSD, so the various BSD operating systems (or even Oracle) are free to incorporate these improvements if they like.

illumos gets global

I just pushed a major set of changes:

8 libc locale work needs updated license files
223 libc needs multibyte locale support for collation
225 libc locale binary files should be in native byte order
309 populate initial locales for illumos

As a result, illumos has gained base support for some 157 different locales, spanning 67 languages and 116 different territories. This includes nearly all the major languages of the world -- missing are Serbian, Javanese, Farsi, Malaysian, Burmese, and some languages spoken in central and west Africa. (Some of these will be very easy for someone else to add... let me know if you want one of these and are willing to do the work.)

The support for these locales includes full POSIX compliant collating support, which was completely absent in illumos before this integration.

Also, included, is a new open source implementation of localedef(1). To my knowledge, this new implementation is the only non-GNU version of localedef that is fully open, and this version is more fully functional than the GNU version. (The GNU localedef lacks full support for collation data.)

Other notes: this is only the base support for these locales. This will for example give localized output from "date". There is quite a lot of additional effort required to fully localize an illumos system, including support for input methods, fonts, and message catalogs for all the various applications. However, with this base support, it makes doing that other work much more practical.

This integration adds nearly 2 million lines to illumos, although far and away the vast majority of it is in the form of data from Unicode and the CLDR (common locale data repository). The ability to import data directly from these sources is the new code that I've written, including a major overhaul of the underlying ctype and collation support in libc to properly support multibyte locales.

Its my belief that with this integration, one of the biggest feature gaps between illumos and Solaris is closed.

Emacs & Gnome Terminal Co-existence Resolved

For many years, I've been stuck with old xterm, because it was the only one that honored my Meta keys in the same way that GNU emacs did. I could never figure out how to make gnome-terminal work, which always bothered me somewhat. (Notably GNOME terminal has better Unicode support which has lately become important to me.)

I finally found a reference that helped me out. I understood that the problem was conflicting ideas about modifier keys; gnome-terminal uses Mod1, but Emacs uses Mod4. What I didn't know was something I found out here, namely that Emacs only uses Mod4 if it exists. So a better solution for me is to simply clear Mod4 altogether, and both programs happily honor Mod1. (This leaves xterm hosed, but if gnome-terminal works, then I don't need xterm anymore.)

My resulting .xmodmap looks like this:

remove Lock = Caps_Lock
keysym Caps_Lock = Control_L
add Control = Control_L
clear Mod4

This makes my PC keyboard behave sensibly. Alt is Meta. And Caps Lock is consigned to oblivion and the large key that used to have that function is now much more usefully assigned to Control.

I'm posting this here in case anyone else has struggled with this particular annoyance in the past. The clear Mod4 trick was the surprise ticket. (What I'd really like is a way to tell programs which Modifier is "really" the Meta key, given that the programs can't seem to agree on this. And with just one preference -- redefining the numerous bindings in emacs for each sequence, while possible, is not my idea of a fun thing to do.)

The other thing I'd like is a standard way in illumos/opensolaris to integrate .xmodmap. Linux/Ubuntu seems to detect my .xmodmap and handles it nicely.

Another ZFS departure

Jeff Bonwick is leaving Oracle.

This is a huge event, because Jeff has been one of the main innovators in operating system technology during his tenure at Sun. While you may know him best for ZFS, he's also the inventor of the slab allocator, which revolutionized memory management when it was created. (And now, pretty much every modern system uses some variation of the slab allocator.)

And he's not just an Oracle VP. Jeff has made integrations into Solaris' ZFS code base on an ongoing basis. This is a guy that has led with actual actions and innovation, backed by code, rather than some boffin who's risen to management and no longer contributes. At some level, he's the model for the kind of technologist I aspire to be.

With so many innovators leaving (and yes, there are other key players in flight), its going to be very interesting to see how Oracle is able to continue to be a thought leader in the OS technology that they've acquired.

One the one hand, its really a shame to see to much of the heart and soul of the Solaris engineer core slowly disintegrating.

On the other hand, I think illumos may be the place where Solaris innovation happens, more so than at Oracle, even sooner than I previously expected.

South/Central American opportunity

I just learned that a peer of mine is looking to add some escalation engineers in Latin America. Job requirements include excellent English, and the ability to deep dive into customer problems including kernel crash dump analysis and C coding ability. If this sounds interesting to you, please let me know.

Oracle/NetApp ZFS lawsuit dismissed

Others have no doubt already picked upon this, but here it is anyway:

http://www.h-online.com/open/news/item/NetApp-and-Oracle-lift-ZFS-patent-cloud-1076313.html

Hopefully this is good news for downstream ZFS consumers.