Thursday, October 17, 2013

illumos corporate entity... non-starter?

I want to give folks a status update on the illumos corporate entity.

In a nutshell, the corporate entity seems to be failing to have traction.  In particular, the various corporate contributors  and downstreams for illumos have declined to step up to ensure that an illumos corporate entity has sufficient backing to make it successful.

While at first blush, this seems somewhat unfortunate, I think this is not nearly quite as bad a thing as it might first seem.  In particular, the failure of a corporate entity does not correlate to the health of the ecosystem -- indeed many successful open source projects operate without an umbrella organization or entity. 

Instead, we see corporate contributors and downstream distributions focusing on developing the communities behind their distributions such as SmartOS and OmniOS.  Those downstreams play an active role in improving illumos for the benefit of all, and its my sincere hope and belief that they will continue to evangelize illumos, and contribute to the common core.

Furthermore, incorporation in the state of California requires paying about $800 of taxes per year.  This is true even for organizations without any revenue.  This is money that would have to be taken from sponsors that would serve only to enrich our state government with no direct benefit to the illumos community.  (Non-profit status is a way around that, but its exceedingly difficult to obtain, and a number of open source organizations are finding themselves under very close scrutiny from the state and federal authorities.    Indeed, the X.org community themselves lost their 501(c)3 status not that long ago.)

So, without corporate sponsors to justify the tax and administrative load, I've decided that the illumos corporate entity should expire.  I do want to thank Deirdré Straughan for the non-trivial amount of effort she put into this, as well as the Software Freedom Law Center for the pro-bono work they did for us while we were trying to navigate the waters of becoming a true non-profit open source organization.

And, if any corporate sponsors are out there watching this, and interested in resurrecting the illumos organization, then I'm happy to entertain the possibilities.  I think there is value in an actual organization with a legal status to receive and distribute funds, and who can hold certain items of intellectual property, including the rights to the illumos trademark itself.   But there has to be enough of a sponsorship behind this to make it worthwhile.

In the meantime, I will continue to hold the illumos mark as a trademark that I keep in trust for the community.  The code is something that is already subject to distributed ownership, and so is completely unimpacted by any of this.

Thanks.

  - Garrett

Sunday, October 13, 2013

Moving on (Adieu to Studio?)

I think illumos is at a key juncture, and the issue relates to our toolchain.

We have gcc 4.4.4 working thanks in large part to the efforts of folks like Rich Lowe.

We have historically relied on Sun Studio (now Solaris Studio) as our "base" or "default" compiler for C and C++ programs, and also to supply lint coverage.

I've been thinking for a long time that its past time we (the illumos community) moved on from this.  Not only are the Studio 12 compilers not available in source form, they are now not available in suitable form for building illumos as binaries either.  (Apparently it is possible under some terms to get Solaris Studio 12.3, but who knows if those compilers are suitable for building illumos.  In the past we have always needed specifically patched compilers for Solaris.)

The situation where a general developer cannot obtain the necessary tool chain for working with illumos is untenable.

Today we require "lint" as part of an official build.  But not just "any" lint, but lint generated by a specific patched older version of Sun Studio 12.  A version that is no longer available to the community at large.

(And in full disclosure, the problem for me is brought to a head by the fact that the machine I had my local installation of these tools on has been retired, and I find I don't have another backup of them readily at hand to install on my new machine.)

So, I'm placing this as a call to the illumos community at large, and especially to our RTI advocates.  Its time.  Really. 

Really.

Studio has to go.

I don't care if we leave the infrastructure in place for people that want to continue to use Studio, but we need to switch to gcc.  Rich's 4.4.4 version gets the job done for now.  It would be great if we could support newer versions, but I understand that this requires some non-trivial amount of effort (some gcc patches that need to be upstreamed, as I understand it.)

We cannot be tied to a closed tool chain; especially a tool chain that doesn't at least include binary redistribution privileges. 

For the record, I've posted the same content to the illumos developer and advocates lists.

Thursday, August 29, 2013

Responding to Trolls

We have an... ahem... toxic personality in our midst in illumos.  I'm going to respond here to his most recent post, to avoid wasting the time on the illumos developers mailing list.  If you don't care about non-sense like this (although it may have entertainment value), please just ignore this post and move on.

The only reason I'm replying even here, is to set the record straight, and establish firmly the history -- at least my side of it -- for posterity.

In the early days of illumos, I invited those I could think of from the Solaris and OpenSolaris communities to participate.  I even reached out to some parties, like Jörg Schilling, who had a bad reputation as difficult people to work with.

I had hoped to repair those broken relationships since there was no longer a corporate controlling entity for illumos.  (Nexenta sponsored my efforts, and paid some of the early operating costs for illumos, but it has always been the case that illumos operates by and for the community -- and not at the behest of any corporate master.  As I've since left Nexenta to start my own company, and illumos has carried on, I think there is fairly ample evidence in support of this fact.)

I reached out, and tried to help Jörg with some integrations; but those integrations did not go well.  Jörg did not like our policy requiring a build or test of changes on the primary distribution (OpenIndiana at the time) used for the vast majority of developers and users, and so several of his changes did not get integrated.  (It turns out this was a good thing, because at least one of those changes, a "fix" for a keyboard problem, was mis-designed and would have broken international keyboards for the majority of illumos' user community.)

We also required code review, as we do now.

Jörg has long had a desire to have "star" integrated as the main replacement for all archivers.  That didn't take place, because he couldn't find reviewers.  And after several negative interactions with him, I withdrew my offer to sponsor his work personally, as I found the process of trying to do so unbelievably painful.  (There was an extremely long telephone call with Joerg that sticks in my memory.)  I suggested he find someone else willing to code review and sponsor his work for integration into illumos.  To date, that has not happened.

Somewhere during that time, Jörg indicated that he would not offer any other code up for integration until star was integrated.  Basically, he took his toys and went home.  Except he just took his toys, but forgot to go home, as he's still trying to create noise on our developer mailing lists.

There was another negative experience we had, mirroring today's thread, where Joerg's response to a tool I wrote (an open replacement for "od") was non-constructive criticism.  (In fact, his response was "it's broken, why don't you use this thing I haven't finished writing yet".  (He modified his "hexdump" utility to behave like od, apparently in response to my effort instead of simply offering helpful review feedback.)

This is typical of Jörg -- the only "right" answer as far as he is concerned is to accept his code, even if it isn't written yet or is buggy!

Jörg is what is known as a poisonous personality.  At least in every context that I've seen him operate. Which is why usually I try to just ignore him, and hope he'll go away; yet like certain garden pests he just keeps coming back.  He's even been kicked out of another prominent open source community.

He's also said made some incorrect and defamatory statements about both Nexenta and illumos at a prominent academic conference in Germany.  (I can't find the link right now, I'll update if I find it.)

So, where in his post today he asks Gordon if he's willing to change the rules for integration, I have this in reply:

Are you willing to change this? 
Gordon isn't empowered to.  Neither, for that matter, am I.  The rules that have been established for collaboration here are by common consent of the major contributors, and no single individual can change them.  Notably, those were the rules in force in Summer 2010, and you were the one that wanted special treatment.   Those rules have served this community well for the past several years, and there has been no movement to change them.

I am interested in setting up a phalanx of OpenSolaris activities because this
is the only way we have a chance to continue with OpenSolaris development but
this is hard as long as Illumos is not willing to play nicely.

OpenSolaris is dead.  The entire illumos community has moved on.  You have not, but then I have long since stopped considering you a part of the illumos community.
Especially given your repeated attacks against both illumos and numerous of its contributors -- even fairly early on you made a number of defamatory and incorrect statements about the nature of both illumos and its contributors in technical presentation before the German academic community.  This was not the behavior of someone who is a constructive contributor.
To be completely honest, I think I'm not alone in wishing that you'd come to the same conclusion: that is that you are not part of this community.

I recently tried to build a bridge by propising that Illumos should implement
the proposals from Summer 2010 to regain credibility. Unfortuntely there was no
reaction.
 
That should speak volumes all by itself. 
As someone else in another open source community said, Jorg Schilling is damage; the community should route around him.

Tuesday, August 27, 2013

An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone.

This milestone was the integration of 195 Need replacement for nfs/lockd+klm

This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at Nexenta.  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix, and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product.

The reason this is such an important milestone, is two fold:

First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of the community.

Second, this effort represents the replacement of the last of the truly closed source code in our kernel proper.  There are still some closed source drivers (although I will point that we're now seeing more traction with vendors to supply open drivers, with recent open source code contributions coming from places like SolarFlare, LSI, and HP.  (HP's contribution -- through illumos partner Joyent -- means that at least one important closed source driver will soon be open -- cpqary3.)

I want to personally thank everyone who contributed to make the integration of the new open source NFS lock manager possible. You've done a big service to the community, and we are grateful!

Thursday, January 17, 2013

ZFS Compression Enhancements in illumos

The ZFS code in illumos has gained another differentiator thanks to the hard work of Saso, who integrated LZ4 improved compression into our code base.

This is a nice achievement, and enhanced compression should be quite useful for most applications of ZFS.  If LZJB was always recommended to be turned on (it was by our team at least!), then LZ4 is even more recommended, unless all of your data is totally incompressible.  (Most data is highly compressible.)

You should check out his wiki post above, which contains a lot more details about the work, including some benchmark data.