illumos hackathon a resounding success

Yesterday we had our first ever illumos hack-a-thon. About a dozen people from the community showed up, and worked on some very very cool things. At the end of the day, about a half-dozen different demos were done, to show what was worked on.

We'll be posting photos shortly. But here are some of the projects that got attention:

• tab completion for dcmds and data types in mdb
• ::print for DTrace
• expanded truss support for ZFS ioctls (nvlist expansion)
• time-ordered output for DTrace
• a comment field in the vdev label for ZFS pool devices
• the ability to change the GUID of a ZFS pool

Several other projects were in progress.

We selected these projects out of a much larger list of project proposals (which I'll post soon), based on the what people thought was most useful, and the ability to achieve results in a single day hack-a-thon. (And what people we willing to either work on, or mentor.)

Most importantly, people got to work on areas that they weren't intimately familiar with, and work with mentors who were much more familiar. For my own part, I had a blast working with George Wilson to implement the ability to alter the GUID of a pool (which is going to have some further uses we can talk about later). My webrev of these changes is now on-line, so I welcome review feedback.

We had domain experts like Adam Leventhal, Matt Ahrens, Eric Schrock, George Wilson, Gordon Ross, and Dan McDonald on hand, to help mentor projects if they needed it, and the amount of cross fertilization was amazing.

While none of the actual code changes are absolutely earth-shattering, they are still very very cool, useful things, that many of us will be happy to have available. I'm already imagining several useful use cases for the ability to reguid a pool, for example. And I can't wait until I have ::print in DTrace and tab completion in mdb. These will make my life significantly better.

Ultimately, this was the most enjoyable coding experience I've had in a long time. I can't wait until we do it again!

NexentaStor 3.1 available now

After a long, and arduous, release cycle, I am pleased to report that NexentaStor 3.1 is available now.

Customers running existing 3.0 installations may upgrade at no cost.

This release includes a number of key features, including some significant improvements for performance and manageability.

Folks using SCSI target mode will probably see the biggest performance boost relative to earlier editions of NexentaStor, especially those folks using NexentaStor to serve up storage to VMware guests -- thanks to the VAAI offload support that is part of this release. And for the record, yes, this release includes the fix the long standing problem with iSCSI timeouts.

For ZFS fans, this release also includes the updates for ZFS version 28. (This does mean that folks upgrading need to be cautious -- their pools will not be automatically updated to ZFS version 28, but if they are manually updated then there will be no way to move those pools back to an older release. That also means that pools should not be updated in HA cluster configurations unless both cluster partners are updated to NexentaStor 3.1 first.) This also means faster snapshot creation and deletion, and the ability to import pools in read only mode or that have encountered a failure of their SLOG device, making some disaster recovery scenarios much less challenging.

Nexenta Core Platform (our general purpose Debian based open core) will be updated shortly to 3.1 as well -- probably sometime in the next week or so. This will be the same core software as supplied with NexentaStor. (We do strongly encourage folks using Nexenta Core Platform for serving up storage to use our commercial NexentaStor product. There is even a free edition for users with up to 18TB of data.)

We are already (and have been for a while actually) hard at work on the next release, which will be based upon illumos, and include a number of other innovative features. Stay tuned for an update.

Bombproof taskqs

As part of fixing some recent bugs, I integrated the following into illumos:

734 taskq_dispatch_prealloc() desired

943 zio_interrupt ends up calling taskq_dispatch with TQ_SLEEP

The interesting one is the first of these. The interface is actually called taskq_dispatch_ent(), and is private to the "consolidation" (i.e. for use within bundled code only).

What this interface provides for, however, is a way to bomb-proof your taskq dispatches, if you can arrange for the dispatching state structure (taskq_ent_t) to be allocated in advance. This means you never have to worry about the possibility of a dispatch failing due to insufficient resources.

What's even cooler, is that the cost of the dispatching is much cheaper; taskq_dispatch() was the hottest piece of code on a very busy storage server. Now it goes much much faster, because it is just twiddling some linked list pointers and sending a signal to wake up the thread processing the taskq.

More importantly, the fact that we don't ever have to sleep, or have an expensive call into the kernel memory subsystem, has solved some frustrating "stalls" in the storage subsystem.

I encourage developers working with taskqs in illumos to have a look at this interface. If you can use it, you will simplify your code, and shorten your run-paths. Both of which are good things. The only limitation: this interface is not available for dynamic taskqs. (Which makes sense since dynamic taskqs might need to allocate whole threads.)

illumos podcast

Constantin Gonzalez recently interviewed me for his "HELDENFfunk" podcast series, while I was in Amsterdam for our European User's Conference. Also interviewed were OpenIndiana founders Alasdair Lumsden and Andrzej Szeszo.

illumos Panel Discussion in SF Bay Area

All:

There will be a panel discussion about illumos as part of the San Fransisco and Silicon Valley OpenSolaris User's Group meeting tomorrow, June 15. I will be joining Bryan Cantrill and Adam Leventhal to answer your questions about illumos.

The doors open at 6:45pm, and we will continue until about 8pm.

The location is 275 Middlefield #50, Menlo Park, California. Hope to see you there!

NexentaStor 3.0.5 available now

NexentaStor 3.0.5 is now available.

Apart from fixing some key bugs, the main thing that this release includes is a significant update to the CIFS stack, which addresses both performance concerns, and AD failover concerns.

Note that NS 3.1 is due out *any day now*, and will include all these changes, plus a boat load of others. I'll have a lot more to say about the 3.1 release soon.

GSoC Candidates Selected

You may be aware that we have selected two candidates for the slots allocated by Google to illumos -- the first is to replace some system utilities from code in perl to native C. The second of which is to bring GRUB2 to illumos.

What you may not know, is that Nexenta will be sponsoring three additional candidates to pursue projects of their own to benefit illumos. These candidates have been selected already, and we will have more to say about them and their work in the future. Stay tuned!

Thanks again, Joyent!

Joyent have continued to demonstrate their commitment to and support of illumos.

In addition to a string of recent source code integrations, they are now hosting some of our infrastructure in their cloud, with more to follow.

After moving the stuff there, we're now enjoying significantly better performance, and enhanced functionality. Try out the new OpenGrok instance yourself to see!

I'd also like to give a special thank you to Circonus, who are providing active monitoring services for our site now, as a gratuity to illumos. Apparently, they're going to be hosting their stuff on illumos based systems as well, so there's additional synergy here.

What is this OSUNIX thing anyway?

So there has been some things brewing in a sub-sect of the illumos community about a project to fork illumos, because of alleged problems with my leadership. You can read the thread here if you want.

I want to address this head on.

First the claim is that I've got omnipotent control over illumos. This is absolutely false. While I created the project, and serve as technical lead, I've offered to step down if the developer-council and admin-council would like to me to do so. Notably my employer (Nexenta) has minority representation on both councils, and I've tried to keep the groups as neutral as possible. I said when I created the illumos project, and I still maintain, illumos is a community project, not a Nexenta one.

I'm working on the process to make this more formal through non-profit governance. I should have more to say here before the end of week. (I've got a meeting about this today.)

I've also handed over determination of the Advocate list (the list of people who get to approve and integrate submissions) to developer-council. So far Nexenta has 75% of the advocate slots, but this can change at the request of developer-council. Since about 75% of the contributions to the illumos code have come from my team at Nexenta, this should hardly be surprising. In fact, I've flatly refused to add any more Nexenta advocates, even though there are meritorious candidates, until we get broader representation here. (Becoming an advocate requires making a number of good, well-formed, contributions. And it requires people willing to perform thorough review.)

There is a claim that I've somehow driven companies away from illumos. I hope not. As of now, I'm not aware of any companies that have requested to participate or contribute, who I've turned away. In fact, the only contributions that have been turned down have been Joerg Schilling's star project (he couldn't find people willing to review the code) and the ksh93 update (which has been unable to pass a technical code review -- ultimately we'll probably take in the ksh93 changes in more piecemeal fashion breaking them apart into reasonable and reviewable integrations instead of a 100KLOC+ set of code of varying quality.) As far as I know, everything else is vaporware.

I'd love to know what companies I've driven away, and what I did to do so. Honestly, if there is constructive criticsm here, then I want to hear it because I want to a better job -- and I want illumos to be as inclusive as possible. The fact that nobody has come forward (and nobody has approached me privately either!) makes me wonder how much this is really happening.

In fact, I have done all I can to encourage contribution, and to give credit for such contributions where it is due. And indeed, we have contributions from Joyent, Areca, and others. And a number of things queued up from names like Intel and LSI.

At the end of the day, if the project forks, so be it. Forks aren't necessarily a bad thing, and if a fork means we get more contributors to the ecosystem, then I welcome it. But I hope that the basis for such a fork is not just because one or two people don't like me.

(For the record, I am perfectly happy that I'm not everyone's favorite person... my job is to do the best I can for the future of the project, not to be the most universally well loved person. The open source world is filled with other personalities who people have strong feelings about -- Linus Torvalds, Richard Stallman, Theo De Raadt, Andrew Tridgell, and I don't think that the projects they lead have suffered any for it.)

Anyway, I hope that explains my position. If someone wants to have an open dialog with me about any of this, I'm happy to do so. I don't monitor the OSUNIX lists normally, but I'm reachable via email, IRC (gdamore), this blog, twitter (gedamore), and the developer list on illumos.org.

Thank you Joyent!

Joyent posted an update -- they've released a branch of illumos, on github, containing much of their illumos contributions.

Some of the stuff is probably fairly Joyent specific, but some of it is highly useful to almost everyone using illumos!

From their mail:

• ZFS I/O fair-share scheduling for zones
• the Joyent brand, which can be used as a template for other non-SysVR4 or IPS zone brands
• Reintroduction of sparse zone images
• Crossbow vnics on demand for zones & non-unique vnic naming (unique per zone, not per system)
• svcs enhancements ( svcs -Z/-z for interrogating zone services, -L for outputting log files directly (no more ls /var/svc/log | grep... ))
• vfsstat and iostat tweaks and ziostat, iostat(1M) for ZFS I/O
• more per-zone IO kstats
• the zonemon utility for zone kernel state troubleshooting
• DTrace enhancements such as llquantize

I just want to say again, thank you very much Joyent! Now, how quickly can we merge this stuff into illumos mainline?