An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone.

This milestone was the integration of 195 Need replacement for nfs/lockd+klm

This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at Nexenta.  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix, and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product.

The reason this is such an important milestone, is two fold:

First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of the community.

Second, this effort represents the replacement of the last of the truly closed source code in our kernel proper.  There are still some closed source drivers (although I will point that we're now seeing more traction with vendors to supply open drivers, with recent open source code contributions coming from places like SolarFlare, LSI, and HP.  (HP's contribution -- through illumos partner Joyent -- means that at least one important closed source driver will soon be open -- cpqary3.)

I want to personally thank everyone who contributed to make the integration of the new open source NFS lock manager possible. You've done a big service to the community, and we are grateful!

ZFS Compression Enhancements in illumos

The ZFS code in illumos has gained another differentiator thanks to the hard work of Saso, who integrated LZ4 improved compression into our code base.

This is a nice achievement, and enhanced compression should be quite useful for most applications of ZFS.  If LZJB was always recommended to be turned on (it was by our team at least!), then LZ4 is even more recommended, unless all of your data is totally incompressible.  (Most data is highly compressible.)

You should check out his wiki post above, which contains a lot more details about the work, including some benchmark data.

hackathon project: pfiles - postmortem analysis

During the first days of October 2012, we had an illumos day, and a one day illumos hackathon that was very well attended by some of the best and brightest in our community (and I daresay, in any open source community.)

The purpose of the hackathon, besides just doing some cool projects, is to give folks a chance to interact with other domain experts, and code, particularly in areas outside their particular area of expertise.

This year a number of interesting projects were worked on, and I think some of these are at the point of starting to bear fruit.  The project I undertook was the addition of post-mortem analysis support for pfiles(1).  That is, making it possible to use pfiles against a core(4) file.  Adam Leventhal suggested the project, and provided the initial suggestion on the method to use for doing the work.

This is a particularly interesting project, because the information that pfiles reports is located in kernel state (in the uarea) and has not traditionally been available in a core file.

To complete this project, I had to work in a few areas that were unfamiliar to me.  Especially I had never looked at the innards of ELF files, nor at the process by which a core file is generated.  To add the necessary information, I had to create a new type of ELF note, and then add code to generate it to libproc (used by gcore) and elfexec (for kernel driven core dumps).  I also had to add support to parse this to libproc (for pfiles), and elfdump.  The project turned out to be bigger than anticipated, consisting of almost a thousand lines of code.  So it took me a little more than a day to complete it.

A webrev with the associated changes is presented for your enjoyment.  I'd appreciate any review feedback, and especially any further testing.

Here's a sample run:

garrett@openindiana{250}> cat < /dev/zero &
[2] 144241
garrett@openindiana{251}> pfiles 144241
144241: cat
  Current rlimit: 256 file descriptors
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:72712192
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
garrett@openindiana{252}> gcore 144241
gcore: core.144241 dumped
garrett@openindiana{253}> pfiles core.144241 
core 'core.144241' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:195125248
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
garrett@openindiana{254}> kill -11 144241
garrett@openindiana{255}> pfiles core
core 'core' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:419414016
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
[2]  - Segmentation fault            cat < /dev/zero (core dumped)

garrett@openindiana{256}> elfdump core

...

  entry [10]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        0
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IRGRP S_IWGRP S_IROTH S_IWOTH ]
        pr_uid:       0                  pr_gid:      3
        pr_major:     526                pr_minor:    0
        pr_rmajor:    67                 pr_rminor:   12
        pr_ino:       35127324
        pr_size:      0                  pr_offset:   419414016
        pr_fileflags: [ O_RDONLY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /devices/pseudo/mm@0:zero

  entry [11]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        1
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2

  entry [12]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        2
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2
...

Latest Apple EarPods - Success!

A couple of months ago I purchased the rather expensive pair of Apple in-ear earphones, to replace an older pair of standard earbuds I'd given to someone else.  They never fit my ear correctly, and I hated them.

Recently I've also taken up getting more physically fit -- so going to the gym more often, including spending a lot of time running on the treadmill.

But my earphones kept falling out.  And my iPhone 4 is kind of heavy and inconvenient.

So I picked up the iPod nano (7th gen) today at Best Buy.  And it comes with a new style of earbud.

I've been wearing them now for about 7 hours, including a fairly intense 40 minutes of running on the treadmill.  They haven't budged once.  And while I'm no audiophile, the audio quality seems to be better than any of the other earbuds I've used.

And the nano -- really light, really nice.  It doesn't do everything, and I've not tried out all the functions yet, but for playing music, it works great.  And, unlike my iPhone, it doesn't have a tendency to change songs or activate other functions while it is in my pocket.   So, +1 on the iPod nano too.

Just be aware that the earpods that come with the nano do not include the remote (and I think they lack a mic as well).  It looks like the $29 separate ones include the remote, and a mic.  I'll be purchasing a pair for other use... I wish they'd package them the same way the in-ear headphones are packaged though.  (Much easier to keep them stored tangle-free.)

GNU grep - A Cautionary Tale About GPLv3

My company, DEY Storage Systems, is in the process of creating a new product around the illumos operating system.  As you might imagine, this product includes a variety of open and proprietary source code.  The product itself is not delivered as a separate executable, but as a complete product.  We don't permit our customers to crack it open, both from the sense of protecting our IP, but also to protect our support and release engineering organizations -- our software releases consist only of a single file and we don't supply tools or source for other parties to modify that file.

One of the pieces that we wanted to integrate into the tree is an excellent little piece of software called Zookeeper, produced by the Apache organization.  Like illumos, Zookeeper has a nice non-viral copyleft license, which makes it nice for integration into our product.

However, I discovered that as part of our integration, one of my engineers had decided to integrate GNU grep.  Why? Because the startup script for Zookeeper apparently makes use of some GNU grep extensions that are not illumos' grep.  (The need for us to enhance illumos' grep utility, and the need to teach folks not to rely on non-portable GNU extensions, are both topics for another time.)

Fortunately, I caught this in time.  Because that one little "worm" - used simply to support a startup script, could have created a situation where we would have been required to open source our entire product.

Now if you're Richard Stallman -- this is your goal -- all source code is "liberated".

But if you're a business trying to create value, this is actively harmful.  Its almost impossible to build a product around GPLv3 code unless the only way you create value is through selling support.  (There are a variety of business reasons this is a bad model... with open code, 3rd parties can start "selling support" for your product, possibly giving your product a bad name, and generally leaching off your engineering effort.   In the end, without proprietary code, there is vastly reduced economic incentive to innovate -- you can't use innovation in software to gain a competitive advantage when all software is open.  I would argue that even the innovation that occurs in Linux exists largely due to economic pressures arising from attempts to compete against proprietary systems. )

So, while the correction here is not big for us -- just a small change to a startup script for now -- the hazard of these viral licenses cannot be understated.  If you make your living, as I do, writing software, then you should be very wary of the lure of code carrying the GPLv3 -- its a potential license bomb that may have nasty consequences for you later.

And if you're a producer of open source software -- as I am -- be wary of your dependencies as well.  While your code may be licensed under reasonably generous terms that support commercial uses, if your dependencies include things like GPLv3 GNU grep, you may find that your product has less wide acceptance than you intended.  Inclusion (by dependency or otherwise) of a GPLv3 product in your own product means that you are willing to abdicate any decisions about the licensing of your own product.

I'll get off my soapbox now...

Infographic - Programming

I was forwarded the following infographic from OnlineCollege.org.  While I'm not normally one to just regurgitate someone else's posting (especially advertising), I think this one is worth looking at -- especially if you're at a point where you're contemplating future career or learning plans.

illumos and ZFS day!

I'm pleased to announce that DEY Storage Systems, Inc., along with Joyent and other community members, is sponsoring the illumos and ZFS Day taking place in October 1st and 2nd at the Children's Creativity Museum in San Francisco.  My partners and I will be in attendance, along with a few other of my colleagues, and we'll be speaking on related topics.

Additionally, Joyent is hosting the 2nd illumos hackathon at their facilities on October 3, and we will definitely be participating there.  We had a blast doing this last year (some excellent work came out of it), and we're looking forward to doing it again.

If you can make, I hope you'll register.  The event itself is free, and while there will be online recordings of the material, its more fun to get together in person -- especially with a beer or three.  (Did I mention there will be a Solaris Family Reunion party on Monday night?

In many ways, I view this event as a belated 2nd birthday party for illumos, so I hope you'll join me in celebration.