Worst Company ... Ever

So I've not blogged in a while, but my recent experience with a major mobile provider was so astonishingly bad that I feel compelled to write about my experiences.  (Executive summary: never do business with T-mobile again!)

I had some friends out from Russia back in November, and they wanted to purchase an iPhone 5s for their son as a birthday gift.  Sadly, we couldn't get unlocked ones from the Apple Store at the time, because they simply lacked inventory.

So we went to the local T-mobile store here on San Marcos Blvd (San Marcos, CA.)  I knew that they were offering phones for full-price (not subsidized), and it seemed like a possible way for them to get a phone.  Tiffany was the customer service agent that "assisted" us.  She looked like she was probably barely out of high school,  but anyway she seemed pleased to help us make a purchase.  I asked her no fewer than three times whether the phone was unlocked, and was assured that, yes, the phone was unlocked and would work fine with Russian GSM operators.  I was told that in order for T-mobile to sell us the phone, we had to purchase a 1-month prepaid plan for $50, and a $10 sim card.  While I was little annoyed at this, I understood that this was their prerogative, and the cost of wanting to buy the device "right now" and not being able to wait 2 weeks for Apple to ship one.  (My friends had less than a week left in California at this time.)  We were buying the phone outright, so it seemed like there ought to be no reason for the device to have a carrier lock on it.

So, of course, the phone was not unlocked.  It didn't work at all when it got to Russia.  Clearly Tiffany didn't have a clue about the product she sold us.  I guess I shouldn't have been too surprised. But we had put down $760 in cash for a device that now didn't work.  Worst of all, I was embarrassed, as was my friend, when his son got what was essentially a lemon for his birthday gift.

I was pretty upset, and went back to the store.  Tiffany was there, and apologized.  She offered to pay up to $25 for me to use a third party service to unlock the phone since "it was her fault".  Of course, the third party service stopped offering this for iPhones, and their cost for iPhone 5's was over $100 when they were offering it.

I tried to get T-mobile to unlock the phone.  I waited in the store for a manager for about 2 hours, but Tiffany finally got upset and called the police to get me to leave the store after I indicated that I preferred to wait on the premises for a manager.   I left the store premises, unhappy, but resolved to deal with this through the corporate office.

I went home, and called T-mobile customer service.  It turns out that T-mobile store is really a 3rd party company, even though they wear T-mobile shirts, and exclusively offer T-mobile products.  (More on that later.)

I spent quite a few hours on the phone with T-mobile.  Somewhere in the process, the customer service people called Tiffany, whereupon she immediately reversed her story, claiming she had informed me that the phone was locked.  This was just the first of a number of false and misleading statements that were made to me by T-mobile or its affiliates.   I spent several hours on the phone that day.  At the end of the call, I was told that T-mobile would unlock my phone only after at least 40 days had passed from the date of activation.   The person at corporate assured me that if I would just wait the 40 days, then I'd be able to get the phone unlocked.  Since the phone was already in Russia, I figured it would take at least that long to send it back and send a replacement from another carrier.

Oh one other thing.  During that first call, I discovered that Tiffany had actually taken the $50 prepaid plan we purchased, and kept it for herself or her friends.  We didn't figure that out until T-mobile customer service told me that I didn't have a plan at all.  Nice little scam.  While dishonest, I would not have begrudged the action had the phone actually worked.  Of course it didn't.

(During all this, on several different occasions, T-mobile customer service personnel tried to refer me to the same 3rd party unlocking service.  Because hey, if T-mobile can't get its own phones unlocked, maybe their customers should pay some shady third party service to get it done for them.  But it turns out that whatever backdoor deal that service had previously doesn't work anymore, because they've stopped doing it for Apple phones.)

So we waited 40 days.

And then we called to have the phone unlocked.

And T-mobile refused again.  This time I was told that I needed to have $50.01 or more, not just $50 on the plan.  After a few more hours on the phone and escalation up through a few levels of their management chain, they credited my account for $0.20, and then resubmitted the unlock request.  I was guaranteed that the phone would be unlocked.

Two days later, T-mobile denied the unlock request.

At this point, I was informed that the phone had to have T-mobile activity on it within 7 days of the unlock request.  This was simply not going to happen.  The phone was in Russia!  I complained, and I spent quite a few more hours on the phone with T-mobile.  It seems like the folks who control the unlocking process of their phones have nothing to do with the people who answer their customer service lines, nor those people's bosses, nor those people's bosses.  Astonishing!

At this point I had spent over a dozen hours trying to get T-mobile to unlock the darned thing.  T-mobile had got their money from me already, and had done pretty much everything possible to upset me.  The amount of money T-mobile spent dealing on the phone with me, trying to enforce a stupid policy, which wouldn't have been necessary if they had just admitted their mistake and fixed it (which would have cost them nothing whatsoever) is astonishing.  Talk about penny-wise and pound foolish.

At that time, T-mobile told me that I would be able to return the other phone, provided I got it back from Russia.  They agreed to this even though the usual 14 days "buyer's remorse" window had passed.

So at this point, I went and purchased a phone from Verizon (and paid a $50 premium because I was at BestBuy instead of the Apple store and the Verizon store wouldn't sell the phone unless it was part of a plan), and I sent it to Russia with my step-son, who was going their for Christmas.  That phone did work, and my step-son exchanged the T-mobile phone for it, bringing the T-mobile phone back.

The next day, I went to return the phone at the store where I bought it.  Sadly, Tiffany was there.  So was her manager, Erica.

After spending about an hour at the store trying to return it, they agreed to take it back, minus a $50 restocking fee.  And as I paid cash, I had to provide my checking account information so they could do a deposit for me, which would take a few days.  I got a paper showing that they were sending me a refund, but nothing indicating the account number.  (Turns out it took a few calls from Erica -- who claimed to be the store manager and probably was about 10 minutes older than Tiffany -- the next day, since she apparently had no clue what she was doing and needed to get two separate pieces of information that she failed to collect while I was in the story.)

I did spend a bunch of time with customer service on the phone -- a few more hours I guess, trying to get that last $50.  At this point it was just a matter of principle.  I resented the whole thing, and I wanted as much of my money back as possible.  The customer service person tried to make it right, but because the store (Hit Mobile is the company apparently) is separate from T-mobile, the store's decision was final.  The store manager (Erica) refused to refund the stocking fee, and there was nothing T-mobile could do about it.  To their credit, they did offer me a $50 service credit if I was inclined to keep an account with them.  Needless to say, I have no interest in ever being a T-mobile customer, so I thanked them and declined... indicating that I'd prefer to have funds back in cash.  (Nobody ever mentioned the restocking fee; on the contrary I was told I'd receive the full purchase price less the $60 service plan and SIM card.  Again, story and reality don't match at T-mobile.)

I did eventually get $650 credited back to my account.

So, I was out $110, plus the extra $50, plus the 13+ hours of my time, plus the embarrassment and long turnaround time to get the replacement out to Russia.  All because T-mobile wouldn't fix a mistake they made, even though numerous people at that company recognized that it was clearly the Right Thing To Do.

Turns out that Verizon iPhones are all unlocked.  And I'm already a Verizon customer.  I was thinking about T-mobile's plans -- some of them are attractive on paper, and potentially could have saved me money relative the rather premium prices I pay for Verizon.  Needless to say, I will not be changing my provider any time soon.  In spite of the high prices, I've always been dealt with honestly and fairly, and I've been happy with the service I've gotten at Verizon.

If for some reason, you decide to get a T-mobile phone, please, please avoid the store located on San Marcos Blvd.  In fact, I urge you to verify that the store is owned by T-mobile corporation.  Its my belief that I might have had more satisfactory results had I been dealing with just a single entity.  That said, numerous people at T-mobile corporate lied to me and made false promises.

I feel so strongly about this, that I'm happy to have spent the hour or so writing up my terrible experiences with them.  I hope that I save someone else these painful experiences.  I won't be unhappy if it also costs T-mobile many potential customers.

For the record, I also think it should be illegal to sell a carrier locked device without clearly indicating this is so as part of the transaction.  The practice of carrier locking devices makes sense when the cost of the device is being subsidized by the carrier as part of a service plan.  But it should be possible to purchase the device outright and remove any such lock.  T-mobile's practices here are a disservice to their customers, and their partners.  The handset manufacturers should apply pressure to them to get them to change their policy here.

illumos corporate entity... non-starter?

I want to give folks a status update on the illumos corporate entity.

In a nutshell, the corporate entity seems to be failing to have traction.  In particular, the various corporate contributors  and downstreams for illumos have declined to step up to ensure that an illumos corporate entity has sufficient backing to make it successful.

While at first blush, this seems somewhat unfortunate, I think this is not nearly quite as bad a thing as it might first seem.  In particular, the failure of a corporate entity does not correlate to the health of the ecosystem -- indeed many successful open source projects operate without an umbrella organization or entity. 

Instead, we see corporate contributors and downstream distributions focusing on developing the communities behind their distributions such as SmartOS and OmniOS.  Those downstreams play an active role in improving illumos for the benefit of all, and its my sincere hope and belief that they will continue to evangelize illumos, and contribute to the common core.

Furthermore, incorporation in the state of California requires paying about $800 of taxes per year.  This is true even for organizations without any revenue.  This is money that would have to be taken from sponsors that would serve only to enrich our state government with no direct benefit to the illumos community.  (Non-profit status is a way around that, but its exceedingly difficult to obtain, and a number of open source organizations are finding themselves under very close scrutiny from the state and federal authorities.    Indeed, the X.org community themselves lost their 501(c)3 status not that long ago.)

So, without corporate sponsors to justify the tax and administrative load, I've decided that the illumos corporate entity should expire.  I do want to thank Deirdré Straughan for the non-trivial amount of effort she put into this, as well as the Software Freedom Law Center for the pro-bono work they did for us while we were trying to navigate the waters of becoming a true non-profit open source organization.

And, if any corporate sponsors are out there watching this, and interested in resurrecting the illumos organization, then I'm happy to entertain the possibilities.  I think there is value in an actual organization with a legal status to receive and distribute funds, and who can hold certain items of intellectual property, including the rights to the illumos trademark itself.   But there has to be enough of a sponsorship behind this to make it worthwhile.

In the meantime, I will continue to hold the illumos mark as a trademark that I keep in trust for the community.  The code is something that is already subject to distributed ownership, and so is completely unimpacted by any of this.

Thanks.

  - Garrett

Moving on (Adieu to Studio?)

I think illumos is at a key juncture, and the issue relates to our toolchain.

We have gcc 4.4.4 working thanks in large part to the efforts of folks like Rich Lowe.

We have historically relied on Sun Studio (now Solaris Studio) as our "base" or "default" compiler for C and C++ programs, and also to supply lint coverage.

I've been thinking for a long time that its past time we (the illumos community) moved on from this.  Not only are the Studio 12 compilers not available in source form, they are now not available in suitable form for building illumos as binaries either.  (Apparently it is possible under some terms to get Solaris Studio 12.3, but who knows if those compilers are suitable for building illumos.  In the past we have always needed specifically patched compilers for Solaris.)

The situation where a general developer cannot obtain the necessary tool chain for working with illumos is untenable.

Today we require "lint" as part of an official build.  But not just "any" lint, but lint generated by a specific patched older version of Sun Studio 12.  A version that is no longer available to the community at large.

(And in full disclosure, the problem for me is brought to a head by the fact that the machine I had my local installation of these tools on has been retired, and I find I don't have another backup of them readily at hand to install on my new machine.)

So, I'm placing this as a call to the illumos community at large, and especially to our RTI advocates.  Its time.  Really. 

Really.

Studio has to go.

I don't care if we leave the infrastructure in place for people that want to continue to use Studio, but we need to switch to gcc.  Rich's 4.4.4 version gets the job done for now.  It would be great if we could support newer versions, but I understand that this requires some non-trivial amount of effort (some gcc patches that need to be upstreamed, as I understand it.)

We cannot be tied to a closed tool chain; especially a tool chain that doesn't at least include binary redistribution privileges. 

For the record, I've posted the same content to the illumos developer and advocates lists.

Responding to Trolls

We have an... ahem... toxic personality in our midst in illumos.  I'm going to respond here to his most recent post, to avoid wasting the time on the illumos developers mailing list.  If you don't care about non-sense like this (although it may have entertainment value), please just ignore this post and move on.

The only reason I'm replying even here, is to set the record straight, and establish firmly the history -- at least my side of it -- for posterity.

In the early days of illumos, I invited those I could think of from the Solaris and OpenSolaris communities to participate.  I even reached out to some parties, like Jörg Schilling, who had a bad reputation as difficult people to work with.

I had hoped to repair those broken relationships since there was no longer a corporate controlling entity for illumos.  (Nexenta sponsored my efforts, and paid some of the early operating costs for illumos, but it has always been the case that illumos operates by and for the community -- and not at the behest of any corporate master.  As I've since left Nexenta to start my own company, and illumos has carried on, I think there is fairly ample evidence in support of this fact.)

I reached out, and tried to help Jörg with some integrations; but those integrations did not go well.  Jörg did not like our policy requiring a build or test of changes on the primary distribution (OpenIndiana at the time) used for the vast majority of developers and users, and so several of his changes did not get integrated.  (It turns out this was a good thing, because at least one of those changes, a "fix" for a keyboard problem, was mis-designed and would have broken international keyboards for the majority of illumos' user community.)

We also required code review, as we do now.

Jörg has long had a desire to have "star" integrated as the main replacement for all archivers.  That didn't take place, because he couldn't find reviewers.  And after several negative interactions with him, I withdrew my offer to sponsor his work personally, as I found the process of trying to do so unbelievably painful.  (There was an extremely long telephone call with Joerg that sticks in my memory.)  I suggested he find someone else willing to code review and sponsor his work for integration into illumos.  To date, that has not happened.

Somewhere during that time, Jörg indicated that he would not offer any other code up for integration until star was integrated.  Basically, he took his toys and went home.  Except he just took his toys, but forgot to go home, as he's still trying to create noise on our developer mailing lists.

There was another negative experience we had, mirroring today's thread, where Joerg's response to a tool I wrote (an open replacement for "od") was non-constructive criticism.  (In fact, his response was "it's broken, why don't you use this thing I haven't finished writing yet".  (He modified his "hexdump" utility to behave like od, apparently in response to my effort instead of simply offering helpful review feedback.)

This is typical of Jörg -- the only "right" answer as far as he is concerned is to accept his code, even if it isn't written yet or is buggy!

Jörg is what is known as a poisonous personality.  At least in every context that I've seen him operate. Which is why usually I try to just ignore him, and hope he'll go away; yet like certain garden pests he just keeps coming back.  He's even been kicked out of another prominent open source community.

He's also said made some incorrect and defamatory statements about both Nexenta and illumos at a prominent academic conference in Germany.  (I can't find the link right now, I'll update if I find it.)

So, where in his post today he asks Gordon if he's willing to change the rules for integration, I have this in reply:

Are you willing to change this? 

Gordon isn't empowered to.  Neither, for that matter, am I.  The rules that have been established for collaboration here are by common consent of the major contributors, and no single individual can change them.  Notably, those were the rules in force in Summer 2010, and you were the one that wanted special treatment.   Those rules have served this community well for the past several years, and there has been no movement to change them.

I am interested in setting up a phalanx of OpenSolaris activities because this
is the only way we have a chance to continue with OpenSolaris development but
this is hard as long as Illumos is not willing to play nicely.

OpenSolaris is dead.  The entire illumos community has moved on.  You have not, but then I have long since stopped considering you a part of the illumos community.
Especially given your repeated attacks against both illumos and numerous of its contributors -- even fairly early on you made a number of defamatory and incorrect statements about the nature of both illumos and its contributors in technical presentation before the German academic community.  This was not the behavior of someone who is a constructive contributor.
To be completely honest, I think I'm not alone in wishing that you'd come to the same conclusion: that is that you are not part of this community.

I recently tried to build a bridge by propising that Illumos should implement
the proposals from Summer 2010 to regain credibility. Unfortuntely there was no
reaction. 

That should speak volumes all by itself. 

As someone else in another open source community said, Jorg Schilling is damage; the community should route around him.

An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone.

This milestone was the integration of 195 Need replacement for nfs/lockd+klm

This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at Nexenta.  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix, and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product.

The reason this is such an important milestone, is two fold:

First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of the community.

Second, this effort represents the replacement of the last of the truly closed source code in our kernel proper.  There are still some closed source drivers (although I will point that we're now seeing more traction with vendors to supply open drivers, with recent open source code contributions coming from places like SolarFlare, LSI, and HP.  (HP's contribution -- through illumos partner Joyent -- means that at least one important closed source driver will soon be open -- cpqary3.)

I want to personally thank everyone who contributed to make the integration of the new open source NFS lock manager possible. You've done a big service to the community, and we are grateful!

ZFS Compression Enhancements in illumos

The ZFS code in illumos has gained another differentiator thanks to the hard work of Saso, who integrated LZ4 improved compression into our code base.

This is a nice achievement, and enhanced compression should be quite useful for most applications of ZFS.  If LZJB was always recommended to be turned on (it was by our team at least!), then LZ4 is even more recommended, unless all of your data is totally incompressible.  (Most data is highly compressible.)

You should check out his wiki post above, which contains a lot more details about the work, including some benchmark data.

hackathon project: pfiles - postmortem analysis

During the first days of October 2012, we had an illumos day, and a one day illumos hackathon that was very well attended by some of the best and brightest in our community (and I daresay, in any open source community.)

The purpose of the hackathon, besides just doing some cool projects, is to give folks a chance to interact with other domain experts, and code, particularly in areas outside their particular area of expertise.

This year a number of interesting projects were worked on, and I think some of these are at the point of starting to bear fruit.  The project I undertook was the addition of post-mortem analysis support for pfiles(1).  That is, making it possible to use pfiles against a core(4) file.  Adam Leventhal suggested the project, and provided the initial suggestion on the method to use for doing the work.

This is a particularly interesting project, because the information that pfiles reports is located in kernel state (in the uarea) and has not traditionally been available in a core file.

To complete this project, I had to work in a few areas that were unfamiliar to me.  Especially I had never looked at the innards of ELF files, nor at the process by which a core file is generated.  To add the necessary information, I had to create a new type of ELF note, and then add code to generate it to libproc (used by gcore) and elfexec (for kernel driven core dumps).  I also had to add support to parse this to libproc (for pfiles), and elfdump.  The project turned out to be bigger than anticipated, consisting of almost a thousand lines of code.  So it took me a little more than a day to complete it.

A webrev with the associated changes is presented for your enjoyment.  I'd appreciate any review feedback, and especially any further testing.

Here's a sample run:

garrett@openindiana{250}> cat < /dev/zero &
[2] 144241
garrett@openindiana{251}> pfiles 144241
144241: cat
  Current rlimit: 256 file descriptors
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:72712192
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
garrett@openindiana{252}> gcore 144241
gcore: core.144241 dumped
garrett@openindiana{253}> pfiles core.144241 
core 'core.144241' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:195125248
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
garrett@openindiana{254}> kill -11 144241
garrett@openindiana{255}> pfiles core
core 'core' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:419414016
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
[2]  - Segmentation fault            cat < /dev/zero (core dumped)

garrett@openindiana{256}> elfdump core

...

  entry [10]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        0
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IRGRP S_IWGRP S_IROTH S_IWOTH ]
        pr_uid:       0                  pr_gid:      3
        pr_major:     526                pr_minor:    0
        pr_rmajor:    67                 pr_rminor:   12
        pr_ino:       35127324
        pr_size:      0                  pr_offset:   419414016
        pr_fileflags: [ O_RDONLY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /devices/pseudo/mm@0:zero

  entry [11]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        1
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2

  entry [12]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        2
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2
...

Latest Apple EarPods - Success!

A couple of months ago I purchased the rather expensive pair of Apple in-ear earphones, to replace an older pair of standard earbuds I'd given to someone else.  They never fit my ear correctly, and I hated them.

Recently I've also taken up getting more physically fit -- so going to the gym more often, including spending a lot of time running on the treadmill.

But my earphones kept falling out.  And my iPhone 4 is kind of heavy and inconvenient.

So I picked up the iPod nano (7th gen) today at Best Buy.  And it comes with a new style of earbud.

I've been wearing them now for about 7 hours, including a fairly intense 40 minutes of running on the treadmill.  They haven't budged once.  And while I'm no audiophile, the audio quality seems to be better than any of the other earbuds I've used.

And the nano -- really light, really nice.  It doesn't do everything, and I've not tried out all the functions yet, but for playing music, it works great.  And, unlike my iPhone, it doesn't have a tendency to change songs or activate other functions while it is in my pocket.   So, +1 on the iPod nano too.

Just be aware that the earpods that come with the nano do not include the remote (and I think they lack a mic as well).  It looks like the $29 separate ones include the remote, and a mic.  I'll be purchasing a pair for other use... I wish they'd package them the same way the in-ear headphones are packaged though.  (Much easier to keep them stored tangle-free.)

GNU grep - A Cautionary Tale About GPLv3

My company, DEY Storage Systems, is in the process of creating a new product around the illumos operating system.  As you might imagine, this product includes a variety of open and proprietary source code.  The product itself is not delivered as a separate executable, but as a complete product.  We don't permit our customers to crack it open, both from the sense of protecting our IP, but also to protect our support and release engineering organizations -- our software releases consist only of a single file and we don't supply tools or source for other parties to modify that file.

One of the pieces that we wanted to integrate into the tree is an excellent little piece of software called Zookeeper, produced by the Apache organization.  Like illumos, Zookeeper has a nice non-viral copyleft license, which makes it nice for integration into our product.

However, I discovered that as part of our integration, one of my engineers had decided to integrate GNU grep.  Why? Because the startup script for Zookeeper apparently makes use of some GNU grep extensions that are not illumos' grep.  (The need for us to enhance illumos' grep utility, and the need to teach folks not to rely on non-portable GNU extensions, are both topics for another time.)

Fortunately, I caught this in time.  Because that one little "worm" - used simply to support a startup script, could have created a situation where we would have been required to open source our entire product.

Now if you're Richard Stallman -- this is your goal -- all source code is "liberated".

But if you're a business trying to create value, this is actively harmful.  Its almost impossible to build a product around GPLv3 code unless the only way you create value is through selling support.  (There are a variety of business reasons this is a bad model... with open code, 3rd parties can start "selling support" for your product, possibly giving your product a bad name, and generally leaching off your engineering effort.   In the end, without proprietary code, there is vastly reduced economic incentive to innovate -- you can't use innovation in software to gain a competitive advantage when all software is open.  I would argue that even the innovation that occurs in Linux exists largely due to economic pressures arising from attempts to compete against proprietary systems. )

So, while the correction here is not big for us -- just a small change to a startup script for now -- the hazard of these viral licenses cannot be understated.  If you make your living, as I do, writing software, then you should be very wary of the lure of code carrying the GPLv3 -- its a potential license bomb that may have nasty consequences for you later.

And if you're a producer of open source software -- as I am -- be wary of your dependencies as well.  While your code may be licensed under reasonably generous terms that support commercial uses, if your dependencies include things like GPLv3 GNU grep, you may find that your product has less wide acceptance than you intended.  Inclusion (by dependency or otherwise) of a GPLv3 product in your own product means that you are willing to abdicate any decisions about the licensing of your own product.

I'll get off my soapbox now...

Infographic - Programming

I was forwarded the following infographic from OnlineCollege.org.  While I'm not normally one to just regurgitate someone else's posting (especially advertising), I think this one is worth looking at -- especially if you're at a point where you're contemplating future career or learning plans.

illumos and ZFS day!

I'm pleased to announce that DEY Storage Systems, Inc., along with Joyent and other community members, is sponsoring the illumos and ZFS Day taking place in October 1st and 2nd at the Children's Creativity Museum in San Francisco.  My partners and I will be in attendance, along with a few other of my colleagues, and we'll be speaking on related topics.

Additionally, Joyent is hosting the 2nd illumos hackathon at their facilities on October 3, and we will definitely be participating there.  We had a blast doing this last year (some excellent work came out of it), and we're looking forward to doing it again.

If you can make, I hope you'll register.  The event itself is free, and while there will be online recordings of the material, its more fun to get together in person -- especially with a beer or three.  (Did I mention there will be a Solaris Family Reunion party on Monday night?

In many ways, I view this event as a belated 2nd birthday party for illumos, so I hope you'll join me in celebration.

The Case for a new Reference Distro for illumos

With the recent announcement and discussion around OpenIndiana, I think its time to bring to the light of day an idea I've been toying with for a while now -- and have discussed with a few others in the community.

illumos needs a reference distribution.

Now just hold on there... before you get all wound up, let's discuss what I mean by a reference distribution.

• A reference distribution should be a base for additional development, and testing. 
• It should support automated installation for mass deployment in test lab/QA/build farm scenarios.
• It must support "self-hosting" -- i.e. illumos-gate itself should be buildable on this distribution -- with as few additional steps as possible.
• It must be valid as a QA platform -- for example, it needs to support the packaging tools we use in order to validate the packaging metadata (manifests).  (Yes, this means it needs to support - and be packaged with - IPS -- in spite of my loathing of IPS itself.)
• It should be a reasonable (but not the only reasonable possible) base to use for other distributions looking to build on top of it.
• Ideally it should be pretty darn minimal.
• It may have graphics support, e.g. an X server, but only in so far as having that would support validation of technologies in illumos-gate.  (Sorry, no big desktops, Gnome, web browsers, or any of that crap.)
• Community driven - it contains the things that the entire community agrees upon as our common core.
• It must require only a modicum of effort around release engineering - none of these multi-week release cycles.
• It should be released biweekly with a numbered build, and probably also have nightlies.  These should be automated.
• Multi-platform - i.e. it needs to support both SPARC and x86.

As a common reference, it must use illumos-gate verbatim, i.e. no local changes.

Here's what it is not:

• A general purpose desktop.
• A general purpose server.
• A hypervisor.
• An appliance.
• Subject to the whims of any corporate entity.
• An experimental version (it isn't Fedora!)

Why do we need this?  OpenIndiana has been filling the role - but OpenIndiana is insufficient for several reasons:

• OpenIndiana is too big and unwieldy -- both to build, and to install.
• OpenIndiana maintains their own fork of illumos-gate.
• OpenIndiana needs the freedom to follow their own course - without being too strictly tied to illumos-gate.
• OpenIndiana lacks any kind of formal QA.  (The recent debacles around 151 are good examples of this.)
• OpenIndiana is not built for SPARC.

(Some of the above items could be addressed with OI, but I think addressing all of them is nigh impossible without totally changing the role of OI.)

So, here's what I propose we do instead.

• I propose that we start with OpenIndiana or OmniOS, but using core illumos-gate (no fork), and we remove things that are not relevant to the reference.  E.g. Gnome, etc.
• We will need lots of assistance with release engineering - I want a formal RE plan in place for this thing, with regular scheduled builds and releases following the "train" model rather than the "station wagon" model.
• We will need to set up some central build and test servers.  (I hope to get some funding for this through the Foundation).
• We will need to have formalized QA.  (Imagine a Jenkins instance that ran a build and regression test group every night.
• The bug database should add fields to reference formal build numbers.  (Introduced-In, Fixed-In, Reported-In, etc.)
• We do not need "an installer".  So the image can just be a basic ISO or tarball, probably.  Think minimalist.  I don't want any customization in the installer, excepting possibly the identification of the initial disk to install to.

Ultimately, if we're very successful, then this will help the entire illumos community.  This reference may ultimately be useful to become a base for OpenIndiana or OmniOS to build upon.  Put another way, I want to make OI's job easier, but I don't intend that this should replace OI.

So, what do I want from the community?

I want to hear from people who think this is a terrible idea.  A descriptive criticism of concerns is far more useful than just another +1.  (If you want to +1 it, do it using Google or FB to like it -- don't add a +1 comment.)  Post your concerns directly to this thread, so that we can keep them for posterity.

I want to hear from people who want - and are able - to actively contribute to this.  Most of this effort will be volunteer based initially, but I think we'll eventually need to staff some work from the Foundation.  Corporate sponsors may help here as well, either by loaning of resources, or gifting of earmarked funds.

Btw, for folks that want to debate names -- let me kill that now.  I want to name this "IRD" -- illumos Reference Distribution.   Its about as unsexy as possible - and that is the point -- this thing should be viewed as something the community uses internally, and not something that is ever given marketing attention.  (Those spot lights belong to illumos itself, and to the other end-user oriented distributions.)

To be clear, I'm not 100% set that this is all the way to go -- but I'm heavily leaning in that direction.  While I can't force the rest of the community to follow along, I can allocate resources to it, and I want to make sure that we can address any major concerns before I start investing resources here.

Thanks.

Response to Alasdair's Resignation

[ Recently, Alasdair Lumsden announced his retirement as leader of the OpenIndiana community.  This is a copy of my response to that. ]

Dear Alasdair,

Here at the illumos Foundation, we are very sad to hear of your resignation as lead of the OpenIndiana distribution. You have demonstrated great leadership and have been a wonderful friend of the community. We understand that such projects can be very time consuming, and we wish you all the best in your other endeavors. 

As a community, many of us share some of your frustrations. At times, progress does appear to be slow. At other times, there may appear to be a lack of interest from many of the former Sun employees. However, although you may feel that there is a lack of support or interest, many of us active in the community feel differently. While OpenIndiana served to carry on the banner of the OpenSolaris distribution, it was obviously a dead end because the desktop wars are over, and even the Linux community could not win. Meanwhile, the flourishing embedded and special purpose markets for illumos technology continue to grow. 

For example, use of illumos technology is growing in the areas of internet and cloud infrastructure, database and storage appliances, and embedded products. Many successful companies are driving the technology forward and the future has never looked brighter for open source developers looking to contribute to the community and find interesting jobs.

Although you may think we don’t see that the expanding free and open source software marketplace is not our total playground, what we do see is a talented, dedicated group of volunteers committed to seeing illumos become an important part of specialized, niche technology solutions, especially for solving critical problems facing the Internet. 

Honestly, we do not expect the illumos core itself to be economically viable. We do expect--and see--distributions built upon the illumos core becoming commercially successful and growing. This model is similar to the successful Linux model, where the distributions can achieve different results based on the needs of the constituents or the community; some commercial, some less so. 

The model of OpenSolaris is broken. The model of OpenIndiana following OpenSolaris is broken. The illumos model is following the successful Linux model. This is exemplified by distributions such as the commercially supported, general purpose OmniOS, Joyent’s SmartOS, Delphix OS, and others. 

In the future, we expect illumos to become a bigger part of the technological community, as many companies now emerging from stealth mode are incorporating our core platform functions. As our community grows, and word continues to spread of our stability, dependability and scalability, we hope to take illumos to even more specialized markets. 

Among the illumos community of volunteers are some of the smartest, most talented, most dedicated and hard-working people I’ve ever had the pleasure of being associated with. Their years of experience, combined with their insight into the serious problems facing the Internet, assure a rock-solid future for illumos. As a community, we have a shared set of values, and while sometimes we’ve had our disagreements, it’s always been those shared values that have come through and helped the community - and the technology we are building - grow and improve. We will continue to build an enterprise-grade, highly dependable operating system. 

Again, we are saddened by your leaving. On a personal note, I have enjoyed the collaboration that we shared in the formative days of illumos and OpenIndiana, and I hope we will be able to do so in the future.  We thank you for all your efforts on behalf of OpenIndiana and illumos, and know that your efforts have not been in vain; we will take all that we have learned under your leadership and build on it. As a natural evolution of the community, distributions come and go; we take the lessons and move on. Our community has never been one to back down from a challenge; we will handle this in stride and continue to move forward with our mission and our future goals. 

Sincerely,

Garrett D’Amore

illumos Founder

Women in Tech Adverstising - a rebuttal

So, there has been some that have argued vehemently against the advertising policies of GoDaddy, and its ilk.  (See https://twitter.com/bdha/status/211210555577466881 for example of the complaints.)

I have a different point of view.  What about sites that always picture a beautiful woman on a headset for their representation of tech support?  Should we also boycott them?  Would you prefer a site with an ugly person answering the phone?  After all, it's a phone or text chat, so it shouldn't matter at all what the person looks like, right?

Conversely, what about all the sites that use men in their advertising?

My hosting provider, bluehost.com, uses both.  Take a look.  Does the pretty blonde in glasses have anything to do with domain names or web hosting?  Of course not.  The guy wearing the headset for the tech support line isn't exactly hideous either, if you swing that way.  Are you likely to talk to this guy if you call?  Probably not -- he's probably just another model.

GoDaddy uses Danica Patrick as their spokesperson.  She's admittedly beautiful, a professional athlete (she drives race cars), and probably makes a fair bit of money endorsing GoDaddy.  Some of the ads she has been in have admittedly been a little tongue in cheek in their appeal to a mostly male clientele, but I think it would be a stretch to say that GoDaddy is using sex to sell domain services.

It would be an even further stretch to say Danica is being exploited by GoDaddy.

(Note that there may be other reasons to avoid doing business with GoDaddy or its CEO, but I am only referring to the use of Danica Patrick, a female spokesperson, and women in advertising in general.)

Would the people who claim that this is somehow evil objectification of women complain as loudly if the spokesperson were a good looking baseball star?  Is this objectification of men, or of male sports stars?  What if the ad somehow appeals more strongly to women, or to gay men?

What if the site featured an animal spokesperson prominently?  Say an orangutan?  (Perhaps with computer animation. :-)  Would this be considered animal exploitation?

I submit to you, dear reader, that each one of these cases are basically the same.  We use humans, animals, or whatever, to provide a human element and appeal in a product being sold.  Sure, it would be nice if the site instead featured some kind of graphic that demonstrated its technical superiority, but come on -- we're talking about a domain registrar, which is about as boring as tech gets.  I can hardly blame GoDaddy for trying to liven up their site and generate some enthusiasm.

In fact, even in the more extreme cases, such as the use of bikini-clad booth babes, I submit that folks who think objectification like this is somehow exploitative are misunderstanding.  These women are very rarely without many choices and options.  Usually by the time we see them, its when they are in a job that they have worked hard to get -- maintaining a perfect figure can be a full-time job, after all.

If folks like author of the aforementioned tweet had their way, they would deny these women the opportunity to earn a living showing off their own accomplishments.  Isn't that even more exploitative?  In fact, in the extreme case, you could consider the more repressive societies of the middle east?  Is this the social ideal to which we should aspire?  Where the only part of woman we see in advertising is a burqa?

(Note that in the case of the site using bikini-clad booth-babes, my gut response is to question what the company is not showing -- their product.  So while the babes may catch my eye, they have yet to draw me to a booth I would not otherwise have visited for the products featured.  Actually, at one former employer a female colleague -- also quite beautiful, but quite knowledgeable -- is aware that she is often mistaken as a booth babe, and regularly uses this to her advantage; customers get surprised somehow when the beautiful woman knows more about the technology than most other men do.)

As for me, I welcome the diversity of our society, and the appreciation of the human form.  If some people are able to use this to humanize their marketing approach, good for them!  Let's face it, I'd rather look at beautiful faces online than ugly ones.

Far better, IMO, for those who would fight against this spend their energies worrying about the women who are truly exploited or repressed -- in the fundamentalist muslim societies of the middle east, in the brothels, and human slave trade.  Don't feel sorry for the woman who is able to make a nice living for herself displaying the body she has worked hard to get and to maintain, and that she is proud of.

And yes, for the record, I usually like Carl's Jr. commercials too.

Ivy Bridge Motherboards -- Don't *Really* Exist!

So, if you're like me, and you've been shopping for a new system lately because you're last one is dying or dead, you might see some fancy looking Intel E3-1200v2 Xeon systems.  (Why Xeon?  Because ECC memory is a good thing.  I attribute some of my difficulty with my last system to using non-ECC memory.  Because its not ECC, I don't really know whether the problem was in RAM or CPU, but I digress...)

Great, so you pick out a E3-1240v2 Ivy Bridge CPU, and then you look for a System Board.

I found the Supermicro X9SCA-F-O.  Looks like a sweet board.  The spec claims support for the E3-1200v2 cpus, and 1600MHz RAM.  It has a separate IPMI LAN support, as well.  Sweet!  (Look closer though....)

So, package arrives from NewEgg, and you're ready to go.

Assemble the kit.... if you're a software guy like me this might take a couple of hours.

Power it on.

Beep!  Beep!  Beep!  Beep!

WTF does that mean!?  Google "4 beeps supermicro".

Oh, "Unsupported processor."  Further reading....

The system needs a BIOS update (to v2.0) in this case.  Seems reasonable.

Download the instructions....

"Step 1. First, boot the motherboard with an E3-1200 series (not E3-1200 V2)
processor..."

WTF!?!  A new E3-1200 (that's Sandy Bridge mind you, still fairly recent stuff) processor is going to cost me like 200 bucks, minimum.  And I can't return it after I buy it except to exchange it with a different CPU.  But I already have the E3-1240v2.

Anyone want to buy a very lightly used E3-1220 from me?  It looks like I'll be purchasing one tomorrow. Ultimately, I can't afford not to burn the $200, because I simply don't have the time to keep screwing around this stuff.  I've burned the better part of today trying to wrangle hardware.  (Hmmm.... where's a Systems Engineer when you need one?)  Conversely, anyone got one lying around they want to give, loan, or sell really cheap?

Shame on you Supermicro.  There are at least several things you could have done better here:

1. Make it much clearer that if buying a unit, it should include a legacy CPU first.  (I.e. that Ivy Bridge support is only for people upgrading CPUs and not building new systems.)

2. Have a separate SKU for systems that have the requisite BIOS changes.  Even if the silk-screen on the board is the same (and it shouldn't be, because the silk screening indicates only 1066 and 1333 MHz RAM supported, although the BIOS enables 1600 MHz), having the separate SKU lets buyers make sure they have the bits needed.

3. Work with retailers to ensure that existing stock is returned and upgraded as soon as possible.

4. Offer an expedited RMA process for people like me who are screwed in the meantime.  (To be fair, I've not called supermicro yet, but Googling tells me people have not had good luck with this option yet.  The only solution that has worked for people is to get their own 2nd processor.)

5. Technically, this board has an IPMI unit on it.  I think that means it has a separate microprocessor. Why can't I update the BIOS using that?  That would eliminate the need to buy a second CPU, at least for those of us with IPMI enabled gear.

Lessons for me:

1. Its probably worth a couple hundred bucks to let someone else put together the system for me.  I'm a driver guy -- futzing around with actual bits of silicon and cabling, its not my forte.

2. There's a reason it's called the bleeding edge.  I think I got cut with it today.

Why I *hate* external dependencies!

So, I've been trying to setup a system that can build illumos-gate.  Because my old system had a memory DIMM fault, and isn't usable anymore.

I thought, hmm... let me try something other than OpenIndiana.

Big mistake.

Only OpenIndiana seems able to build vanilla illumos-gate right now.

Why?  Because of external dependencies!  I tried building on OmniOS.  The stopper there -- Python 2.4!  On SmartOS, the dependency is IPS itself.  It seems only OpenIndiana is suitable for building stock illumos-gate.

The problem is that our build is inherently very sensitive to the build environment.  It makes life incredibly unpleasant if you try to build on any system that is not configured exactly as we specified.

This, IMO, is an unacceptable situation.

I will speak loudly against any attempt to make changes that introduce further external dependencies.  The fact that some already exist is no excuse.  Every external dependency makes working on the tree more painful.  If you're thinking of a change that would move something out of tree, but add a build-server dependency, think again!  

These dependencies are a serious barrier for contribution.

We (illumos) should take a page from NetBSD, who have successfully arranged that their environment is self contained to the point that they don't care about their build server at all -- all it needs to be able to do is bootstrap gcc and run automake/autoconf.  There is something very elegant about this.

Expect to see me on yet another crusade to kill dependencies on external runtimes, be it Python, Perl, or whatever.   Our tree should, IMO, be as "stand-alone" as possible.

If that means you have to write C, so be it.  Grow up and get a real compiler dude.  I know languages like Python and Perl are attractive because they have a lower barrier to learn -- but depending on them -- and far worse -- depending on *specific* versions of them -- is toxic for anyone else who wants to work on the code in a different environment.   (This has nothing to do with your language of your choice, but everything to do with the pain and suffering your language choice creates for anyone who just wants to do "make install".)

illumos hackathon a resounding success

Yesterday we had our first ever illumos hack-a-thon. About a dozen people from the community showed up, and worked on some very very cool things. At the end of the day, about a half-dozen different demos were done, to show what was worked on.

We'll be posting photos shortly. But here are some of the projects that got attention:

• tab completion for dcmds and data types in mdb
• ::print for DTrace
• expanded truss support for ZFS ioctls (nvlist expansion)
• time-ordered output for DTrace
• a comment field in the vdev label for ZFS pool devices
• the ability to change the GUID of a ZFS pool

Several other projects were in progress.

We selected these projects out of a much larger list of project proposals (which I'll post soon), based on the what people thought was most useful, and the ability to achieve results in a single day hack-a-thon. (And what people we willing to either work on, or mentor.)

Most importantly, people got to work on areas that they weren't intimately familiar with, and work with mentors who were much more familiar. For my own part, I had a blast working with George Wilson to implement the ability to alter the GUID of a pool (which is going to have some further uses we can talk about later). My webrev of these changes is now on-line, so I welcome review feedback.

We had domain experts like Adam Leventhal, Matt Ahrens, Eric Schrock, George Wilson, Gordon Ross, and Dan McDonald on hand, to help mentor projects if they needed it, and the amount of cross fertilization was amazing.

While none of the actual code changes are absolutely earth-shattering, they are still very very cool, useful things, that many of us will be happy to have available. I'm already imagining several useful use cases for the ability to reguid a pool, for example. And I can't wait until I have ::print in DTrace and tab completion in mdb. These will make my life significantly better.

Ultimately, this was the most enjoyable coding experience I've had in a long time. I can't wait until we do it again!

NexentaStor 3.1 available now

After a long, and arduous, release cycle, I am pleased to report that NexentaStor 3.1 is available now.

Customers running existing 3.0 installations may upgrade at no cost.

This release includes a number of key features, including some significant improvements for performance and manageability.

Folks using SCSI target mode will probably see the biggest performance boost relative to earlier editions of NexentaStor, especially those folks using NexentaStor to serve up storage to VMware guests -- thanks to the VAAI offload support that is part of this release. And for the record, yes, this release includes the fix the long standing problem with iSCSI timeouts.

For ZFS fans, this release also includes the updates for ZFS version 28. (This does mean that folks upgrading need to be cautious -- their pools will not be automatically updated to ZFS version 28, but if they are manually updated then there will be no way to move those pools back to an older release. That also means that pools should not be updated in HA cluster configurations unless both cluster partners are updated to NexentaStor 3.1 first.) This also means faster snapshot creation and deletion, and the ability to import pools in read only mode or that have encountered a failure of their SLOG device, making some disaster recovery scenarios much less challenging.

Nexenta Core Platform (our general purpose Debian based open core) will be updated shortly to 3.1 as well -- probably sometime in the next week or so. This will be the same core software as supplied with NexentaStor. (We do strongly encourage folks using Nexenta Core Platform for serving up storage to use our commercial NexentaStor product. There is even a free edition for users with up to 18TB of data.)

We are already (and have been for a while actually) hard at work on the next release, which will be based upon illumos, and include a number of other innovative features. Stay tuned for an update.

Bombproof taskqs

As part of fixing some recent bugs, I integrated the following into illumos:

734 taskq_dispatch_prealloc() desired

943 zio_interrupt ends up calling taskq_dispatch with TQ_SLEEP

The interesting one is the first of these. The interface is actually called taskq_dispatch_ent(), and is private to the "consolidation" (i.e. for use within bundled code only).

What this interface provides for, however, is a way to bomb-proof your taskq dispatches, if you can arrange for the dispatching state structure (taskq_ent_t) to be allocated in advance. This means you never have to worry about the possibility of a dispatch failing due to insufficient resources.

What's even cooler, is that the cost of the dispatching is much cheaper; taskq_dispatch() was the hottest piece of code on a very busy storage server. Now it goes much much faster, because it is just twiddling some linked list pointers and sending a signal to wake up the thread processing the taskq.

More importantly, the fact that we don't ever have to sleep, or have an expensive call into the kernel memory subsystem, has solved some frustrating "stalls" in the storage subsystem.

I encourage developers working with taskqs in illumos to have a look at this interface. If you can use it, you will simplify your code, and shorten your run-paths. Both of which are good things. The only limitation: this interface is not available for dynamic taskqs. (Which makes sense since dynamic taskqs might need to allocate whole threads.)

illumos podcast

Constantin Gonzalez recently interviewed me for his "HELDENFfunk" podcast series, while I was in Amsterdam for our European User's Conference. Also interviewed were OpenIndiana founders Alasdair Lumsden and Andrzej Szeszo.

illumos Panel Discussion in SF Bay Area

All:

There will be a panel discussion about illumos as part of the San Fransisco and Silicon Valley OpenSolaris User's Group meeting tomorrow, June 15. I will be joining Bryan Cantrill and Adam Leventhal to answer your questions about illumos.

The doors open at 6:45pm, and we will continue until about 8pm.

The location is 275 Middlefield #50, Menlo Park, California. Hope to see you there!

NexentaStor 3.0.5 available now

NexentaStor 3.0.5 is now available.

Apart from fixing some key bugs, the main thing that this release includes is a significant update to the CIFS stack, which addresses both performance concerns, and AD failover concerns.

Note that NS 3.1 is due out *any day now*, and will include all these changes, plus a boat load of others. I'll have a lot more to say about the 3.1 release soon.

GSoC Candidates Selected

You may be aware that we have selected two candidates for the slots allocated by Google to illumos -- the first is to replace some system utilities from code in perl to native C. The second of which is to bring GRUB2 to illumos.

What you may not know, is that Nexenta will be sponsoring three additional candidates to pursue projects of their own to benefit illumos. These candidates have been selected already, and we will have more to say about them and their work in the future. Stay tuned!

Thanks again, Joyent!

Joyent have continued to demonstrate their commitment to and support of illumos.

In addition to a string of recent source code integrations, they are now hosting some of our infrastructure in their cloud, with more to follow.

After moving the stuff there, we're now enjoying significantly better performance, and enhanced functionality. Try out the new OpenGrok instance yourself to see!

I'd also like to give a special thank you to Circonus, who are providing active monitoring services for our site now, as a gratuity to illumos. Apparently, they're going to be hosting their stuff on illumos based systems as well, so there's additional synergy here.

What is this OSUNIX thing anyway?

So there has been some things brewing in a sub-sect of the illumos community about a project to fork illumos, because of alleged problems with my leadership. You can read the thread here if you want.

I want to address this head on.

First the claim is that I've got omnipotent control over illumos. This is absolutely false. While I created the project, and serve as technical lead, I've offered to step down if the developer-council and admin-council would like to me to do so. Notably my employer (Nexenta) has minority representation on both councils, and I've tried to keep the groups as neutral as possible. I said when I created the illumos project, and I still maintain, illumos is a community project, not a Nexenta one.

I'm working on the process to make this more formal through non-profit governance. I should have more to say here before the end of week. (I've got a meeting about this today.)

I've also handed over determination of the Advocate list (the list of people who get to approve and integrate submissions) to developer-council. So far Nexenta has 75% of the advocate slots, but this can change at the request of developer-council. Since about 75% of the contributions to the illumos code have come from my team at Nexenta, this should hardly be surprising. In fact, I've flatly refused to add any more Nexenta advocates, even though there are meritorious candidates, until we get broader representation here. (Becoming an advocate requires making a number of good, well-formed, contributions. And it requires people willing to perform thorough review.)

There is a claim that I've somehow driven companies away from illumos. I hope not. As of now, I'm not aware of any companies that have requested to participate or contribute, who I've turned away. In fact, the only contributions that have been turned down have been Joerg Schilling's star project (he couldn't find people willing to review the code) and the ksh93 update (which has been unable to pass a technical code review -- ultimately we'll probably take in the ksh93 changes in more piecemeal fashion breaking them apart into reasonable and reviewable integrations instead of a 100KLOC+ set of code of varying quality.) As far as I know, everything else is vaporware.

I'd love to know what companies I've driven away, and what I did to do so. Honestly, if there is constructive criticsm here, then I want to hear it because I want to a better job -- and I want illumos to be as inclusive as possible. The fact that nobody has come forward (and nobody has approached me privately either!) makes me wonder how much this is really happening.

In fact, I have done all I can to encourage contribution, and to give credit for such contributions where it is due. And indeed, we have contributions from Joyent, Areca, and others. And a number of things queued up from names like Intel and LSI.

At the end of the day, if the project forks, so be it. Forks aren't necessarily a bad thing, and if a fork means we get more contributors to the ecosystem, then I welcome it. But I hope that the basis for such a fork is not just because one or two people don't like me.

(For the record, I am perfectly happy that I'm not everyone's favorite person... my job is to do the best I can for the future of the project, not to be the most universally well loved person. The open source world is filled with other personalities who people have strong feelings about -- Linus Torvalds, Richard Stallman, Theo De Raadt, Andrew Tridgell, and I don't think that the projects they lead have suffered any for it.)

Anyway, I hope that explains my position. If someone wants to have an open dialog with me about any of this, I'm happy to do so. I don't monitor the OSUNIX lists normally, but I'm reachable via email, IRC (gdamore), this blog, twitter (gedamore), and the developer list on illumos.org.

Thank you Joyent!

Joyent posted an update -- they've released a branch of illumos, on github, containing much of their illumos contributions.

Some of the stuff is probably fairly Joyent specific, but some of it is highly useful to almost everyone using illumos!

From their mail:

• ZFS I/O fair-share scheduling for zones
• the Joyent brand, which can be used as a template for other non-SysVR4 or IPS zone brands
• Reintroduction of sparse zone images
• Crossbow vnics on demand for zones & non-unique vnic naming (unique per zone, not per system)
• svcs enhancements ( svcs -Z/-z for interrogating zone services, -L for outputting log files directly (no more ls /var/svc/log | grep... ))
• vfsstat and iostat tweaks and ziostat, iostat(1M) for ZFS I/O
• more per-zone IO kstats
• the zonemon utility for zone kernel state troubleshooting
• DTrace enhancements such as llquantize

I just want to say again, thank you very much Joyent! Now, how quickly can we merge this stuff into illumos mainline?

CFV: illumos content authors

I'm looking for people interested in contributing content to the illumos website. Right now we have a test website but it needs help with producing content. First and foremost we need English content, but the new framework will support other localizations as well.

If you're interested in contributing here, drop me an email. I'll be setting up a mailing list for this soon.

Thank you Areca! 6Gbps 1880 support in illumos

A big thank you goes out to Areca.

Areca have provided the source code for their Solaris driver, including support for the newer 6G RAID adapters. As a result, I've integrated a (somewhat cleaned up) copy this code as an update to arcmsr(7d) in illumos, under generous open source licensing:

changeset:   13305:fb26af81b9b2
tag:         tip
user:        Garrett D'Amore 
date:        Fri Mar 25 22:14:56 2011 -0700
description:
      834 need support for Areca 1880 6Gbps
      Reviewed by: Dan McDonald <danmcd@nexenta.com>
      Reviewed by: Albert Lee <trisk@nexenta.com>
      Reviewed by: Richard Lowe <richlowe@richlowe.net>

This will make another HBA option available to folks. (Note these cards do support a JBOD mode, so you don't have to use hardware RAID -- indeed I would recommend that you don't when you have ZFS on the disks.

Another outlet..

So, at the recommendations of others, I'm on twitter now.. Don't know how often I'll keep it updated, but I'll try.

illumos has Serbian Family Language Support

I just integrated:

changeset: 13312:537259ad27f6

tag: tip

user: Garrett D'Amore

date: Wed Mar 23 08:35:14 2011 -0700

description:

324 need serbian locale support

Reviewed by: Rich Lowe

Approved by: Garrett D'Amore

This is a bit unusual relative to most of the locales, because Serbo-Croatian is a language fraught with some unique political considerations:

There is a common root language, that everyone speaks and understands. But speakers of it rarely agree on what to call it. In Serbia its Serbian. In Bosnia its Bosnian. And so on for Croatian and Montenegrin.

In illumos, we have followed the Unicode CLDR example, and we now have these locales:

hr_HR.UTF-8 - Croatian in Croatia

sr_BA.UTF-8 - Serbian in Bosnia and Herzegovina

sr_ME.UTF-8 - Serbian in Montenegro

sr_RS.UTF-8 - Serbian in Serbia

I want to apologize to anyone offended by this decision, but rather than make a contentious decision on our own, I decided it was best to simply follow the decisions of an international standards body. I believe that there is no fundamental difference in the languages, although some national variances appear to be present in the data files. If someone has more accurate names for these, or believes that some aliased locales will assist with compatibility with other operating systems, then I would be happy to hear suggestions. Ideally from someone familiar with accepted practice in these locations.

There is another wrinkle in all this too. This language -- thanks largely to occupation by Soviet forces as part of the SFR Yugoslavia, is commonly represented using two different alphabets -- Cyrillic and Latin. Generally most locations use Latin, but within Serbia, Cyrillic is mandated by law. So sr_RS uses Cyrillic, while the others use Latin.

Here are the two alphabets:

А Б В Г Д Ђ Е Ж З И Ј К Л Љ М Н Њ О П Р С Т Ћ У Ф Х Ц Ч Џ Ш

A B C Č Ć D Dž Đ E F G H I J K L Lj M N Nj O P R S Š T U V Z Ž

Anyway, if someone sees room for corrections or improvements here, especially if they are familiar with the language(s) and/or region(s), I would appreciate hearing back from you.

Planet OpenSolaris *isn't*

It would appear that the old Planet OpenSolaris is no longer a community site.

At least the only blog posts that seem to be there anymore are those that are hosted on blogs.sun.com.

Certainly my posts, which used to show up there until quite recently, no longer do so.

Its possible that this is just a technical snafu, but the recent burst of posts there from Oracle employees suggest a shuffling of things internally in how Oracle handles blogs, and I suspect that eradication of community posts is just one more step along the way.

Of course, if I'm wrong, this post will show up there, and I'll have egg all over my face. :-)

Update: illumos is accepted as GSoC Mentoring Org

Great news! We (illumos) have been accepted as a Google Summer of Code mentoring organization.

If you are a student and want to get paid this summer to work on an enterprise grade operating system, please have a gander at our ideas page, and then go ahead and start an application.

You can find our application template on our organization information page on the GSoC 2011 site.

Good luck to all the applicants!

illumos gets documentation!

With this integration:

changeset:   13304:b54231762cfa
tag:         tip
user:        Richard Lowe 
date:        Mon Mar 14 14:05:30 2011 -0400

description:
        243 system manual pages should live with the software
        Reviewed by: garrett@nexenta.com
        Reviewed by: gwr@nexenta.com
        Reviewed by: trisk@opensolaris.org
        Approved by: gwr@nexenta.com

We now have manual pages in illumos. (Only the English pages -- POSIX locale -- are kept in the illumos code repository.)

This is key because it means that code and documentation can be maintained together, which is how some other projects (but not Solaris) manage it.

So, got a problem with the man(1) pages on illumos? File a bug! There is a category called "manpage"... please let us know, or even better, contribute a fix!

Google Summer of Code & illumos

Got a pet project for illumos you would like someone to take up, and would do yourself if you had time? Like working with bright up and coming stars?

Are you a student looking to get involved with a nascent community of world class engineers, and have some free time on your hands this summer?

Maybe you can participate in Google's Summer of Code. We hope illumos will be selected to participate this. Some ideas are posted on our wiki already, but I'd love to hear other proposals. We have a very short window of time before we have to submit our mentoring org application, so let us know!

COMSTAR and SCSI UNMAP

I just pushed this for Dan McDonald into illumos:

changeset:   13297:4b9dc4ca8e9f
tag:         tip
user:        Dan McDonald <danmcd@nexenta.com>
date:        Fri Mar 04 13:57:09 2011 -0800
description:
701 UNMAP support for COMSTAR
Reviewed by: Garrett D'Amore <garrett@nexenta.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Approved by: Garrett D'Amore <garrett@nexenta.com>

This change represents a significant new feature in COMSTAR and ZFS, which will greatly benefit people use SCSI target mode functionality in situations involving over-provisioning. More on that in a minute...

The feature itself was developed by Sumit Gupta for Nexenta, and is part of our upcoming 3.1 release of NexentaStor.

Subsequently, Dan took ownership of that code, and working with Eric and George (who are well established ZFS gurus and had significant and useful feedback) improved it still further, and got the code into illumos proper. I believe this represents the first significant ZFS feature to go into the tree since the illumos fork, and also amply demonstrates the collaboration in the illumos community.

I'm looking forward to more collaboration like this in the community.

Now, what does this feature give you? Well, if you're using SCSI Target Mode (either via iSCSI or Fibre Channel or FCoE) to serve up storage to systems running NTFS or ext4, you will be able to make better use of your storage.

Traditionally, when a file was deleted from a filesystem, it was mostly a matter of book-keeping in the meta data in the filesystem. There was nothing to note this in the underlying storage.

With newer SSDs, and with COMSTAR, the ability to get back this notification is incredibly useful. SSDs want it to do garbage collection or other optimizations thereby improving performance.

COMSTAR wants it because now when your thinly-provisioned zvol gets the notification, we can return the storage back to the pool. Prior to this change, the zvol could only grow, it could never shrink. Now, we can give storage back to the pool when you delete a file on the initiator. This is huge in environments running with a lot of VMs using thinly provisioned storage with overallocation.

Anyway, this is now in illumos thanks to Nexenta, and notably, Oracle doesn't have it. Of course, they are welcome to pick up the code for it, but they will need to follow the terms of the CDDL if they choose to do so, the same as everyone else.

SCALE illumos Photos

As promised, I'd send illumos photos from SCALE. Here's the illumos booth staff, from left to right there is Roland, Garrett (your humble author), Delya, Albert, and Rocky. Rocky was there representing Area Data Systems, who are both Nexenta partners and illumos sponsors.

We also have a facebook photo gallery up, which seems somehow apropos since we were right next to the facebook both at SCALE.

I'm also pleased to report that a number of other Nexenta partners were present as well, showing off Nexenta based products. Next year, we hope they'll be back show casing technology based on illumos and NexentaStor 4.0.

Open Source Opportunities near Boston!

Its now fairly locked into stone that we will be opening an engineering office somewhere not far from Boston (probably just north of it) sometime during 1H2011.

As a result, we've started an aggressive recruiting campaign in the area. I am interested in talking to people with backgrounds in kernel software or device drivers -- especially if the background is on Solaris, but Linux and BSD backgrounds are fine too.

The culture here is startup, and while I'd love to find a few more architect-level candidates, I'm also keen to find folks just beginning their career with a high level of enthusiasm who are talented and driven to become the industry's next generation of storage and networking gurus.

Nexenta itself is still a small company, and so its still a great opportunity to get into the ground floor of what may well prove to be the fastest growing storage company ever. And even better, you can feel good knowing that your contributions will contribute to the greater good -- we are a company committed to Open Source and .. as our motto says, "Enterprise Storage for Everyone."

We'll also be hiring Quality Engineers in the same office, so if breaking stuff is more up your alley than building stuff, then there may also be a place for you.

If this sounds interesting to you, please let me know.

Note that we do not work with recruiters -- individual candidates only please.

illumos at SCALE

So, I've spent the past day or so here at the Hilton LAX with Albert, Roland, and Delya manning the illumos booth at SCALE. We're at booth #72. We've been handing out disks with OpenIndiana, Nexenta Core Platform (4.0 alpha release based on illumos), and NexentaStor 3.1 (alpha release) as well as the first ever illumos T-shirts for those folks who install one of these on their system (either physical or virtual.) I'll have photos to post soon -- sorry they're not ready yet.

The show has been amazing, and we've made a lot of new connections, both with other members of the open source community, and with new potential users and contributors as well as other potential collaborators.

If you're around tomorrow, please drop by and say hello!

(Oh yeah, and come watch Richard Elling and I mix it up with the BTRFS guys at the open source filesystems panel tomorrow!)

And if you're looking for an open source job, drop by and chat. We are hiring in a number of areas, and I'd love to have a chance to chat with candidates.

See you tomorrow!

New audio driver

I've just pushed

changeset:   13278:dabee83e3bb7
tag:         tip
user:        Garrett D'Amore 
date:        Mon Jan 31 17:40:15 2011 -0800
description:
    519 RFE audiocmihd
    Reviewed by: gwr@nexenta.com
    Reviewed by: dev@opensound.com
    Reviewed by: trisk@nexenta.com
    Reviewed by: ams@nexenta.com
    Approved by: trisk@nexenta.com

This represents the first significant contribution to illumos by a third party other than Nexenta, and is also the first hardware driver illumos has support for which Solaris does not. This is for ASUS Xonar cards.

You can tell you have a card that could benefit from this if prtconf -vp | grep pci13f6,8788 shows a result.

I expect this will be introduced soon into a forthcoming OpenIndiana build. Enjoy, and a big thanks again to 4Front Technologies for making this contribution!

Welcome to new Nexentians

I'd like to take a minute to publicly welcome the following engineers, who are well known in the OpenSolaris community, to my team within Nexenta.

Dan McDonald - Dan just started today, and joins us from Oracle (and previously Sun), where he was one of the lead engineers on IPsec and networking security in general. He's also famous as the creator of the internal "punchin" tool used by Sun engineers for many years. At Nexenta he'll be doing some different things, but also will be our go-to man for issues involving the TCP/IP stack within NexentaStor and we expect to see contributions coming from him back into illumos.

Andrew Stormont - Andy started on Jan 1, and is our first software engineer in the UK. He's known for creating the StormOS desktop distribution based on XFCE on top of Nexenta Core Platform. We're looking forward to capitalizing on the synergy of StormOS and Nexenta Core Platform forward.

Roland Mainz - Roland is known throughout the community as "Mr. Ksh93", and previously was the contributor of the single largest integration from the open community into OpenSolaris. We expect he'll be continuing some of the work on improving our userland within illumos, fixing ksh93 bugs, and doing some other interesting things which should result in a direct improvement in measurable quality for both illumos and NexentaStor.

So, please join me in welcoming these three engineers to the Nexenta team, and also recognizing their past and future contributions into the illumos open source community.

Changes to illumos Contribution Process

First off, let me state that the following changes are aimed at both easing the challenging of contributing changes to illumos, while increasing our level of "confidence" in what changes are being integrated into our source code tree.

Up until now, illumos has used a contribution model that is primarily derived from the model used within Sun and Oracle for Solaris development.

This development model is based on the notion that all contributors have (or had at least) the direct ability to "push" code to the repository, after a certain number of review steps had been followed.

This model works well with a small team, or where all contributors are reasonably well trusted. This is also not typical at all of the way most FOSS projects work. (Indeed, with OpenSolaris, this model was not used for external contribution.)

Going forward, we want to enable a much wider group of developers, some of whom may not hang around long enough in our community to get a high level of "trust".

We also want to enable a contribution process that is more similar to what other FOSS projects use.

So, to this end, we are going to move from "developer push" to "advocate pull". "Advocates" are just our version of "maintainers" or "gatekeepers". (The Linux equivalent of this is Linus' "lieutenants".)

So now, rather than developers pushing changes directly to our mercurial tree, going forward Advocates will take patches from Contributors (either via hg export or patch file), verify that the content of the patch is what was reviewed, and will then be responsible for integrating those changes into our shared master.

Note that as part of this process, the Advocate will be ensuring that the original Contributor is still credited in the SCM change history. So Contributors still get credit for their work.

Also, we will still be insisting on other parts of the contribution process that we already have, such as code review, testing, and verification of legal right to receive the contribution.

The main implication for Contributors is that they can supply changes in the form of regular patches, which frees them from having to deal directly with one SCM or the other (more on that below). The other

implication is that if a merge conflict occurs that the Advocate can reject a change and ask the contributor to resolve the conflict and resubmit.

Note that this whole process is much much more similar to the process used by other big name open source projects, such as Linux.

At this point, I'd like to point out that we have a "clone" of illumos-gate on github. So you can use git if you want to. We also have an hg clone at bitbucket.

For now, Advocates use hg as our "master" repository, but we are also talking about a conversion to git to make life better. That's a more detailed topic of conversation, but mostly the concern about whether we are using git or hg should be irrelevant to contributors, as they can use either and are not directly exposed to the integration step (hg push or git push for example.)

The final tidbit here is that we need to set up a public page with a list of Advocates, but for now the list of illumos-gate Advocates is:

• Garrett D'Amore
• Albert Lee
• Rich Lowe
• Gordon Ross

As more people contribute and demonstrate a level of throughness and trustworthiness, I hope the above list will expand somewhat.

illumos sysad/integrator position (NYC)

I have an illumos partner that is interested in finding a strong system administrator/system integrator candidate for work on an illumos-based product in New York. Candidates need to be strong with Solaris, security, scripting (perl, python and/or ruby, etc.) and should have cross platform experience. If this sounds interesting to you, please contact me directly.

need C programmers/interns in So. Cal.

I'm looking to hire a few really smart folks who can work in southwest Riverside County, CA (near Temecula, CA). I want people who are sharp C programmers, want to work on open source kernel software, and are eager to stretch themselves. This is a great learning opportunity. If you know anyone like that, let me know!

illumos update in LA

I'll be giving an update on illumos, as part of my talk in LA this evening. If you're around, please join us!