Worst Company ... Ever

So I've not blogged in a while, but my recent experience with a major mobile provider was so astonishingly bad that I feel compelled to write about my experiences.  (Executive summary: never do business with T-mobile again!)

I had some friends out from Russia back in November, and they wanted to purchase an iPhone 5s for their son as a birthday gift.  Sadly, we couldn't get unlocked ones from the Apple Store at the time, because they simply lacked inventory.

So we went to the local T-mobile store here on San Marcos Blvd (San Marcos, CA.)  I knew that they were offering phones for full-price (not subsidized), and it seemed like a possible way for them to get a phone.  Tiffany was the customer service agent that "assisted" us.  She looked like she was probably barely out of high school,  but anyway she seemed pleased to help us make a purchase.  I asked her no fewer than three times whether the phone was unlocked, and was assured that, yes, the phone was unlocked and would work fine with Russian GSM operators.  I was told that in order for T-mobile to sell us the phone, we had to purchase a 1-month prepaid plan for $50, and a $10 sim card.  While I was little annoyed at this, I understood that this was their prerogative, and the cost of wanting to buy the device "right now" and not being able to wait 2 weeks for Apple to ship one.  (My friends had less than a week left in California at this time.)  We were buying the phone outright, so it seemed like there ought to be no reason for the device to have a carrier lock on it.

So, of course, the phone was not unlocked.  It didn't work at all when it got to Russia.  Clearly Tiffany didn't have a clue about the product she sold us.  I guess I shouldn't have been too surprised. But we had put down $760 in cash for a device that now didn't work.  Worst of all, I was embarrassed, as was my friend, when his son got what was essentially a lemon for his birthday gift.

I was pretty upset, and went back to the store.  Tiffany was there, and apologized.  She offered to pay up to $25 for me to use a third party service to unlock the phone since "it was her fault".  Of course, the third party service stopped offering this for iPhones, and their cost for iPhone 5's was over $100 when they were offering it.

I tried to get T-mobile to unlock the phone.  I waited in the store for a manager for about 2 hours, but Tiffany finally got upset and called the police to get me to leave the store after I indicated that I preferred to wait on the premises for a manager.   I left the store premises, unhappy, but resolved to deal with this through the corporate office.

I went home, and called T-mobile customer service.  It turns out that T-mobile store is really a 3rd party company, even though they wear T-mobile shirts, and exclusively offer T-mobile products.  (More on that later.)

I spent quite a few hours on the phone with T-mobile.  Somewhere in the process, the customer service people called Tiffany, whereupon she immediately reversed her story, claiming she had informed me that the phone was locked.  This was just the first of a number of false and misleading statements that were made to me by T-mobile or its affiliates.   I spent several hours on the phone that day.  At the end of the call, I was told that T-mobile would unlock my phone only after at least 40 days had passed from the date of activation.   The person at corporate assured me that if I would just wait the 40 days, then I'd be able to get the phone unlocked.  Since the phone was already in Russia, I figured it would take at least that long to send it back and send a replacement from another carrier.

Oh one other thing.  During that first call, I discovered that Tiffany had actually taken the $50 prepaid plan we purchased, and kept it for herself or her friends.  We didn't figure that out until T-mobile customer service told me that I didn't have a plan at all.  Nice little scam.  While dishonest, I would not have begrudged the action had the phone actually worked.  Of course it didn't.

(During all this, on several different occasions, T-mobile customer service personnel tried to refer me to the same 3rd party unlocking service.  Because hey, if T-mobile can't get its own phones unlocked, maybe their customers should pay some shady third party service to get it done for them.  But it turns out that whatever backdoor deal that service had previously doesn't work anymore, because they've stopped doing it for Apple phones.)

So we waited 40 days.

And then we called to have the phone unlocked.

And T-mobile refused again.  This time I was told that I needed to have $50.01 or more, not just $50 on the plan.  After a few more hours on the phone and escalation up through a few levels of their management chain, they credited my account for $0.20, and then resubmitted the unlock request.  I was guaranteed that the phone would be unlocked.

Two days later, T-mobile denied the unlock request.

At this point, I was informed that the phone had to have T-mobile activity on it within 7 days of the unlock request.  This was simply not going to happen.  The phone was in Russia!  I complained, and I spent quite a few more hours on the phone with T-mobile.  It seems like the folks who control the unlocking process of their phones have nothing to do with the people who answer their customer service lines, nor those people's bosses, nor those people's bosses.  Astonishing!

At this point I had spent over a dozen hours trying to get T-mobile to unlock the darned thing.  T-mobile had got their money from me already, and had done pretty much everything possible to upset me.  The amount of money T-mobile spent dealing on the phone with me, trying to enforce a stupid policy, which wouldn't have been necessary if they had just admitted their mistake and fixed it (which would have cost them nothing whatsoever) is astonishing.  Talk about penny-wise and pound foolish.

At that time, T-mobile told me that I would be able to return the other phone, provided I got it back from Russia.  They agreed to this even though the usual 14 days "buyer's remorse" window had passed.

So at this point, I went and purchased a phone from Verizon (and paid a $50 premium because I was at BestBuy instead of the Apple store and the Verizon store wouldn't sell the phone unless it was part of a plan), and I sent it to Russia with my step-son, who was going their for Christmas.  That phone did work, and my step-son exchanged the T-mobile phone for it, bringing the T-mobile phone back.

The next day, I went to return the phone at the store where I bought it.  Sadly, Tiffany was there.  So was her manager, Erica.

After spending about an hour at the store trying to return it, they agreed to take it back, minus a $50 restocking fee.  And as I paid cash, I had to provide my checking account information so they could do a deposit for me, which would take a few days.  I got a paper showing that they were sending me a refund, but nothing indicating the account number.  (Turns out it took a few calls from Erica -- who claimed to be the store manager and probably was about 10 minutes older than Tiffany -- the next day, since she apparently had no clue what she was doing and needed to get two separate pieces of information that she failed to collect while I was in the story.)

I did spend a bunch of time with customer service on the phone -- a few more hours I guess, trying to get that last $50.  At this point it was just a matter of principle.  I resented the whole thing, and I wanted as much of my money back as possible.  The customer service person tried to make it right, but because the store (Hit Mobile is the company apparently) is separate from T-mobile, the store's decision was final.  The store manager (Erica) refused to refund the stocking fee, and there was nothing T-mobile could do about it.  To their credit, they did offer me a $50 service credit if I was inclined to keep an account with them.  Needless to say, I have no interest in ever being a T-mobile customer, so I thanked them and declined... indicating that I'd prefer to have funds back in cash.  (Nobody ever mentioned the restocking fee; on the contrary I was told I'd receive the full purchase price less the $60 service plan and SIM card.  Again, story and reality don't match at T-mobile.)

I did eventually get $650 credited back to my account.

So, I was out $110, plus the extra $50, plus the 13+ hours of my time, plus the embarrassment and long turnaround time to get the replacement out to Russia.  All because T-mobile wouldn't fix a mistake they made, even though numerous people at that company recognized that it was clearly the Right Thing To Do.

Turns out that Verizon iPhones are all unlocked.  And I'm already a Verizon customer.  I was thinking about T-mobile's plans -- some of them are attractive on paper, and potentially could have saved me money relative the rather premium prices I pay for Verizon.  Needless to say, I will not be changing my provider any time soon.  In spite of the high prices, I've always been dealt with honestly and fairly, and I've been happy with the service I've gotten at Verizon.

If for some reason, you decide to get a T-mobile phone, please, please avoid the store located on San Marcos Blvd.  In fact, I urge you to verify that the store is owned by T-mobile corporation.  Its my belief that I might have had more satisfactory results had I been dealing with just a single entity.  That said, numerous people at T-mobile corporate lied to me and made false promises.

I feel so strongly about this, that I'm happy to have spent the hour or so writing up my terrible experiences with them.  I hope that I save someone else these painful experiences.  I won't be unhappy if it also costs T-mobile many potential customers.

For the record, I also think it should be illegal to sell a carrier locked device without clearly indicating this is so as part of the transaction.  The practice of carrier locking devices makes sense when the cost of the device is being subsidized by the carrier as part of a service plan.  But it should be possible to purchase the device outright and remove any such lock.  T-mobile's practices here are a disservice to their customers, and their partners.  The handset manufacturers should apply pressure to them to get them to change their policy here.

illumos corporate entity... non-starter?

I want to give folks a status update on the illumos corporate entity.

In a nutshell, the corporate entity seems to be failing to have traction.  In particular, the various corporate contributors  and downstreams for illumos have declined to step up to ensure that an illumos corporate entity has sufficient backing to make it successful.

While at first blush, this seems somewhat unfortunate, I think this is not nearly quite as bad a thing as it might first seem.  In particular, the failure of a corporate entity does not correlate to the health of the ecosystem -- indeed many successful open source projects operate without an umbrella organization or entity. 

Instead, we see corporate contributors and downstream distributions focusing on developing the communities behind their distributions such as SmartOS and OmniOS.  Those downstreams play an active role in improving illumos for the benefit of all, and its my sincere hope and belief that they will continue to evangelize illumos, and contribute to the common core.

Furthermore, incorporation in the state of California requires paying about $800 of taxes per year.  This is true even for organizations without any revenue.  This is money that would have to be taken from sponsors that would serve only to enrich our state government with no direct benefit to the illumos community.  (Non-profit status is a way around that, but its exceedingly difficult to obtain, and a number of open source organizations are finding themselves under very close scrutiny from the state and federal authorities.    Indeed, the X.org community themselves lost their 501(c)3 status not that long ago.)

So, without corporate sponsors to justify the tax and administrative load, I've decided that the illumos corporate entity should expire.  I do want to thank Deirdré Straughan for the non-trivial amount of effort she put into this, as well as the Software Freedom Law Center for the pro-bono work they did for us while we were trying to navigate the waters of becoming a true non-profit open source organization.

And, if any corporate sponsors are out there watching this, and interested in resurrecting the illumos organization, then I'm happy to entertain the possibilities.  I think there is value in an actual organization with a legal status to receive and distribute funds, and who can hold certain items of intellectual property, including the rights to the illumos trademark itself.   But there has to be enough of a sponsorship behind this to make it worthwhile.

In the meantime, I will continue to hold the illumos mark as a trademark that I keep in trust for the community.  The code is something that is already subject to distributed ownership, and so is completely unimpacted by any of this.

Thanks.

  - Garrett

Moving on (Adieu to Studio?)

I think illumos is at a key juncture, and the issue relates to our toolchain.

We have gcc 4.4.4 working thanks in large part to the efforts of folks like Rich Lowe.

We have historically relied on Sun Studio (now Solaris Studio) as our "base" or "default" compiler for C and C++ programs, and also to supply lint coverage.

I've been thinking for a long time that its past time we (the illumos community) moved on from this.  Not only are the Studio 12 compilers not available in source form, they are now not available in suitable form for building illumos as binaries either.  (Apparently it is possible under some terms to get Solaris Studio 12.3, but who knows if those compilers are suitable for building illumos.  In the past we have always needed specifically patched compilers for Solaris.)

The situation where a general developer cannot obtain the necessary tool chain for working with illumos is untenable.

Today we require "lint" as part of an official build.  But not just "any" lint, but lint generated by a specific patched older version of Sun Studio 12.  A version that is no longer available to the community at large.

(And in full disclosure, the problem for me is brought to a head by the fact that the machine I had my local installation of these tools on has been retired, and I find I don't have another backup of them readily at hand to install on my new machine.)

So, I'm placing this as a call to the illumos community at large, and especially to our RTI advocates.  Its time.  Really. 

Really.

Studio has to go.

I don't care if we leave the infrastructure in place for people that want to continue to use Studio, but we need to switch to gcc.  Rich's 4.4.4 version gets the job done for now.  It would be great if we could support newer versions, but I understand that this requires some non-trivial amount of effort (some gcc patches that need to be upstreamed, as I understand it.)

We cannot be tied to a closed tool chain; especially a tool chain that doesn't at least include binary redistribution privileges. 

For the record, I've posted the same content to the illumos developer and advocates lists.

Responding to Trolls

We have an... ahem... toxic personality in our midst in illumos.  I'm going to respond here to his most recent post, to avoid wasting the time on the illumos developers mailing list.  If you don't care about non-sense like this (although it may have entertainment value), please just ignore this post and move on.

The only reason I'm replying even here, is to set the record straight, and establish firmly the history -- at least my side of it -- for posterity.

In the early days of illumos, I invited those I could think of from the Solaris and OpenSolaris communities to participate.  I even reached out to some parties, like Jörg Schilling, who had a bad reputation as difficult people to work with.

I had hoped to repair those broken relationships since there was no longer a corporate controlling entity for illumos.  (Nexenta sponsored my efforts, and paid some of the early operating costs for illumos, but it has always been the case that illumos operates by and for the community -- and not at the behest of any corporate master.  As I've since left Nexenta to start my own company, and illumos has carried on, I think there is fairly ample evidence in support of this fact.)

I reached out, and tried to help Jörg with some integrations; but those integrations did not go well.  Jörg did not like our policy requiring a build or test of changes on the primary distribution (OpenIndiana at the time) used for the vast majority of developers and users, and so several of his changes did not get integrated.  (It turns out this was a good thing, because at least one of those changes, a "fix" for a keyboard problem, was mis-designed and would have broken international keyboards for the majority of illumos' user community.)

We also required code review, as we do now.

Jörg has long had a desire to have "star" integrated as the main replacement for all archivers.  That didn't take place, because he couldn't find reviewers.  And after several negative interactions with him, I withdrew my offer to sponsor his work personally, as I found the process of trying to do so unbelievably painful.  (There was an extremely long telephone call with Joerg that sticks in my memory.)  I suggested he find someone else willing to code review and sponsor his work for integration into illumos.  To date, that has not happened.

Somewhere during that time, Jörg indicated that he would not offer any other code up for integration until star was integrated.  Basically, he took his toys and went home.  Except he just took his toys, but forgot to go home, as he's still trying to create noise on our developer mailing lists.

There was another negative experience we had, mirroring today's thread, where Joerg's response to a tool I wrote (an open replacement for "od") was non-constructive criticism.  (In fact, his response was "it's broken, why don't you use this thing I haven't finished writing yet".  (He modified his "hexdump" utility to behave like od, apparently in response to my effort instead of simply offering helpful review feedback.)

This is typical of Jörg -- the only "right" answer as far as he is concerned is to accept his code, even if it isn't written yet or is buggy!

Jörg is what is known as a poisonous personality.  At least in every context that I've seen him operate. Which is why usually I try to just ignore him, and hope he'll go away; yet like certain garden pests he just keeps coming back.  He's even been kicked out of another prominent open source community.

He's also said made some incorrect and defamatory statements about both Nexenta and illumos at a prominent academic conference in Germany.  (I can't find the link right now, I'll update if I find it.)

So, where in his post today he asks Gordon if he's willing to change the rules for integration, I have this in reply:

Are you willing to change this? 

Gordon isn't empowered to.  Neither, for that matter, am I.  The rules that have been established for collaboration here are by common consent of the major contributors, and no single individual can change them.  Notably, those were the rules in force in Summer 2010, and you were the one that wanted special treatment.   Those rules have served this community well for the past several years, and there has been no movement to change them.

I am interested in setting up a phalanx of OpenSolaris activities because this
is the only way we have a chance to continue with OpenSolaris development but
this is hard as long as Illumos is not willing to play nicely.

OpenSolaris is dead.  The entire illumos community has moved on.  You have not, but then I have long since stopped considering you a part of the illumos community.
Especially given your repeated attacks against both illumos and numerous of its contributors -- even fairly early on you made a number of defamatory and incorrect statements about the nature of both illumos and its contributors in technical presentation before the German academic community.  This was not the behavior of someone who is a constructive contributor.
To be completely honest, I think I'm not alone in wishing that you'd come to the same conclusion: that is that you are not part of this community.

I recently tried to build a bridge by propising that Illumos should implement
the proposals from Summer 2010 to regain credibility. Unfortuntely there was no
reaction. 

That should speak volumes all by itself. 

As someone else in another open source community said, Jorg Schilling is damage; the community should route around him.

An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone.

This milestone was the integration of 195 Need replacement for nfs/lockd+klm

This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at Nexenta.  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix, and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product.

The reason this is such an important milestone, is two fold:

First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of the community.

Second, this effort represents the replacement of the last of the truly closed source code in our kernel proper.  There are still some closed source drivers (although I will point that we're now seeing more traction with vendors to supply open drivers, with recent open source code contributions coming from places like SolarFlare, LSI, and HP.  (HP's contribution -- through illumos partner Joyent -- means that at least one important closed source driver will soon be open -- cpqary3.)

I want to personally thank everyone who contributed to make the integration of the new open source NFS lock manager possible. You've done a big service to the community, and we are grateful!

ZFS Compression Enhancements in illumos

The ZFS code in illumos has gained another differentiator thanks to the hard work of Saso, who integrated LZ4 improved compression into our code base.

This is a nice achievement, and enhanced compression should be quite useful for most applications of ZFS.  If LZJB was always recommended to be turned on (it was by our team at least!), then LZ4 is even more recommended, unless all of your data is totally incompressible.  (Most data is highly compressible.)

You should check out his wiki post above, which contains a lot more details about the work, including some benchmark data.

hackathon project: pfiles - postmortem analysis

During the first days of October 2012, we had an illumos day, and a one day illumos hackathon that was very well attended by some of the best and brightest in our community (and I daresay, in any open source community.)

The purpose of the hackathon, besides just doing some cool projects, is to give folks a chance to interact with other domain experts, and code, particularly in areas outside their particular area of expertise.

This year a number of interesting projects were worked on, and I think some of these are at the point of starting to bear fruit.  The project I undertook was the addition of post-mortem analysis support for pfiles(1).  That is, making it possible to use pfiles against a core(4) file.  Adam Leventhal suggested the project, and provided the initial suggestion on the method to use for doing the work.

This is a particularly interesting project, because the information that pfiles reports is located in kernel state (in the uarea) and has not traditionally been available in a core file.

To complete this project, I had to work in a few areas that were unfamiliar to me.  Especially I had never looked at the innards of ELF files, nor at the process by which a core file is generated.  To add the necessary information, I had to create a new type of ELF note, and then add code to generate it to libproc (used by gcore) and elfexec (for kernel driven core dumps).  I also had to add support to parse this to libproc (for pfiles), and elfdump.  The project turned out to be bigger than anticipated, consisting of almost a thousand lines of code.  So it took me a little more than a day to complete it.

A webrev with the associated changes is presented for your enjoyment.  I'd appreciate any review feedback, and especially any further testing.

Here's a sample run:

garrett@openindiana{250}> cat < /dev/zero &
[2] 144241
garrett@openindiana{251}> pfiles 144241
144241: cat
  Current rlimit: 256 file descriptors
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:72712192
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:72704245
garrett@openindiana{252}> gcore 144241
gcore: core.144241 dumped
garrett@openindiana{253}> pfiles core.144241 
core 'core.144241' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:195125248
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:161882375
garrett@openindiana{254}> kill -11 144241
garrett@openindiana{255}> pfiles core
core 'core' of 144241: cat
   0: S_IFCHR mode:0666 dev:526,0 ino:35127324 uid:0 gid:3 rdev:67,12
      O_RDONLY|O_LARGEFILE
      /devices/pseudo/mm@0:zero
      offset:419414016
   1: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
   2: S_IFCHR mode:0620 dev:527,0 ino:893072809 uid:101 gid:7 rdev:27,2
      O_RDWR|O_NOCTTY|O_LARGEFILE
      /dev/pts/2
      offset:296960430
[2]  - Segmentation fault            cat < /dev/zero (core dumped)

garrett@openindiana{256}> elfdump core

...

  entry [10]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        0
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IRGRP S_IWGRP S_IROTH S_IWOTH ]
        pr_uid:       0                  pr_gid:      3
        pr_major:     526                pr_minor:    0
        pr_rmajor:    67                 pr_rminor:   12
        pr_ino:       35127324
        pr_size:      0                  pr_offset:   419414016
        pr_fileflags: [ O_RDONLY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /devices/pseudo/mm@0:zero

  entry [11]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        1
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2

  entry [12]
    namesz: 0x5
    descsz: 0x440
    type:   [ NT_FDINFO ]
    name:
        CORE\0
    desc: (prfdinfo_t)
        pr_fd:        2
        pr_mode:      [ S_IFCHR S_IRUSR S_IWUSR S_IWGRP ]
        pr_uid:       101                pr_gid:      7
        pr_major:     527                pr_minor:    0
        pr_rmajor:    27                 pr_rminor:   2
        pr_ino:       893072809
        pr_size:      0                  pr_offset:   296960430
        pr_fileflags: [ O_RDONLY O_NOCTTY O_LARGEFILE ]
        pr_fdflags:   0
        pr_path:      /dev/pts/2
...