Thursday, December 25, 2008

Boomer Beta Source Code Posted

Boomer, aka PSARC 2008/318 -- the new audio subsystem for Solaris, and the project I've been working on for the last several months -- has released source code for our work in progress. A status update and link to the webrev is available here.

Wednesday, December 3, 2008

Floppy drive suspend/resume fixed

Earlier today I just pushed a fix for suspend/resume when the machine has a floppy disk drive. Surprisingly, a lot of modern machines still have floppy disk drives in them, and this fix enables those platforms to participate in the S3 suspend work.

The fix will be in b105.

One such machine is the Dell Precision M390. I imagine there are many other desktop class systems of this nature.

While suspend/resume has been focused largely on laptops (where battery life is a paramount concern), I suspect that enabling suspend-to-RAM for desktop class systems will ultimately have a larger environmental impact, since it is those systems that typically consume the most power. So, perhaps this small change will have a somewhat significant impact.

If you have a platform that this enabled suspend/resume to work on, tell me about it. Include smbios output in your message, too. :-)

Tuesday, December 2, 2008

SDcard Fixes

I've heard a number of problems with folks trying to use SDcard with Ricoh controllers (such as found on certain newer Toshiba Tecra laptops), as well as lack of quiesce(9e) support, and various other problems.

To address these problems, I've made a bunch of changes, and am looking for code reviewers and testers -- the webrev is an open review.

I've made binaries available to anyone who wants to try these out. To try out the new binaries, just untar the sdcard-20081202.tar.gz file in your / directory. You might need to reboot for them to take effect. (If you know how to modunload the drivers from kernel memory, that will work too, provided that you don't have any SDcard media inserted at the time.)

Please let me know any feedback you might have. Thanks.

Monday, November 10, 2008

locking hints for device drivers

It seems that I often run into the same problems over and over again, and I see many device drivers which often suffer from the same problem. Here are some strategies I use in my own coding -- maybe someone else will find them useful. To my knowledge they have not been documented elsewhere.

  1. KISS. Start with as few locks as you need to get the job done. Only add locks when performance analysis shows you need them. I usually start with an assumption that I'll need one lock per device instance, and possibly a single global lock. (If you don't have global state, you can elide the global lock.) For some kinds of drivers (NICs in particular) I introduce a lock for each major traffic direction (so one lock for rx, and one for tx) per instance.
  2. Global state is evil. Global state requires global locks. Global locks often introduce lock ordering problems, and can also more likely to be contended in systems with lots of devices.
  3. Thou shalt not sleep in STREAMs entry put(9E) and srv(9E) entry points. This one I frequently see violated. These can run in interrupt context. Don't call cv_wait(9F) or its friends here.
  4. Use mutex(9F)es as your lock primitive of choice. rwlock(9F)s are slightly more expensive (and can be more challenging to get read versus write sorted out properly), and semaphore(9F)s can fall prey to priority inversion.
  5. Hold those mutexes for as short as possible. If you can do some work ahead of time, or defer it to later, outside of the time you hold the mutex, you'll greatly reduce opportunities for contention. Uncontended mutexes are good. Contended mutexes are bad. Sometimes small bits of code reordering can have significant performance impact.
  6. lockstat(1M) is your friend. It can really help you to understand what the hot locks in your driver are.
  7. ASSERT(9F) is your friend. You can place assertions up front of functions that have to be called with certain locks held. Its easier to debug assertion failures than it is to debug deadlock or (far far worse) race conditions.
  8. Be uncompromising about order of lock acquisition. Ensure you always, always acquire mutexes in the same order.
  9. Avoid locking side effects. Functions should usually exit with the same locks held that they started with.
  10. Leaf locks are easier to understand. If you can reduce a lock to a leaf lock (one that is never held across functions which do any other locking), then you're guaranteed that the lock is safe.
  11. Avoid assumptions about locks held in the framework. With very few exceptions (bcopy(9F), strlen(9F), etc.) you can generally assume almost any DDI routine you call will need to acquire a lock. Hopefully most of them are leaf locks.
  12. Be very concerned if you think you need to drop a lock only to reacquire it. (Such as dropping a lock to make a call up into the DDI.) This is one of the more frequently encountered problem areas. Locks used to create atomicity are only effective if the atomicity is not broken. (If you drop a lock, you must not assume any of the conditions that held true when it was first acquired remain true when it is reacquired.)
  13. Minimize asynchronous activity. Do you really need to fire off a taskq, or use timeout(9F) to perform that operation? More asynchronous threads is more complexity, and violates principle 1 (KISS).
Of course, there are often good reasons to violate one or more of the principles above. I find that if I try to to adhere to the above principles though, I run into fewer ugly problems in debugging kernel software. So I always do a double take if I think I need to violate one of the principles -- probably about 70-80 percent of the time, that double take leads to a simpler or better approach that doesn't violate the principles, and ultimately saves me lots of pain later.

Tuesday, November 4, 2008

BMC on concurrency

Bryan Cantrill has just posted one of the more excellent blog posts I've seen this year -- he manages to put to words some of my heretofore unvoiced doubts about transactional memory. I'm a big fan of KISS, and both TM and multilevel threading seem to fail on that front.

Saturday, November 1, 2008

fdc suspend/resume support

In another update, I've updated fdc (the floppy disk controller) so that it supports suspend/resume even if a floppy drive is present. There is one caveat though -- namely that the drive must be empty. Still, its far easier to pull a disk out of a drive (if you even have one -- most floppy drives these days probably see very little, if any, use), than to disconnect the drive entirely.

You can download the file from the device drivers download page (look for a file named fdc-2008-11-01.tar.gz). A webrev of the changes is also posted.

For the curious, the reason for the above caveat is best stated by the following explanation extracted from a comment in the code:
Bad, bad, bad things will happen if someone changes the disk in the drive while it is mounted and the system is suspended. We have no way to detect that. (Undetected filesystem corruption. Its akin to changing the boot disk while the system is suspended. Don't do it!)

So we refuse to suspend if there is media present in the drive. This limits some of the usability of suspend/resume, but it certainly avoids this potential filesytem corruption from pilot error. Given the decreasing popularity of floppy media, we don't see this as much of a limitation.
Anyway, I suspect these changes may have little value to laptops, but may help a lot of desktops. It certainly addresses the problem of SUSPEND for my Dell Precision M390.

Update: Thanks to J├╝rgen Keil's testing, we have found a bug, which I've fixed. The webrev and the binaries have been updated accordingly. Look for the 11-04 release instead.

iprb updated with suspend/resume & quiesce

I've updated iprb (Intel Pro/100 Ethernet driver) to support suspend/resume and quiesce (fast reboot). I've also made some other changes, such as updating to the latest microcode from Intel, fixing some potential races, and improving the internal implementation of the statistics routine.

I'd like folks to try it out on their Intel or AMD platforms. The tarball can be downloaded from here (look for iprb-2008-11-01.tar.gz).

I still want to get this open sourced, but I haven't got approval from the lawyers yet. But here are the binary bits in case anyone is waiting for them.

(I'm not planning on porting this driver to SPARC. There are too many places in the code that would have to be changed to use endian-safe calls to ddi_get, and I think there is probably very little demand for a SPARC version of the driver. If you are one of the people who do want SPARC support for iprb, please let me know.


Tuesday, October 28, 2008

Boomer: Next Generation Audio for Solaris

I've posted draft inception materials for our PSARC case, 2008/318, for Boomer. Boomer is the code name for the effort to integrate a modern OSS-compatible API, and additional drivers, into Solaris.

PSARC inception review for this case is scheduled for November 5, 2008. Please check the ARC web page for call in details (any engineer in the community may dial in.) Be advised that the meeting is an engineering meeting, though. (PSARC provides engineering architectural review for projects.)

We have a working prototype for most of the bits we are ARC'ing at this point, and I expect to be opening up the source code tree, and binary test bits, in the coming weeks. Perhaps by Thanksgiving.

Monday, October 13, 2008

Sun position available in Austin

A new position in Austin, TX has opened in my group (onsite only). This is for a junior or mid-level administrator or QA type person -- great for someone who loves playing with new hardware, software, etc. No specific programming skills beyond every day scripting required (though perl and C would probably be useful.) Send replies back to me. (Full disclosure: I get a referral bonus for anyone I refer.)

Thursday, October 9, 2008

Microsoft & NetBSD

Some folks might be interested in this. I used to hack on NetBSD at my previous job. I recently had an inquiry from a recruiter at Microsoft (don't worry Neal, I'm not going anywhere! :-) which I thought I'd share here.

Location: Redmond, WA
Client: Microsoft.
Job Description:

Seeking a talented NetBSD software developer interested in helping Danger (a subsidiary of Microsoft) ship the next generation of Danger’s Sidekick platform. Specifically, we’re looking for NetBSD developers interested in commercializing the NetBSD platform for an embedded mobile computing device, focusing on performance and optimization, bug fixing, and integration with Danger’s higher-level platform code, with an emphasis on kernel and driver support.
- 8+ years of software development experience
- 8+ years experience with C
- Strong understanding of OS concepts (particularly with NetBSD) such as multi-threaded program design and synchronization, processes & memory protection, etc.
- Strong communication skills
- Strong understanding of the NetBSD/GNU software development process and embedded development & debugging techniques
- Deep understanding of NetBSD, including timers, RPC, TCP/IP, etc.
- Experience with ARM processors highly desirable

So, not Windows CE or some other Windows platform. Cool. Wonder when they'll start posting job offers for Solaris engineers? (Probably as soon as Solaris is useful for small embedded devices -- i.e. I wouldn't hold my breath. :-)

Friday, October 3, 2008

New Stuff in OpenSolaris 2008.11

OpenSolaris 2008.11 is coming soon. Build 101 is the stabilization build before it, and as usual, new features are excluded from this build. So we can say pretty certainly what features are in, and what are out.

While there are a lot of new features coming in this release, there are some in particular that I've been more involved with.

  1. SDcard support. Numerous laptops can now use SDcard and MultiMediaCard media directly. A good way to know if you're laptop is one of them is to search for pciclass,0805 in the output of prtconf -vp. While it isn't completely conclusive (in particular some models from Texas Instruments are schizophrenic here), if you see this, there's an excellent chance it will work. I'd always like to hear feedback about this feature -- if your unit works, or doesn't work, let me know. (Also, of course, the Tadpole SPARCLE models are supported. )
  2. AudioHD improvements. Notably, many more laptops will now have working audio. The audiohd driver is also updated to support Suspend/Resume.
  3. Fast Reboot. I participated as a consultant. I'll also be updating (post OpenSolaris 2008.11) additional drivers to support this feature. The upshot of this project is that a healthy system can reboot much more quickly now.
  4. Brussels (NIC Administration). I've participated in converting several drivers to Brussels, and in generally improving Brussels (I'm also the ARC sponsor for this work.) The upshot of this project is greatly improved manageability for network interfaces.
  5. Suspend (S3) Support. I helped review the conversions of several drivers, and provided fixes for several NIC devices.
  6. Bug Fixes. I've worked on a number of them, and of course, there are huge numbers of bugs that have been fixed in this release.
All said, I think OpenSolaris 2008.11 is going to be great -- I confess that I was skeptical about the earlier releases, but this release is shaping up to be really awesome.

Thursday, October 2, 2008

Ancient History Exhumed

Okay, maybe not so ancient (circa 2000), but I recently got an e-mail from the Sun IT group notifying me about changes that impacted web pages I set up for Alternate Pathing, which was the very first project I did that involved work within the Solaris kernel. Apparently they didn't notice that I'd left the company and returned.

Of course Alternate Pathing was canceled a long time ago, although someone may still be using it on older E10K Solaris 8 systems.

Here's the Sun internal URL to which the e-mail referred. The bug tracking pages are broken, since the scripts behind them were developed (by me) to talk to the BugTraq+ Sybase server, which Sun hasn't used in ~forever.

Monday, September 15, 2008

IOMMU comes to Solaris x86

This weekend, the code for the IOMMU for Solaris on Intel (PSARC 2008/560) was pushed. This has potentially profound ramifications for folks working on Solaris device drivers, and I thought I'd take some time to talk about them.

First off, it needs to be noted that we've had an IOMMU on Solaris SPARC pretty much for as long as we've had Solaris on SPARC. (In fact, most SPARC platforms have to use an IOMMU -- they have no choice.) But on x86 this technology is new.

The benefits that IOMMU brings are many fold.

  1. It virtually guarantees that all DMA requests can be set up with a single DMA cookie, reducing complexity and eliminating the use of bounce buffers by the DMA framework -- even for old devices with unfortunate restrictions (such as an inability to perform dual address cycles on PCI -- i.e. no 64-bit support.) Such restrictions are actually fairly common place.
  2. It allows for strong isolation to be given for devices that can be accessed via other virtual machines or domains. This can prevent one misbehaving xVM domain from crashing others by misprogramming the DMA engine on a physical device.
  3. It allows for isolation of faulty devices, so that they cannot scribble into arbitrary PCI spaces, preventing a misbehaving device from accessing regions which it should not. This has major benefits for fault resilience, as well as diagnosability. (To be fair, I'm not 100% sure the code is in place yet to leverage all of this benefit, or integrate it with FMA.)
  4. It facilitates debugging of faulty device drivers. To give a concrete example, when I was working on an audio driver recently it took me a long time to figure why the device was emitting white noise. It turns out that I had not initialized the DMA address register properly. With IOMMU, instead of the device just getting random data, I'd have gotten a bus fault that would have contained information that I could have used to see that the device was trying to access some weird place memory to which it had no right.
Now, these features don't come without some cost.
  1. Setup and tear-down of DMA operations is potentially significantly more expensive than with the simple translation layer previously used. Device drivers that assume such operations are inherently cheap may be in for a surprise.
  2. Drivers still have to retain the ability to operate in an environment without an IOMMU. Effectively, this means that they need to be prepared to see more than one DMA cookie or window per DMA region. Generally speaking, well written drivers should make no assumptions about the number of cookies used beyond the limits expressed in the DDI DMA attributes. (Namely, the framework is free to use any number of ddi_dma_cookie(9s) >= 1 and <= dma_attr_sgllen.)
  3. Drivers that require physical rather than virtual memory be used (i.e. that need to bypass the IOMMU) can request mappings using DDI_DMA_FORCE_PHYSICAL in the ddi_dma_attributes(9s) dma_attr_flags field, but such requests are not guaranteed succeed. A portable driver must retry such a request without the flag set, if the first attempt with it set fails.
  4. Generally, correctly written drivers will Just Work with the integration, without any changes to them. I would discourage driver authors from disabling the use of DDI_DMA_FORCE_PHYSICAL unless they have specific performance requirements. (And, normally, there are better solutions such as reuse of mappings, so that DDI_DMA_FORCE_PHYSICAL remains unnecessary.)
  5. The IOMMU imposes (or should impose) a new test requirement -- namely that device drivers are tested on systems both with and without an IOMMU. While code that works on systems without an IOMMU is unlikely to notice the introduction of the IOMMU, the reverse is not true. If drivers were developed in the presence of an IOMMU, it is not unusual for them to fail on systems without an IOMMU, as the lack of an IOMMU often requires mappings to be made with multiple DMA cookies, especially for resources that page boundaries.
I've not had a chance to play with the new framework myself yet, but I look forward to doing so. Also, be aware that same feature set is coming soon for AMD platforms, see PSARC 2008/561.

Sunday, September 7, 2008

audiohd pushed

The latest & greatest audiohd driver, which includes vastly improved support for a large number of codecs, suspend/resume support, and a generic "codec parser" has now been "pushed" into build 98 of ON. Many thanks to the Beijing audio team, who worked long and hard to bring this project to fruition.

Tuesday, August 12, 2008

New audiohd driver posted

We've gotten a lot of good feedback from previous posting of the audiohd driver, and the Beijing engineering team has come up with a new version (Aug 12, 2008.) Note that the latest audiohd driver will always be posted here.

One note: please use the obj32/ or obj64/ versions of the driver unless you are running a debug kernel. There are some binary dependencies where the debug drivers won't work on a production kernel.

Friday, August 8, 2008

SDcard pushed...

Finally, after months of delay, the SDcard bits have been putback^Wpushed.

This was interesting, because the SDcard bits were also some of the first bits to have been pushed into the new Mercurial tree -- and they're already in the clone. Adventurous people can start trying the code out now. Or just wait until b97.

Of course, this also means that this will be a new feature in the OpenSolaris 2008.11 release. Yay!

If you do try the bits out, let me know what works, and what doesn't. There seems to be some evidence from the Linux community that not all SDHCI compliant controllers are created equal, although the code I have works for the few variants that I've had access to.

The bugster category for bugs is solaris/driver/sdcard

Thursday, August 7, 2008

SDcard RTI submitted... webrev posted

Finally got legal approval!

A webrev is available, as well, in case anyone wants to look at it. I hope to have this integrated into b97.

Wednesday, August 6, 2008

SDcard Home Stretch

The SDcard Phase I project is in the home stretch -- the only thing remaining at this point is two signatures. And one of those is my managers! (The other one is the final sign-off from legal, which I expect to happen tomorrow.) I'll file my RTI tomorrow evening, most likely. If folks, I can also post a public webrev at the same time (though that would just be informational, as the actual required code reviews have already taken place.)

Tuesday, August 5, 2008

Classic! Another Work I Don't Care About

Apparently Twitter has been hit by another Trojan, according to the BBC.

What I love is the last paragraph in the article:
Only those using Microsoft Windows are vulnerable to infection from these malicious programs.
Of course, I only run Solaris at home, so I guess I have naught to fear. See, there is an upside to having a tiny market share on the desktop!

Teamware Is Dead

I still recall seeing the mail years ago, when the tools group told us that Teamware was EOL, and would not be supported any longer. I recall thinking, OMG, what are we going to do for ON? At the time, it looked (to me at least), like BitKeeper was a logical replacement.

Now here we are, years later, and with the closure of build 96 last night, we have finally stopped using Teamware for new development at Sun.

Mercurial/Cadmium seems like a powerful and capable replacement, and has the advantage of being open source. This is one of the most significant technical hurdles to fall that has kept external folks from being able to commit directly to Hg. (Opening up RTI, and the bug tracking database are the two remaining big ones. RTI can probably be opened without too much difficulty. The bug tracking problem is a different matter altogether.)

As a nice side effect, external folks can now have their changesets integrated directly, without requiring sponsors to do conversion work, and without requiring sponsors to place a "contributed by" credit. (Because the changeset will already have the contributor's e-mail.) This should significantly reduce the amount of effort required for sponsors, and consequently make it easier for more people to contribute.

Big thanks to the SCM team!

Monday, August 4, 2008

Why You Don't Want to Force Link Modes

So, I got a message today, not uncommon, indicating that some poor guy is dealing with an IT organization that still requires link parameters to be forced to 100 Full.

It hardly seems reasonable that in today's day and age, folks wouldn't be using auto-negotiated link parameters, yet, there it is!

This blog entry is dedicated to those IT organizations stuck in the early 90's, still requiring their users to force link mode.

1. Back Story

Once upon a time, 100 Mbps full duplex 100Base-T was new. And in the early days, there was not universal adoption of a standard (in the very early days there wasn't a standard at all!) for autonegotiation of link speed and mode.

The IEEE 802.3 group (the folks that oversee the Ethernet standards) rectified this with a standard called "IEEE 802.3u", which provided the rules for auto-negotiation.

Most of the world, and pretty much all of the equipment manufacturers, adopted this standard in short order, and there was much joy and happiness.

2. Except....

That some IT organizations had older equipment, and had compatibility concerns. So, they insisted that that their users "force" the link parameters. This allowed everyone to take advantage of full-duplex operation in those organizations.

3. The Conflict

The problem with this situation is that disabling auto-negotiation on the switches in the machine room meant that the default auto-negotiation which all hardware shipped with enabled by default didn't work properly. With the back-end not doing 802.3u, the front end can't decide, and defaults to a "safe" mode of 100 Mbps half.

Of course, this safe mode causes all kinds of problems on its own, if the back-end is presuming full duplex is OK. (This will manifest as errors and collisions on the link, in bad cases it can make the link entirely unusable.)

4. IEEE Steps In

So, for the next round of standards for ethernet, the Gigabit standards, the IEEE required the use of auto-negotiation to select link speed and duplex. Its possible to control what is advertised/supported by a partner, but you must not disable auto-negotiation itself. The upshot of this is gigabit equipment is going to have difficulty with those backwards IT organizations that still require forced modes.

5. The Problem Today

The problem today is that some IT organizations have carried this practice on for years, leading to excessive support headaches, etc. They have avoided the one-time cost of transitioning to auto-negotiation, but they do so at the continued cost of support headaches resulting from this decision.

Added up over time, I expect that the support costs for dealing with problems from mismatched duplex settings due to one end having auto-negotiation administratively disabled probably totals in the millions of dollars. I'll leave the math to the bean counters, but its not trivial. (One of the problems is that each time a new OS or a new piece of equipment is added, someone has to figure out how to change the defaults. I've probably spent over a dozen hours helping people with this problem myself over the years -- which is silly because IMO nobody should have this problem, at least not outside of possible testing scenarios.

As another example of what can happen ... recognize that it is not always possible for end equipment to support full-duplex. All hubs, by definition, cannot support full duplex operation. So, what happens in your organization when some bloke wants to add a hub to get an extra Ethernet port? Does everyone in your IT department know about this limitation? (How many hours have been spent tracking down this particular problem, I wonder?) If auto-negotiation is used, it won't matter.. everything Just Works, regardless of whether the end user inserts a hub or a switch.

(Hubs are also extremely useful for performing network debugging, because it provides a point at which bits can be examined on the wire using tcpdump or snoop or similar tools. Obviously, this mode of debug is not available when organizations force the switch to use full-duplex operation.)

6. What You Can Do

If you're in an IT organization that boycotts 802.3u, consider strongly whether this practice is worth continuing. (I'd argue it isn't, at least if your organization ever grows.) New ports should be allocated without this requirement, old ports should be converted as they are able to be, if the organization can't convert them all at once.

In the era of 1G and 10G (notably 10G not only requires auto-negotiation, but also does away with half-duplex operation!), its past time that IT organizations caught up into the current millenium with their management strategies.

Feel free to repost this where your IT head of department will see it.

Sunday, August 3, 2008

OpenSolaris as a boot server

I recently upgraded to the "latest" OpenSolaris on my desktop. This is also my primary NIS and NFS server and boot server.

I was rather dismayed at first by some of the bits that were missing. NIS, DHCP, and more. I finally got it all working, and I thought I'd document it here for posterity.

First install NIS. "pkg install SUNWyp"

(Configuration of NIS left as an "exercise" for the reader.)

Second, DHCP was missing, so we install it as well:

"pkg install SUNWdhcs SUNWdhcsb SUNWdhcm"

(Configuration of DHCP left as an exercise...)

Then, it turns out that there was no manifest for TFTPD, not even a commented out entry in /etc/services. I copied the line from /etc/inetd.conf on another system:

tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot

Then run "inetconv".

My question to the OpenSolaris/Indiana folks -- why wasn't this listed in the stock inetd.conf -- even as a commented out entry? What I'd have preferred is to see a default service installed, but not enabled by this for default. It took some man page reading and investigation to figure out where tftp was in the "new" world.

Anyway, its all working now... I'm booting a b95 image over the network as I type this...

Friday, August 1, 2008

Another SDcard Status Update

Some of you may have been wondering about the work I've done for SDcard. Well, it looks like we have basic agreement at this point, and I am only waiting for the lawyers to draft the final text granting the approval, and then I'll be able to supply the bits publicly, as well as integrate them into OpenSolaris. I'm expecting that this could happen as soon as build 97. Stay tuned, and sorry for the delay.

(Internal folks can access bfu archives for Intel based system at /net/zcube.west/export/ws/sdcard/hg-x86/archives or for Tadpole SPARCLE /net/zcube.west/export/ws/sdcard/hg-sparc/archives. There's an internal webrev up as well at http://jurassic.sfbay/~gd78059/sdcard/ -- enjoy!)

Lint cleanups in kernel

A few weeks ago, I committed changes to start the clean up of a bunch of lint overrides in the kernel, and provided text explaining how to continue the work (along with admonishments to do so.)

I am very happy to report that it seems that some folks took what I said to heart, and over the past couple of weeks there have been a continual stream of lint related fixes, such as this one.

Thanks to everyone who's helping out!

Thursday, July 31, 2008

The TeXTing Swindle

I recently started communicating with some family via "texting", because, for reasons I don't fully understand, they will happily respond to a text message (and send them out on their own), but won't answer a regular phone.

So ignoring the annoyance of trying to draft up a message on the toy 10 key keypad on my phone -- which takes approximately 20 times as long as it would to say the same thing in a voice call, I started thinking about the economics behind the texting fad.

My provider charges me $0.20 per text message. My overlimit minutes cost about $0.25. (Picture messages cost more -- $1.95 each, I think.)

This is the biggest swindle made by phone companies in recent history.

Look at the data involved.

A phone call (voice message) takes the following bit of bandwidth:

  • Assume 8-bit ULAW or ALAW sampling (12 bits compressed to 8 bits)
  • Assume 8 kHz sampling, which is typical for "phone quality" audio
That gives up 8 * 8000 = 64,000 bits per second or 3,840,000 bits per minute

Now, consider a typical text message. Say 12 lines at 40 characters each. (That's more than my phone will display), each character at 8 bits (we'll assume ASCII for now, I can't imagine trying to conjure up CJK characters on my phone!), gives me 12 * 40 * 8 = 3,840 bits. (I didn't try to make the numbers work out as evenly as they do for this example, it just happened!) So, 3840 bits per message.

To a rough approximation a text message costs about the same to the consumer as a minute of talk time. HOWEVER, the bandwidth involved is approximately 1/1000. If everyone is texting instead of using the voice service, the service provider should have about 1,000 times more capacity!

Put another way, text bits are about 1,000 times more expensive to end user!

Now, consider pictures. $1.95 is nearly 10x the cost of a text message or a minute of talk time.

A fully decompressed 16-bit VGA (640x480) resolution picture takes about 4.9Mbits.

To a rough approximation, a picture takes about as much bandwidth as a minute of talk time. The picture is therefore 100x less expensive per bit to send than a text message, and about 10x more expensive than a voice call.

Of course the secret here is that texting is incredibly efficient use of bandwidth to send a message (compared to a voice call), especially with those wacky abbreviations (such as "c u l8r"), but the end user doesn't realize that it is costing the provider far less to send the same message via a text message. (Put another way, the end user has no idea how much network bandwidth is simply wasted with traditional voice calls.)

The mobile providers here have made a real swindle. The fact that they aren't trying to undercut each other on the text messaging does allude rather strongly to "collusion" in market pricing on them -- there should be a lot more competitive pressure to drive the cost of texting down, or even to offer it for free (especially with plans that have unlimited voice, such as mine).

In fact, the phone company should be begging me to use texting instead of voice calls, instead of charging me extra for texting.

Anyone from the mobile companies want to explain how you can get away with this swindle?

Ask your mobile provider why they aren't giving you texting for free with your unlimited voice plan!

Wednesday, July 16, 2008

New Experimental AudioHD Driver

I've just posted the latest experimental version of the OpenSolaris audiohd driver. This driver includes the latest work from the engineers in Beijing, including a widget parser that enables the driver to work on a much larger variety of audio configurations.

As most motherboards these days ship with an audiohd compliant device, this should greatly expand the support for audio on Solaris systems.

The above driver binary also includes suspend/resume support.

Note that this is not the anticipated Open Sound System driver, but rather an extension of the pre-existing Sun driver. It won't work with OSS. However, there is some chance (I've not tested it myself) that this will even work on Solaris 10.

A webrev of the changes should be posted soon.

In the meantime, if you want to give this a whirl, let me know the results. (To test it, just copy the binary objects into /kernel/drv/amd64/ or /kernel/drv and reboot. -- Yes there are ways to do this without a reboot, but the caveats are too many to list here.)

We're interested in any problem reports, as well as notification of hardware configurations where this either did or did not work. Thanks!

[ update 10/13/2008: I've changed the link to the files list so that readers can find the latest version posted up on the site ]

Tuesday, July 8, 2008

NIC driver family trees

This may be interesting to some people -- and its recorded here for posterity. A number of NIC drivers in Solaris have been developed using "copy-paste" approaches. As a result, there are a couple of family "lines" for NIC drivers.

le (ancient 10 Mbit only) driver, is the common root for one heritage. From le was developed hme. From hme were developed qfe, eri, and gem. From these was developed ce. And ultimately, from these was developed nxge (although nxge shares little with its ancestors anymore.)

Another heritage goes back to ancient dnet (tulip) drivers on x86. dnet begat dmfe, which begat bge. There are probably other drivers that bge spawned, but its hard to know for sure how much copying may have occurred.

afe and mxfe were developed pretty much independently from the others, although afe was developed first, and begat mxfe (which is for very similar hardware -- a good counter example for a case where copying the code made sense -- of course I'm also the original author of both drivers). (They started life as pure DLPI from-scratch drivers, but migrated to GLDv2, and later to GLDv3.)

Murayama's drivers (e.g. sfe) owe some ancestry to Linux, although we're told that all the code was changed so that no derivative code from Linux remains.

I believe e1000g owes its ancestry to a combination of Linux code at Intel, and unique code written just for Solaris. (There was another driver, ipge, for the same hardware, which was part of the le/hme heritage, but it died in late toddlerhood.)

The wifi drivers owe most of their existence to the work done in FreeBSD and OpenBSD.

The other older drivers (pcelx, elxl, pcn, etc.) appear to share little if any heritage with any of the other NIC drivers.

Cut & Paste Device Driver Design

A common approach to device driver design is to start with a device driver for a similar piece of hardware that someone else wrote, and copy it, then modify it. However, this approach is fraught with peril, and I thought it might be a good idea to record my thoughts on this for posterity. If this post saves just one future developer from falling into this pit, it will have been worthwhile.

The first problem is that most device drivers have a number of non-trivial "features" in them that relate to the specific issues of a particular piece of hardware. Often this means that there are internal design features of the driver that are not ideal for a new driver. (A classic example would be nxge's use of particular hacks relating to the location of it in relation to the IOMMU on certain Niagra processors. No other driver is ever going to need that.)

The second problem is that most device drivers are not 100% bug free. The copy-modify approach to coding tends to lead to bugs being copied from one driver to another.

The third, and possibly most significant problem, is that when the copy-modify approach is used, the author doing the copying rarely has a complete understanding of all of the original source driver's features, and the end result is a new driver that the original author isn't aware of (and isn't maintaining), and with code that the new author might not fully understand.

A fourth problem is that real drivers are complicated. I frequently hear advice given to new NIC driver authors to "just copy bge". That is terrible advice to give to someone who may be trying to get their head around a new DDK. As a real driver, bge carries a lot of legacy baggage with it, that new drivers definitely shouldn't repeat. All that baggage obfuscates the original code, and the would-be-copier may have little ability to contain the entire device driver in his head.

This leads to the fifth problem, which is that copying repeats all the optimization steps, but elides the measurement and experience that led to them. This violates one of my core design principles, which is KISS (keep it simple, stupid) -- design for simplicity first, and optimize only when proven necessary. (Example: does a 100MB NIC really need to have the logic associated with "buffer loan up" in it? Almost certainly not! And in fact, despite the fact that many NIC drivers have such logic in them, dating back to older architectures, the loan up actually has significant problems, and results in lower performance than the far simpler unconditional bcopy.

A sixth problem is that this approach can often lead to reproduction of design mistakes or limitations. For example, if one were to build a new driver for SPARC hardware by copying hme, the end driver would likely not be 100% portable to x86. (And even if it were made portable, the first pass would probably be quite suboptimal from a performance standpoint owing to hme's use of SPARC-only dvma interfaces.)

The upshot of all this is that I would strongly encourage prospective device driver authors not to start a new driver by copying source code from another driver.

A far better approach, IMO, is to start with a "skeleton driver" (Writing Device Drivers has some such skeleton drivers for different kinds of drivers) and flesh code as you go. Ask questions, on the mailing list as well, if you're not familiar with a particular type of driver.

For folks who might argue that starting fresh from a skeleton violates code-reuse and increases development costs, I would say that

  1. The end driver will almost certainly be more maintainable, as the author will understand every line of code.
  2. The cost of developing a new driver from scratch is probably not that high compared to the copy-modify approach (unless the source and target drivers are for devices that are very very similar. )
  3. The educational benefit of taking this approach is very great as well. I can think of no better way to truly understand the DDI/DDK than to write a device driver from scratch.
  4. For commonly reused code, a library with well defined interfaces is a far far better approach. The driver frameworks in Solaris such as SCSA and GLD are good examples of this. (Third party approaches such as Murayama's GEM also qualify in this regard.) This way there is only one real copy of the code, and clear interface boundaries make it possible for multiple parties to reuse the code safely, without depending on "implementation details".
There are some folks who "start from scratch" - I hope I'll see more of them in the future.

Thursday, June 12, 2008

ISA sbpro driver removed

Starting with build 93, OpenSolaris will no longer have support for the ISA based SoundBlaster Pro, SoundBlaster 16, and SoundBlaster AWE32 devices. This shouldn't impact anyone, although its possible that bochs or qemu users may be impacted. (I've not heard of any such users successfully able to use the sbpro driver in a Solaris guest environment though.)

In the future, we may look into adding support for emulated ESS 1370 devices.

In the meantime, VirtualBox emulates an Intel ICH based AC'97 support, which works with the audio810 driver.

Blogger Bug

Its possible that is just showing up for me, but recently the "toolbar" at the top of my blog has become unusuable. Instead of a normal set of links, I just get a row of repeated Blogger icons. I'm using firefox on Solaris Nevada.

Anyone else seen this?

Wednesday, June 4, 2008

Audio Driver Poll

I'm interested in collecting data about audio devices that folks are using. So, I've a few questions, which maybe folks could send answers to me. Here are the questions:

  1. Is there already support in Solaris for your audio devices/needs? (If yes, stop here, and no need to send me any information -- you're already covered as a "mainstream" user.)
  2. If you're using 4Front's OSS, and you're using a hardware driver other than apci97, hdaudio, ich, via8233, atiaudio, then please let me know what driver you are using, and what the actual audio hardware is (cat /dev/sndstat might be useful here.) (If the audio is built into a motherboard, I'd like to know the make/model of computer, and the rough date -- year -- that you purchased it.)
  3. Do you use/would you use hardware MIDI support?
  4. Are you using digital audio (Dolby Digital/AC3, and/or SPDIF) on your Solaris system? If so, please provide detail.
  5. Do you have any use for more than a single active input source? (I.e. do you need more than a microphone, or line input, to be supported simultaneously for recording?) If so, please provide detail.
  6. Do you have any other audio needs for Solaris beyond normal business/consumer audio? (I.e. I assume that most folks want audio good enough for DVD playback, gaming, and video conferencing.) Particularly, if you want to use Solaris audio to do production work, internet radio broadcasting, etc. then I'd like to know about that.
  7. If the answer to #1 is "no", then if you're willing, I'd like to have your e-mail address so that I might contact you to discuss your particular needs/application further.
Thanks. My hope is to better understand the community needs so we can focus on the things that people need most.

Tuesday, June 3, 2008

Boycott Laptops with Broadcom WiFi

It seems to be a recurring theme, but we keep hearing about different WLAN parts that don't work with NDIS wrapper, or other problems building the NDIS wrapper etc.

I've never used the NDIS wrapper, and I refuse to do so. I also refuse to purchase or recommend any laptop with a Broadcom WLAN part on it, at least until Broadcom changes their position and makes it possible for 3rd parties (even under NDA) to develop device drivers for their WLAN products.

The reason is simple: until laptop manufacturers start losing sales due to people who take the same position, they won't stop including Broadcom WLAN on their products. The loss of a few individual WLAN cards won't impact Broadcom at all, but if Gateway or Dell stops purchasing WLAN parts, then its a whole new ballgame. And the more laptop vendors that we can get to understand that use of Broadcom leads to lost sales, the more impact it will have.

Either Broadcom will take notice, and correct their behavior (e.g. by offering 3rd parties access to device driver info under NDA, writing drivers themselves for platforms like OpenSolaris, or even better, offering up open technical specs), or their WLAN products will gradually fade from popularity so that they are no longer relevant.

To be honest, I don't care which result comes about. But please, don't use or purchase Broadcom WLAN. And to those of you writing neat things like NDIS wrapper and reverse engineering efforts like the bwi driver in *BSD, I'd recommend you rethink whether enabling further sales of laptops bundled with Broadcom WLAN is really something you want to encourage.

(For specific alternatives to Broadcom WLAN, look for either Intel or Atheros WIFI.)

Suspend/Resume Goodness!

Its been a busy week.

In the past week, there have been three separate putbacks:

1) Kerry Shut putback a fix for audiohd to suspend/resume
2) Brian Xu putback a fix for iwk to suspend/resume
3) Judy Chen putback a fix for ath to suspend/resume

The upshot of this is, if you have a laptop with an Nvidia graphics card, its a fair chance that your laptop will support suspend (and resume!)

A big thanks to everyone who's been working on this.

Wednesday, May 21, 2008

Brussels NDD compatibility code cleanup

I've just putback the changes to afe and mxfe to rip out the driver-private ndd support code and replace it with much cleaner and simpler mc_setprop(), mc_getprop() property access functions supplied by Brussels. For common link parameters Brussels does the NDD compatibility support for us. Yay! Drivers can be smaller.

There's a couple of opportunities here for folks to contribute driver improvements:

1) convert existing NIC drivers to the newer framework. E.g. rge, dmfe, maybe others.... (hme and eri for sure, but they may be hard due to the plethora of driver private properties they support via ndd).

2) try hard to remove private driver ioctl() support in favor of Brussels property functions

3) ADMtek centaur parts can support flow control, on certain hardware (pretty much anything shipped in the past 5-7 years.) Adding support for this in afe might be a relatively simple project, especially for someone familiar with ethernet flow control.

Contact me if you want to work on any of the above.

-- Garrett

Friday, April 25, 2008

New Project Direction -- Audio

So, since some folks may be wondering what I'm up to, I thought I'd briefly mention it here.

I've been asked to serve as tech. lead on the project integrate the software from 4Front's Open Sound System into OpenSolaris.

Surprisingly, this task isn't quite as straight-forward as it might seem. There are a number of outstanding issues that have to be resolved before the project can integrate, and we're working frantically to get them all resolved. We've also staffed up the project enough to increase the man power significantly beyond what was associated with in the past. So we are looking to drive this project to successful conclusion soon.

I'll have news about this in the future -- watch this space. But I will say this much -- it looks like in the not-distant-future there will be the ability to use OSS APIs from userland applications on *all* Solaris systems. This includes systems with older chips not supported by 4Front today, and Sun Ray thin client systems. Stay tuned.

(The other upshot with this project is that it is taking a great deal of my time, so my participation in other forums may appear to have dropped off -- but that is only so that I can devote as much of my time to making the OSS project a success. This is the same reason that I will not be attending the OpenSolaris Developer Summit this go around.... anything that detracts from getting work done is being set aside for now.)

SDcard Status Update

For those of you who've been wondering what happened to the SDcard work...

The technical work finished quite a while ago. However, due to some vague language in the disclaimers associated with the SDcard simplified specifications, Sun has decided it is best if we are a member and have a full license to the SDcard specifications (although only the simplified specs were used in its development.)

Of course, this got the lawyers involved in reviewing license agreements and membership agreements, and purchasing machinery engaged, since there is now a transfer of funds involved. (The funds transfers have already been approved.)

The very last hurdle to having this stuff in OpenSolaris is clearing the hurdles with the legal group (and the latest is some concern about trademark rules associated with the SDcard org.) Once we get those final hurdles cleared, hopefully I'll be able to putback the code.

And yes, its all Open Source -- CDDL. Watch for in build 90 (hopefully).

(The rule that any time you involve lawyers in a project, take your original time estimate, double it, and move to the next largest unit of measurement has just about held true in this particular case.)

Friday, March 28, 2008


This week (Tuesday and Wednesday) my father took my 8-year old daughter to Joshua Tree National Park to do some rock climbing. She'd done some simpler climbing before, briefly, a few years ago, and had enjoyed it. (Of course, she did awesome, and everyone around seemed quite impressed by her awesome instincts. Watching her route-find, and use hand holds and moves with flexibility that I can only dream about being able to do was very, very cool.)

The added bonus here was that I was invited to go along as well -- I had never been rock climbing before, and I was anxious to try it myself. (Dad's been climbing for about two years now, and talking about it pretty much continuously since -- now I think I know why.) It was awesome! First off Joshua Tree National Park is absolutely amazing... and it's only about 90 minutes away by car from where I live -- I can't believe I've been missing out on this. (Even if you don't rock climb, there are some beautiful hikes, world-class rock scrabbling -- which is basically half-way between climbing and hiking -- no rope required -- usually, and the natural beauty of the place is astonishing.)

But what was really cool was the climbing. Over two days, we did a number of different climbs (all top-rope climbs), varying from about 5.4 to 5.7. (This is a scale of difficulty, which is too much to explain here.) Prior to the 5.6 and 5.7 climbs, I recall looking up with butterflies in my stomach thinking, "I'm going to climb what? Surely you jest!" (Looking for a foothold on a vertical face, that might be less than a quarter inch wide... and then actually being able to use it to hold your entire weight... well you've got to try it to believe it. Climbing shoes stick like glue.)

During the climb, the butterflies completely vanished, and I was able to focus on getting the job done. (Probably because I never looked further down than my next foothold...)

The best part, after having done it, was the endorphin high at the top, having actually done the climb without giving up, and without actually falling (though a fall is only a couple of inches with a belayed top-rope). Its a huge sense of achievement. To anyone who's not tried this before, I highly recommend it.

Yeah, I'll be going back. It was cool out-climbing Dad (gee, wonder where I got that competitve gene) on the final 5.7, but I'm disappointed that I didn't try one of the 5.9 routes he did on the first day, and I definitely want to go back and do the multi-pitch climb that we turned back on after Brandy got an understandable case of the jitters and chills. (Hanging out on a windy ledge about nearly 100 vertical feet up, knowing that there were three more pitches to go, I certainly sympathized with her sudden onset of acrophobia.)

Congrats to the new OGB

The results of the OpenSolaris 2008 ballot are in -- congratulations to the members-elect. It looks like a solid group of folks, and I am encouraged for the new year! (On a side node, I'd like to have seen a bit more representation from non-Sun employees, but the elected members are all folks I believe have a high level of integrity, and will serve the community's interests well.)

Monday, March 10, 2008

I voted!

I just recorded my vote in OpenSolaris. If you're a Core Contributor, please go to for instructions to register your vote!

For the curious, I voted FOR the two amendments, and my priority list is for a public bug system, public RTI system, SPARC build farm, x64 build farm, and to clean up inactive CGs.

I am not reporting my OGB selections, other than to say that it included a mix of candidates from Sun and non-Sun candidates, and included some former OGB/CAB members, and some fresh faces.

Friday, March 7, 2008

return of iwk

Owners of laptops with Intel 4965 802.11n hardware will be glad to know, iwk has returned. Hopefully, all the legal confusion is sorted out properly this time, so it should be here to stay. For very small technical changes, there was a lot of work involved to make this happen, and a big thank-you to everyone who got it done, and to the community who've been patient with us while we made sure we were Doing The Right Thing.

Now I just need to get one of my own.

Monday, February 4, 2008

What me, impulsive? Nah....

Well at least I'm not the only one in my household. But it can be really fun. The past week has been a great example of this.

For our 4th anniversary last month, my wife and I bought a 56 gallon freshwater setup. At the time we had a 10 gallon setup and 2.5 gallon that we were using to house babies from our livebearers. (Platy/swordtail hybrids, I think.)

That was about 3 weeks ago.

Today, we have a 46 gallon bow front, a 20 gallon, and another 10 gallon. Plus the 56 gallon and original 10 gallon tank. (Now both my of my daughters have their own 10g freshwater setup, the wife has the 46 g for her community ... mostly she wants a home for more more mollies and a peacock eel, but also probably some angels and a few red-eye tetras.)

The pet store had a sale. We uhm, went a little crazy. (A complete 20G freshwater setup was only $40.)

The 56 gallon display tank (24" tall) got converted to saltwater. This is my first marine tank. I only got it filled up last night. (And let me tell you, at about $5/lb, live rock is expensive. Between live rock, live sand, and water -- 89 cents/gallon, I've already spent about $300, and I've not even put any fish in it yet! That doesn't include the tank and equipment of course.) The setup is going to be a FOWLR (fish-only with live rock) tank.

To really get the full picture though, you need to picture me standing out in the cold wind and rain, in wet jeans, barefoot, hosing the 56 gallon out to make sure I've flushed out any freshwater bacteria properly. I must be half nuts. But Debbie came out and helped me, so I'm not the only one.

What is really cool is that my wife has had as much fun with this as I have. I am fortunate indeed to be married to a woman who enjoys so many of the same things I do.

Now we just need to wait.... and wait... and wait.... (New salt water tanks need to "cycle" for about 3-4 weeks before adding fish.)

Friday, January 25, 2008

Brussels putback

This post discusses the 2nd flag-day putback yesterday, which is Brussels (phase I). Brussels also changes the way NIC drivers are administered, but it is focused on simplifying and centralizing the administration of network driver tunables -- these are the values used to tune the device itself, or in some cases, the link layer properties. The most common of these tunables are the values associated with duplex and link speed settings.

Historically these values have been configured with ndd(1M) or driver.conf(4). Many people know how I feel about those methods, but let me just reiterate: "ndd must die!" (And driver.conf, as well.)

The Brussels putback represents another opportunity for community members interested in kernel programming though. A lot of these NIC drivers need to be converted to use the property access methods that Brussels offers, and have the ndd support ioctls removed. (And yes, I strongly desire to see the ndd(1M) ioctl support removed from drivers. A follow-on phase for Brussels will offer ndd compatibility at the Brussels layer.)

Brussels provided a conversion of bge(7d), but many other NIC drivers remain. I plan on converting my two drivers, afe(7d) and mxfe(7d), as well as a few drivers that are still closed source (iprb(7d) and rtls(7d)). But there remain many others. And conversion of a driver to support Brussels is just the sort of bite-sized task that is great for learning how to develop in the kernel. Some possible drivers to convert are sfe(7d), rge(7d), and nge(7d). If you're interested in working on one of those, let me know (you need the hardware, though!)

Clearview/UV putback

Folks watching Nevada putbacks will have noticed at least 3 flag days in the past 24 hours. I want to take a second to talk about the first of them. (I'll talk about the second in a follow up post.)

The first, Clearview/UV, is about providing GLDv3-like features to legacy NIC drivers, and about providing friendlier names to device drivers. I will confess that I've not had a chance to play with any of these features yet, but I think that they are likely to be one of the more important putbacks to OpenSolaris this year. This putback fundamentally changes network administration by offering the ability to use "logical naming" for network device drivers.

The other important thing here is that some folks may believe that the Nemo Unification offered by Clearview/UV means that those legacy drivers don't need to be converted. This is not true. Conversion to GLDv3 still offers significant and tangible benefits to network device drivers:
  1. Performance. The translation layer that Clearview provides adds a performance hit for legacy drivers. Its also the case that legacy NIC drivers are unable to benefit from several of the performance benefits that GLDv3 offered (direct function calls, mblk chaining, etc.)
  2. Full VLAN support. Legacy drivers that don't support the undocumented VLAN features aren't able to offer full size VLAN frames. VLANs still work, but you have to shrink your MTU by 4 bytes.
  3. Certain upcoming GLDv3/Crossbow features. Legacy drivers won't be able to take advantage of upcoming features in GLDv3 from Crossbow. These include various interrupt mitigation techniques and multiple hardware ring support.
The upshot is that the Nemo Unification represents a band-aid for legacy NIC drivers. If you own the code for one or more of these, you really should still have a plan for GLDv3 conversion, unless your willing to accept that your driver is a second class citizen in OpenSolaris. (Note that GLDv3 is still Consolidation Private. Figuring out that Corollary to that fact is left as an exercise to the readers.)

Folks that want help with such a conversion should contact me.

Wednesday, January 9, 2008

making good on a promise (Velocity Networks == bad)

My former Internet service provider, OChosting, was recently purchased by a much larger company, Velocity Networks.

As part of that acquisition, they moved my e-mail to some more central server. However, they screwed it up really really badly. The DNS MX records for my domain pointed to an old server, but the CNAME I was using for IMAP was pointing to the old one. The helpdesk was completely useless/powerless to fix the DNS records. (As part of this they were transitioning the old systems to a new management system, ala CPanel, as well. The helpdesk people were only able to deal with accounts that had been transitioned.

Finally I told them they'd not only lose my business, but I'd post my negative experiences here if they didn't get someone to help me quickly. That much they did. But ultimately, the promise that DNS records would clear up when caches flushed never materialized. Two weeks later their servers are still giving out incorrect DNS information. I've been able to access my e-mail by manually editing my /etc/hosts file, supplying a workaround IP address. (I have noticed that their IMAP server has gotten significantly slower since the transition as well. It can take up to 2-3 seconds to delete a message sometimes. Typically it takes about 1 second for the IMAP delete to occur. On my other servers IMAP deletes appear to be "instantaneous".) This doesn't work well for my wife though, and I'm fed up with trying to force feed these guys a clue.

I will point out that I was paying these bozos over $20/month for very modest hosting needs, which is 2-3x the typical market rate. And I'd been doing this for about the last 5 years, with nary a service call in the interim.

Anyway, now I'm making good on my original promise to them.

Anyway, I've been able to recover all my old e-mails (at least the ones that didn't bounce, and who knows how many of those occured!), and I've given my business to Bluehost as of about two days ago. So far, it seems to be going quite well. My e-mail performance seems to be good, and the software they have deployed for CPanel is a bit more sensible as well. (For one, they seem to understand the problem of matching TLS/SSL certificates to hostnames when used for IMAP or SMTP.) I'm also paying $7.95 a month (full year paid in advance) instead of $19.95, and my disk and throughput quotas are much much higher (not that I need them).

I would strongly urge folks considering ISPs to avoid Velocity Networks or any of their affiliates if at all possible. My experience is that they are clueless, and their helpdesk staff are completely hobbled by a combination of restrictions on what they can perform and their own lack of ability.

A big kudos to Bluehost, as well.