Bombproof taskqs

As part of fixing some recent bugs, I integrated the following into illumos:

        734 taskq_dispatch_prealloc() desired
        943 zio_interrupt ends up calling taskq_dispatch with TQ_SLEEP

The interesting one is the first of these.  The interface is actually called taskq_dispatch_ent(), and is private to the "consolidation" (i.e. for use within bundled code only).

What this interface provides for, however, is a way to bomb-proof your taskq dispatches, if you can arrange for the dispatching state structure (taskq_ent_t) to be allocated in advance.  This means you never have to worry about the possibility of a dispatch failing due to insufficient resources.

What's even cooler, is that the cost of the dispatching is much cheaper; taskq_dispatch() was the hottest piece of code on a very busy storage server.  Now it goes much much faster, because it is just twiddling some linked list pointers and sending a signal to wake up the thread processing the taskq.

More importantly, the fact that we don't ever have to sleep, or have an expensive call into the kernel memory subsystem, has solved some frustrating "stalls" in the storage subsystem.

I encourage developers working with taskqs in illumos to have a look at this interface.  If you can use it, you will simplify your code, and shorten your run-paths.  Both of which are good things.  The only limitation: this interface is not available for dynamic taskqs.  (Which makes sense since dynamic taskqs might need to allocate whole threads.)








































Comments





Post a Comment













Popular posts from this blog









SP (nanomsg) in Pure Go





















I'm pleased to announce that this past weekend I released the first version  of my implementation of the SP (scalability protocols, sometimes known by their reference implementation, nanomsg ) implemented in pure Go . This allows them to be used even on platforms where cgo is not present.  It may be possible to use them in playground (I've not tried yet!)   This is released under an Apache 2.0 license .  (It would be even more liberal BSD or MIT, except I want to offer -- and demand -- patent protection to and from my users.)   I've been super excited about Go lately.  And having spent some time with ØMQ in a previous project, I was keen to try doing some things in the successor nanomsg project.   (nanomsg is a single library message queue and communications library.)   Martin (creator of ØMQ ) has written rather extensively  about how he wishes he had written it in C instead of C++.  And with nanomsg, that is exactly what he is done.   And C is a great choice for implement








Read more










An important milestone





















 Yesterday, with very little fanfare, illumos  passed an important milestone.   This milestone was the integration of  195 Need replacement for nfs/lockd+klm   This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at  Nexenta .  Gordon started the work, picking up bits from FreeBSD  as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix , and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product.   The reason this is such an important milestone, is two fold:   First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of th








Read more










The Hand May Be Forced





















Well, as you may have read , Oracle has decided that at some point very soon, we're going to lose normal regular access to the source code for OS/Net.  (I.e. the Solaris kernel and supporting programs.) While I would have vastly preferred for Illumos  to have a cooperative and collaborative relationship with Oracle , it appears that Oracle doesn't value this.  In fact, the exact words were from the management at Oracle were as follows: Solaris is not something we outsource to others, it is not the assembly of someone else’s technology, and it is not a sustaining-only product. While I understand the need to own the technology, there are few things that could be stated that show a stronger NIH  attitude than this.  Its unlikely that there will ever be a way for Oracle and the greater community to have a collaborative relationship. This is a dark day for OpenSolaris  -- its effectively dead now.  (Its parent, Solaris, lives on however.) How unfortunate. For Oracle that is. Because








Read more