ZFS disk monitoring...

So I've posted this on zfs-discuss at opensolaris dot org, but its been suggested I mention it here too.

It turns out that the ZFS/FMA integration doesn't pick up on drive removals for most disk devices until the filesystem attempts to perform some I/O to the drive. This is rather unfortunate, because if a file system is not busy, you might suffer a loss of redundancy and not find out about it until too late.

It also means that you won't know about failures of hot spare devices until you need to put them into service, since by definition they are idle. (Note: as an exception running periodic scrubs should detect this too, although scrubs are highly intrusive to the overall I/O load on the system and probably should not be performed too often as a result.)

I'm told the Oracle 7000 series appliances have a solution for this problem, but of course the source for that is not in OpenSolaris. (Apparently there are quite a few differences in the core OS between the 7000 series and vanilla OpenSolaris -- unfortunately we can't know because -- unlike with NexentaStor -- we don't have access to the kernel source tree!)

This is not good for folks who use ZFS with ordinary Solaris 10 or OpenSolaris... or with derivatives such as NexentaStor.

To address that problem, I've developed a some code called "zfs-monitor" that periodically monitors the health of any physical vdev (disk) that is part of a ZFS pool (hot spare, log, or real device). This code is implemented as an FMA module. When a disk goes offline, zfs-monitor detects it, and triggers an FMA event, which allows ZFS to do the right thing. This means if a disk goes away, even if it isn't in use, whatever action is appropriate will be performed. (Logged in FMA fault logs, and if appropriate, a hot spare will be recruited to replace the failed or offline device.)

This code is part of NexentaStor 3.0.3. As there are some semantic differences of opinion (what constitutes device failure versus intentional removal by an administrator), the code is unlikely to be pushed into ON without further change. (At the same time, I've fixed a different problem in the ZFS FMRI parsing code, and I've submitted a request to get that fix integrated -- but I've not heard back from anyone at Oracle who is willing to sponsor the change yet.)

I'm happy to share the code for zfs-monitor to anyone who requests it. (In fact, you can examine the code in our open Mercurial repository directly!) Note that for it to work properly, you also will need the fix for the ZFS FMRI parsing bug just mentioned.

At Nexenta, we're committed to innovating and improving upon the great foundation of ZFS and OpenSolaris, and to the reasonable extent possible, we want to share those innovations with the greater OpenSolaris community. Hopefully changes like this demonstrate this commitment in a tangible fashion.

Comments

grant beattie said…
very cool. will this also make its way into NCP?
July 7, 2010 at 5:24 PM
Unknown said…
Yes. In fact, NexentaStor is based on NCP -- the nexenta-gate I referenced in the post is used as the source for both NCP and nexenta-gate. There are some NexentaStor additions (the storage management software!) and customizations, and the timing of release of bits may differ from NCP slightly, but NS is built on top of NCP.
July 7, 2010 at 10:00 PM
maetu said…
Garret - this is great! :-)
July 8, 2010 at 1:29 PM
Post a Comment

Popular posts from this blog

An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone. This milestone was the integration of  195 Need replacement for nfs/lockd+klm This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at  Nexenta .  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix , and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product. The reason this is such an important milestone, is two fold: First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of th
Read more

GNU grep - A Cautionary Tale About GPLv3

My company, DEY Storage Systems , is in the process of creating a new product around the illumos operating system.  As you might imagine, this product includes a variety of open and proprietary source code.  The product itself is not delivered as a separate executable, but as a complete product.  We don't permit our customers to crack it open, both from the sense of protecting our IP, but also to protect our support and release engineering organizations -- our software releases consist only of a single file and we don't supply tools or source for other parties to modify that file. One of the pieces that we wanted to integrate into the tree is an excellent little piece of software called Zookeeper , produced by the Apache organization.  Like illumos, Zookeeper has a nice non-viral copyleft license, which makes it nice for integration into our product. However, I discovered that as part of our integration, one of my engineers had decided to integrate GNU grep.  Why? Becaus
Read more

The Hand May Be Forced

Well, as you may have read , Oracle has decided that at some point very soon, we're going to lose normal regular access to the source code for OS/Net. (I.e. the Solaris kernel and supporting programs.) While I would have vastly preferred for Illumos to have a cooperative and collaborative relationship with Oracle , it appears that Oracle doesn't value this. In fact, the exact words were from the management at Oracle were as follows: Solaris is not something we outsource to others, it is not the assembly of someone else’s technology, and it is not a sustaining-only product. While I understand the need to own the technology, there are few things that could be stated that show a stronger NIH attitude than this. Its unlikely that there will ever be a way for Oracle and the greater community to have a collaborative relationship. This is a dark day for OpenSolaris -- its effectively dead now. (Its parent, Solaris, lives on however.) How unfortunate. For Oracle that is. Because
Read more