COMSTAR and SCSI UNMAP

I just pushed this for Dan McDonald into illumos:


changeset:   13297:4b9dc4ca8e9f
tag:         tip
user:        Dan McDonald <danmcd@nexenta.com>
date:        Fri Mar 04 13:57:09 2011 -0800
description:
701 UNMAP support for COMSTAR
Reviewed by: Garrett D'Amore <garrett@nexenta.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Approved by: Garrett D'Amore <garrett@nexenta.com>



This change represents a significant new feature in COMSTAR and ZFS, which will greatly benefit people use SCSI target mode functionality in situations involving over-provisioning. More on that in a minute...

The feature itself was developed by Sumit Gupta for Nexenta, and is part of our upcoming 3.1 release of NexentaStor.

Subsequently, Dan took ownership of that code, and working with Eric and George (who are well established ZFS gurus and had significant and useful feedback) improved it still further, and got the code into illumos proper. I believe this represents the first significant ZFS feature to go into the tree since the illumos fork, and also amply demonstrates the collaboration in the illumos community.

I'm looking forward to more collaboration like this in the community.

Now, what does this feature give you? Well, if you're using SCSI Target Mode (either via iSCSI or Fibre Channel or FCoE) to serve up storage to systems running NTFS or ext4, you will be able to make better use of your storage.

Traditionally, when a file was deleted from a filesystem, it was mostly a matter of book-keeping in the meta data in the filesystem. There was nothing to note this in the underlying storage.

With newer SSDs, and with COMSTAR, the ability to get back this notification is incredibly useful. SSDs want it to do garbage collection or other optimizations thereby improving performance.

COMSTAR wants it because now when your thinly-provisioned zvol gets the notification, we can return the storage back to the pool. Prior to this change, the zvol could only grow, it could never shrink. Now, we can give storage back to the pool when you delete a file on the initiator. This is huge in environments running with a lot of VMs using thinly provisioned storage with overallocation.

Anyway, this is now in illumos thanks to Nexenta, and notably, Oracle doesn't have it. Of course, they are welcome to pick up the code for it, but they will need to follow the terms of the CDDL if they choose to do so, the same as everyone else.

Comments

Anonymous said…
Hello, Garrett.
Please clear this news for me.
I will get benefit using new SSD (supporting TRIM) for OS install?
Or for L2ARC cache?
Not using COMSTAR.
March 6, 2011 at 6:39 PM
Unknown said…
rty: No. At the moment this is *only* for COMSTAR. Soon we'll get this for ZFS and then you will see a benefit for SSDs.
March 6, 2011 at 7:16 PM
Unknown said…
From which build does SCSI UNMAP is implemented in opensolairs?
March 16, 2011 at 3:47 AM
Unknown said…
From which build onwards SCSI UNMAP is supported into opensolaris?
March 16, 2011 at 3:48 AM
Unknown said…
From which build of opensolaris SCSI UNMAP is implemented?
March 16, 2011 at 3:49 AM
Unknown said…
Your question (Vinayak) doesn't really make sense. OpenSolaris is dead, but when it was still an active project there were a number of releases at different points.

illumos started from build 147 and has continued from there.
March 16, 2011 at 8:12 AM
JimKlimov said…
Garrett, sorry, but I think rebuffing interested people like this was impolite. And you might scare off potential faint-hearted community members who have not yet figured out the multiple distro naming changes ;)

I hope Vinayak's question can be interpreted in illumos/OpenIndiana terms - like "in which pkg version of ZFS package is the feature included for public testing/usage?" with the answer being perhaps an URL like http://repo.openindiana.org/dev/zfs/123.456 or whatever :)

Other than that - great news and a killer feature, I guess. Thanks ;)

By the way, how does COMSTAR learn that NTFS or Linux ext* freed some block? Does it intercept TRIM notifications or whatever the virtualized OS intended for an SSD driver?

In short, would this work with any guest OS or with specific ones - like "Windows 7 and above" or "Linux kernel 2.x.y.z and above built with TRIM support"?

Thanks again,
//Jim Klimov
March 22, 2011 at 5:40 AM
Unknown said…
Jim.

1. My reply to Vinayak was not meant to be be a rebuff. I honestly did not understand (and still don't) the original question -- we don't have "builds" as such for illumos, and I'm not aware of any build of OpenIndiana that has incorporated this change yet.

2. COMSTAR gets it when filesystem on the initiator notifies the underlying SCSI disk driver, which should then issue a SCSI UNMAP (which is the SCSI moral equivalent of SATA TRIM). What this means is, it will work with any initiator which issues SCSI UNMAP, which is a standard T10 request. (This includes Windows NTFS, Linux ext4, and forthcoming VMware products.A)
March 22, 2011 at 7:40 AM
Unknown said…
Great stuff. I've been hoping for this for some time.

I wonder if it is possible to build support for this into the NFS server as well in the future? I realize it's not as straight forward, but would be very useful for those of us hosting VMs over NFS rather than SCSI targets.
March 22, 2011 at 9:51 AM
Unknown said…
Over NFS, you wind up not needing to overprovision, so it isn't the same problem.

But we will also be doing SATA TRIM and SCSI UNMAP for local ZFS resources (and NFS/CIFS shares of those resources) soon.
March 22, 2011 at 10:09 AM
Unknown said…
Thanks for the response, I suppose I should be more clear.

As you said, NFS doesn't need over-provisioning. However, when you host VMs in image files which are then stored on NFS, you run into the over-provisioning problems inside the guest VM. I suppose this can also still occur if you're hosting on a SCSI target.

I guess what I'd really hope for at some point is the ability to "tunnel" these UNMAP requests over whatever protocol in order to get it back to ZFS?

Anyway, I really appreciate the effort you've put into continuing to improve zfs.
March 22, 2011 at 11:52 AM
NiTRo said…
Garrett, by forthcoming VMware products, you mean VMFS 5 ?
April 4, 2011 at 3:05 PM
Post a Comment

Popular posts from this blog

An important milestone

Yesterday, with very little fanfare, illumos passed an important milestone. This milestone was the integration of  195 Need replacement for nfs/lockd+klm This is work that I originally tasked Gordon Ross and Dan Kruchinin to work while we were colleagues at  Nexenta .  Gordon started the work, picking up bits from FreeBSD as a starting point and gluing it to the illumos entry points.  Dan continued with it, and fleshed out a lot of the skeleton that Gordon had started.  It was subsequently picked up by the engineers at Delphix , and -- after some important bug fixing work was completed -- was integrated into their product quite some time ago -- reportedly its been stable since December in their product. The reason this is such an important milestone, is two fold: First, this represents a substantial collaborative effort, involving contributions from parties across several organizational boundaries.  The level of collaboration achieved here is a win for the greater good of th
Read more

GNU grep - A Cautionary Tale About GPLv3

My company, DEY Storage Systems , is in the process of creating a new product around the illumos operating system.  As you might imagine, this product includes a variety of open and proprietary source code.  The product itself is not delivered as a separate executable, but as a complete product.  We don't permit our customers to crack it open, both from the sense of protecting our IP, but also to protect our support and release engineering organizations -- our software releases consist only of a single file and we don't supply tools or source for other parties to modify that file. One of the pieces that we wanted to integrate into the tree is an excellent little piece of software called Zookeeper , produced by the Apache organization.  Like illumos, Zookeeper has a nice non-viral copyleft license, which makes it nice for integration into our product. However, I discovered that as part of our integration, one of my engineers had decided to integrate GNU grep.  Why? Becaus
Read more

The Hand May Be Forced

Well, as you may have read , Oracle has decided that at some point very soon, we're going to lose normal regular access to the source code for OS/Net. (I.e. the Solaris kernel and supporting programs.) While I would have vastly preferred for Illumos to have a cooperative and collaborative relationship with Oracle , it appears that Oracle doesn't value this. In fact, the exact words were from the management at Oracle were as follows: Solaris is not something we outsource to others, it is not the assembly of someone else’s technology, and it is not a sustaining-only product. While I understand the need to own the technology, there are few things that could be stated that show a stronger NIH attitude than this. Its unlikely that there will ever be a way for Oracle and the greater community to have a collaborative relationship. This is a dark day for OpenSolaris -- its effectively dead now. (Its parent, Solaris, lives on however.) How unfortunate. For Oracle that is. Because
Read more