Friday, March 4, 2011


I just pushed this for Dan McDonald into illumos:

changeset: 13297:4b9dc4ca8e9f
tag: tip
user: Dan McDonald <>
date: Fri Mar 04 13:57:09 2011 -0800
701 UNMAP support for COMSTAR
Reviewed by: Garrett D'Amore <>
Reviewed by: Eric Schrock <>
Reviewed by: George Wilson <>
Approved by: Garrett D'Amore <>

This change represents a significant new feature in COMSTAR and ZFS, which will greatly benefit people use SCSI target mode functionality in situations involving over-provisioning. More on that in a minute...

The feature itself was developed by Sumit Gupta for Nexenta, and is part of our upcoming 3.1 release of NexentaStor.

Subsequently, Dan took ownership of that code, and working with Eric and George (who are well established ZFS gurus and had significant and useful feedback) improved it still further, and got the code into illumos proper. I believe this represents the first significant ZFS feature to go into the tree since the illumos fork, and also amply demonstrates the collaboration in the illumos community.

I'm looking forward to more collaboration like this in the community.

Now, what does this feature give you? Well, if you're using SCSI Target Mode (either via iSCSI or Fibre Channel or FCoE) to serve up storage to systems running NTFS or ext4, you will be able to make better use of your storage.

Traditionally, when a file was deleted from a filesystem, it was mostly a matter of book-keeping in the meta data in the filesystem. There was nothing to note this in the underlying storage.

With newer SSDs, and with COMSTAR, the ability to get back this notification is incredibly useful. SSDs want it to do garbage collection or other optimizations thereby improving performance.

COMSTAR wants it because now when your thinly-provisioned zvol gets the notification, we can return the storage back to the pool. Prior to this change, the zvol could only grow, it could never shrink. Now, we can give storage back to the pool when you delete a file on the initiator. This is huge in environments running with a lot of VMs using thinly provisioned storage with overallocation.

Anyway, this is now in illumos thanks to Nexenta, and notably, Oracle doesn't have it. Of course, they are welcome to pick up the code for it, but they will need to follow the terms of the CDDL if they choose to do so, the same as everyone else.


rty said...

Hello, Garrett.
Please clear this news for me.
I will get benefit using new SSD (supporting TRIM) for OS install?
Or for L2ARC cache?
Not using COMSTAR.

Garrett D'Amore said...

rty: No. At the moment this is *only* for COMSTAR. Soon we'll get this for ZFS and then you will see a benefit for SSDs.

Vinayak said...

From which build does SCSI UNMAP is implemented in opensolairs?

Vinayak said...

From which build onwards SCSI UNMAP is supported into opensolaris?

Vinayak said...

From which build of opensolaris SCSI UNMAP is implemented?

Garrett D'Amore said...

Your question (Vinayak) doesn't really make sense. OpenSolaris is dead, but when it was still an active project there were a number of releases at different points.

illumos started from build 147 and has continued from there.

JimKlimov said...

Garrett, sorry, but I think rebuffing interested people like this was impolite. And you might scare off potential faint-hearted community members who have not yet figured out the multiple distro naming changes ;)

I hope Vinayak's question can be interpreted in illumos/OpenIndiana terms - like "in which pkg version of ZFS package is the feature included for public testing/usage?" with the answer being perhaps an URL like or whatever :)

Other than that - great news and a killer feature, I guess. Thanks ;)

By the way, how does COMSTAR learn that NTFS or Linux ext* freed some block? Does it intercept TRIM notifications or whatever the virtualized OS intended for an SSD driver?

In short, would this work with any guest OS or with specific ones - like "Windows 7 and above" or "Linux kernel 2.x.y.z and above built with TRIM support"?

Thanks again,
//Jim Klimov

Garrett D'Amore said...


1. My reply to Vinayak was not meant to be be a rebuff. I honestly did not understand (and still don't) the original question -- we don't have "builds" as such for illumos, and I'm not aware of any build of OpenIndiana that has incorporated this change yet.

2. COMSTAR gets it when filesystem on the initiator notifies the underlying SCSI disk driver, which should then issue a SCSI UNMAP (which is the SCSI moral equivalent of SATA TRIM). What this means is, it will work with any initiator which issues SCSI UNMAP, which is a standard T10 request. (This includes Windows NTFS, Linux ext4, and forthcoming VMware products.A)

T said...

Great stuff. I've been hoping for this for some time.

I wonder if it is possible to build support for this into the NFS server as well in the future? I realize it's not as straight forward, but would be very useful for those of us hosting VMs over NFS rather than SCSI targets.

Garrett D'Amore said...

Over NFS, you wind up not needing to overprovision, so it isn't the same problem.

But we will also be doing SATA TRIM and SCSI UNMAP for local ZFS resources (and NFS/CIFS shares of those resources) soon.

T said...

Thanks for the response, I suppose I should be more clear.

As you said, NFS doesn't need over-provisioning. However, when you host VMs in image files which are then stored on NFS, you run into the over-provisioning problems inside the guest VM. I suppose this can also still occur if you're hosting on a SCSI target.

I guess what I'd really hope for at some point is the ability to "tunnel" these UNMAP requests over whatever protocol in order to get it back to ZFS?

Anyway, I really appreciate the effort you've put into continuing to improve zfs.

NiTRo said...

Garrett, by forthcoming VMware products, you mean VMFS 5 ?