O_SYNC behavior not honored
UPDATE (6/21/2010): This problem is apparently solved in b142. Probably other builds as well. But I was unable to reproduce this problem with real hardware on b142.
Note that VMware does not honor cache flushing, so VMware (and possibly other v12n users) will potentially still see this issue.
So, it turns out that ZFS in recent (somewhere after build 134 apparently) builds has a critical bug ... O_SYNC writes are not really synchronous. This leads to potential data loss.
I've not yet figured out which change introduced the bug, but I hope to work on it next week.
In the meantime, I would strongly discourage use of post-134 binaries for anything where data integrity is important.
I've filed a P1 bug with Oracle for this issue. I'll be trying to nail it down further next week; if I'm able to fix it before Oracle can, I'll offer up my fix.
I'll post the CR number when I receive the number back.
I imagine that this bug, which is trivially reproducible, will be getting top priority from the ZFS engineers next week.
UPDATE: CR number is 6958848
The link to access it isn't available yet.