I've been doing some investigation. I'm seeing a situation where the VMware emulated audioens device behaves rather differently from the real hardware.
For one, it seems to insist on using real interrupts. In particular, the sample count registers do not appear to be updated unless one receives and acknowledges an interrupt. (By toggling the interrupt enable bit.) This means that this virtualized device will never be able to run "interrupt free" like the other audio devices (or real audio hardware).
For another, it appears that the audio device has some weird dependency on the relationship between the size of the audio buffer, and the interrupt rate (the number of samples at which to interrupt). Using different values gives, strange results.
Finally, I cannot, for the life of me, figure out how to cause the device to actually trigger an interrupt. I've been able to make some progress by simulating a soft interrupt at 100Hz, which is how the interrupt free framework works anyway, but from what I can tell, nothing is causing a real interrupt to be delivered. This is really strange. (Without this functionality, I am able to process audio at a reasonable rate, but it still stutters, and is not really suitable for real-world use.)
My guess is that the virtual device has some weird dependencies that we don't know about. For example, while the hardware spec identifies registers as being 8, 16, or 32 bits wide, and we use those at the right bit widths, other FOSS drivers all seem to just blithely use 32-bit wide accesses. Is there a hidden dependency here?
If any reader from VMware is seeing this, and can help me understand the behavior of the simulated device, I'd appreciate it. I'd like to make audio work in this environment, if possible. I'm pretty close, I think.
Actually, it seems kind of crazy that these environments emulate such complex audio hardware. (For example sophisticated sample rate conversion hardware.) Much better, I think, would be a simple paravirtualization driver that just exposes a simple buffer and some control functions. If someone at VMware wants to work on that as a solution, I'd be happy to help with Solaris support for it. Since these things run isochronously, and chew up a fair bit of cpu when they run, such a solution would probably be quite useful. (For example, its silly to perform multiple sample rate conversions in software... instead we could express native sample rates via a PV driver to the guest, ensuring only a single SRC operation is performed appropriately in the guest operating system.)