Applications like VLC do not detect S/PDIF as supporting AC3 Passthrough. Instead the audio gets decoded and sent as 2 channel PCM. I'm not sure exactly where this would be implemented but I'm guessing the driver has to publicize that it supports AC3 and PCM instead of just PCM.
I should add that if anyone has any ideas I'd be more than happy to make source code modifications and test. I poked around a bit and it appears that apple uses 2 different constants for AC3, the AC3 one is "ac-3" and there's another one ...1937AC3 which maps to "cac3". I read that cac3 is the right format for bitstreaming but VoodooHDA does not make reference to it. Anyway I'm certainly no expert so help would be greatly appreciated!