I've got an AVI file with kAudioFormatAVIAC3 with a sample rate of 44100 and 2 channels. A52Codec throws an exception because the input sample rate doesn't match the output sample rate. This is in A52/ACShepA52Codec.cpp in ACShepA52Codec::Initialize() around line 45. I see mSampleRate/mChannelsPerFrame of 44100/2 (in) and 48000/2 (out). Stubbing the exception throw, as one would expect, does bad things, but it does convince QuickTime that the codec can handle the audio track, even if the codec is outright lying! This is clearly not a good solution to the problem, but it helps demonstrate what's going wrong.
So I see this big comment in A52/ACShepA52Decoder.cpp about not caring about sample rates and channels, and it seems to be a bit out of date with respect to the code. It talks about specifying zeroes, but instead the code is building up these input/output formats to vary the channels. Unfortunately, the sample rates are never varied. So somehow or other QuickTime accepts the 44100 sample rate of my AVI file as in input format even though the codec is not advertising it as a supported format. The codec does not advertise an output format with a sample rate of 44100 either, so it picks a 48000 output format that otherwise matches.
Adding additional input and output formats to vary the sample rate as well as the channel solves the problem. The result is that QuickTime picks a 44100/2 input format and a 44100/2 output format, they match, and everybody's happy; the AVI file now plays just fine. I can provide a patch or whatever if you want, but it's pretty straightforward and hopefully I've explained it well enough.
