Because HLE::Source is initialized as an array in the member initializer, it is hard to let it accept the reference on ctor, so it has a second init stage performed by DspHle::Impl::Impl
The current code inserts and deletes elements from the beginning of the audio buffer, which is very inefficient in an std::vector.
Profiling was done using VisualStudio2017's Performance Analyzer in Super Mario 3D Land.
Before this change: AudioInterp::Linear had 14.14% of the runtime (inclusive) and most of that time was spent in std::vector's insert implementation.
After this change: AudioInterp::Linear has 0.36% of the runtime (inclusive)