The Stream.ReadByte method is implemented by allocating a 1 byte buffer and calling into Read(byte[], int, int). Override ReadByte in derived classes to avoid needing to allocate this small temp buffer.
Also, fix some bugs in the stream implementations. Remove Write capability from MergedStream that didn't make sense. Add guards into SegmentStream to ensure reads and writes belonged to the segment - otherwise a reader or writer could access regions of the base stream that were outside the intended segment.
- Where it is accessible, use the length of the stream to presize the MemoryStream to the correct size.
- Instead of copying out the result via ToArray, grab the underlying buffer via GetBuffer and use that to create the sound source.
This avoids extraneous copying of the array containing the audio.