The Stream.ReadByte method is implemented by allocating a 1 byte buffer and calling into Read(byte[], int, int). Override ReadByte in derived classes to avoid needing to allocate this small temp buffer.
Also, fix some bugs in the stream implementations. Remove Write capability from MergedStream that didn't make sense. Add guards into SegmentStream to ensure reads and writes belonged to the segment - otherwise a reader or writer could access regions of the base stream that were outside the intended segment.