[Nexus 5X Themes and Apps] chdloc:
You don't need to design a filter to apply attenuation across all frequencies (allpass) if you can live with one dB increments, use the IIR input gain control instead:

<ctl name="IIR1 INP1 Volume" value="84" />
<ctl name="IIR2 INP1 Volume" value="84" />

As always, 84 equals 0 dB and it adjusts up and down in one dB increments, so 83 equals -1 dB and 85 equals +1 dB, etc. Note that you have two sets of IIR input controls in your mixer_paths.xml file. You should get rid of one set.

In general there are three domains where distortion is introduced:

digital: try to avoid digital clipping, i.e. digital amplification to the point where a signal gets clipped in the DAC

analog: even a decent power amplifier can introduce audible distortion at high gains

physical: the transducer is fed with too much power such that either the magnetic coil travels outside the linear magnetic field of the magnet or when the membrane itself seizes to move coherently and breaks up in modes. This also happens when a membrane is forced to reproduce frequencies that it physically can't, i.e. below its resonance frequency.

So how can we get the distortions under control? In your current mixer_paths file you are adding gain to the earpiece to try to match the levels between left and right loudspeaker, i.e. "RX1 Digital Volume". You are also adding up to 6 dB of gain by mixing left and right channels for the main loudspeaker. As you found out, you can counteract some of that added gain by introducing some attenuation via the IIR filter. Bottom line, as you alluded to, a compromise needs to be found between maximum playback level and acceptable distortion. This compromise depends on both the audio material and individual preferences. Not an easy problem to solve and you may need to give the user the option to change the gains themselves. An Aroma installer would offer this ability.

Also, brave souls can also try to design their own filters.

With respect to item number 2, Qualcomm exposes some coarse-gained control of the respective power amplifiers, i.e. "SPK DRV Volume" for the main loudspeaker and "EAR PA Gain" for the earpiece. You can try to trade-off analog and digital gain. Simply run

tinymix "EAR PA Gain"

and

tinymix "SPK DRV Volume"

while playing media for current and possible values.

With respect to number 3, all we can do is limit the frequencies to the transducer that it cannot reproduce. Other than that we are stuck with what we have.

Crossfeed would actually be the perfect solution as you could knock down the gain by 6 dB before the mixing stage. Did you follow my instructions here? If you did and couldn't hear a difference tells me that the Nexus 5x kernel (just like the Nexus 5) does not expose the required plumbing and that you would need to apply the kernel patch I outlined in the referenced post.

Other than that, distortion (and of course playback levels) can be controlled with the Volume rocker (can be seen as pre-gain). At the top playback levels, each "tick" corresponds to 3 dB, so if you avoid the last two ticks, i.e. max out the audio signal at -6 dB full-scale going into the mixer, then "digital distortion" can be avoided completely according to your current mixer_paths.xml file.