[Android Software Development] g3v0:
For those who are having trouble and are willing to experiment, here's a general guide to how I got it running on a Pixel 2 XL Oreo 8.1 + Magisk 15.3.

Magisk already had Busybox for Android NDK and Magisk SELinux Permissive Script installed.

NOTE: This is very hacky, not a final product by any means.

Enter at your own risk!

Pre-requisites

Computer (Windows in my case)

adb drivers

Connect your phone to your computer, open cmd prompt (Win+R, "cmd", enter).

Type

adb shell

If you get an error, google how to get ADB to work.

If it works, you should see a prompt with a dollar sign $ at the end.

Type

su

If the cmd prompt hangs for a bit, check your phone for a SU prompt. Grant permission forever.

If there's an error, double check Magisk -> Superuser is not blocking it, and that the Magisk -> Settings -> Superuser Access is set to "ADB and Apps".

If it works, you the prompt should now end with #

If at any point you are bootlooping, you can now recover easier. You would be able to edit files with adb shell EXCEPT the sdcard folder as that's not decrypted after reboot until you unlock your phone.

A quick recover sequence (to disable the module) when your phone is stuck at the boot animation is

adb shell
su
cd /magisk/dolbyatmos
cat > disable
CONTROL Z <--- (actually press those buttons)
reboot

or perhaps shorter

adb shell
su -c cat > /magisk/dolbyatmos/disable
CTRL + Z
reboot

Getting it to boot

Open the latest zip in your standard zip/file manager, and extract the /dolby/custom folder. Put this 'custom' folder on the root of your sdcard.

Flash latest Unity (20180207) in MagiskManager. Don't reboot! Rebooting here bootloops. Logcat says it's missing libradioservice.so

From the zip that was flashed, copy libradio.so and libradioservice.so from the custom lib directories (dolby/custom/lib and dolby/custom/lib64) into /magisk/dolbyatmos/system/vendor. Then copy the 2 vendor lib folders into /magisk/dolbyatmos/system/.

Assuming you extracted the zip and put the dolby/custom folder in the root of your sdcard, you would do

adb shell
su
cp /sdcard/custom/lib/libradio.so /magisk/dolbyatmos/system/vendor/lib
cp /sdcard/custom/lib64/libradio.so /magisk/dolbyatmos/system/vendor/lib64
cp /sdcard/custom/lib/libradioservice.so /magisk/dolbyatmos/system/vendor/lib
cp /sdcard/custom/lib64/libradioservice.so /magisk/dolbyatmos/system/vendor/lib64
cp -r /magisk/dolbyatmos/system/vendor/lib /magisk/dolbyatmos/system
cp -r /magisk/dolbyatmos/system/vendor/lib64 /magisk/dolbyatmos/system

Reboot.

Doesn't bootloop! But app FC's. Also, I noticed I don't get any service. The /magisk/dolbyatmos/post-fs-data.sh file messes with things so I renamed the extension to .bak (to disable it from running).

Now type:

adb shell
su -c dumpsys media.audio_flinger

Go to the very beginning of the output. It says:

Libraries NOT loaded:
Library (null)
path: vlldp
Library (null)
path: atmos

So the libraries aren't being loaded, even though they're in the correct spot and config options look correct. Let's figure out why.

Create a new folder on your desktop for logcats. Open it, and in the address bar, type "cmd" and hit enter. It should open up a command prompt for the current directory.

While connected to your phone, type

adb reboot && adb logcat > logcat1.txt

This should reboot the phone and start generating a logcat. We want to know why the library isn't being loaded at boot, that's when the logcat matters!

After you get to the lockscreen, hit Ctrl-C in the cmd prompt to stop logging and write it to file.

Open up the newly generated logcat1.txt in your favorite text editor (I use Sublime), and Ctrl+F for "libatmos".

The first few matches are just Magisk bind_mounts, that's unimportant. But the last find...

E EffectsFactoryConfigLoader: Could not dlopen library /vendor/lib/soundfx/libatmos.so: dlopen failed: library "libmedia.so" not found
E EffectsFactoryConfigLoader: Could not dlopen library /vendor/lib/soundfx/libswvlldp.so: dlopen failed: library "libmedia.so" not found

So it tries to load the library, but the dependency libmedia.so is not found!

So I looked for libmedia.so on my phone, and found it in /system/lib instead of /vendor/lib!

I copied the libmedia.so files from /system/lib and /system/lib64 into /magisk/dolbyatmos/system/vendor/lib + lib64 folders.

adb reboot && adb logcat > logcat2.txt

Failed because of another lib!

I repeated this process for a few more libs, but wanted to see if this would even work, so I copy pasted the whole lib folders (waste of space, etc, I know). But the libraries did load and the app didn't FC!

This is how I copy-pasted it. Note that a basic copy and paste in a file manager WILL NOT WORK since the magisk partition is shrunk to be only a little larger than it's contents after module installs.

So download and push Magisk-IMG-Resizer.zip and Magisk-SuperSURecoveryManagement.zip (included in atmos zip) onto your sdcard.

In adb shell, type

echo '1500' > /data/.imgsize

We're making it a little huge to ensure there's enough space. Magisk should automatically shrink/resize after a reboot, so we gotta do this in recovery!

Reboot into recovery ("adb reboot recovery" or just "reboot recovery" in adb shell). Flash Magisk-IMG-Resizer, to resize to what we set above. Then flash RecoveryManagement, to mount it so we can manipulate it.

Important: Press back a few times to go to the main TWRP menu, go to Mount, and select System and Vendor; Data should already be selected. You can also select mount system read-only if you like.

With your phone connected, go to adb shell. The prompt should be some random corrupted symbols, and running commands like cp gave me weird linker errors (even though it would execute). So don't worry about those error messages. Copy and paste the following into the shell (right-click):

cp -r /system/system/lib/* /magisk/dolbyatmos/system/vendor/lib
cp -r /system/system/lib64/* /magisk/dolbyatmos/system/vendor/lib64
rm -f /magisk/dolbyatmos/system/vendor/lib/libc.so
rm -f /magisk/dolbyatmos/system/vendor/lib64/libc.so
reboot

Press enter just in case the last line didn't execute.

I remove the copy of the libc.so file since it crashes my camera.

It reboots and the Atmos app shouldn't FC anymore (though the UI itself is a little buggy).

HUGE PROBLEM though! dumpsys media.audio_flinger shows us that while VLLDP is running correctly in session 0, Atmos is running in a different session and effect chain!

The challenge now is to get it running correctly.

After a lot of experimentation, I had it working for a bit, but Viper wasn't processing, so I ran the deep buffer cleaner module, said it detected an old version of aml and wiped the configs >:(

To get it working, go to /magisk/aml/system/vendor/etc/audio_effects.xml

Remove the <!-- and --> surrounding the music_helper effect. Then clean up the music section. Remove the '^' at the start of the line, and clean the rest to look like

<stream type="music">
<apply effect="atmos"/>
<apply effect="music_helper"/>
</stream>

Now open /magisk/aml/system/vendor/etc/audio_policy_configuration.xml in your favorite text editor, do a search and replace for

deep_buffer,

(include the comma) and replace with nothing.

Now replace /magisk/dolbyatmos/post-fs-data.sh with this: https://pastebin.com/06xSq8VB

Reboot, and kill the audioserver as sonopholis suggested:

su -c killall audioserver

Play a song, and mess with the equalizer in atmos to see if you get an effect. Please note the UI is still very buggy, as in when you press the power or reset buttons in app, you'll have to hit back to close the window and reopen it to see the change.

---------- Post added at 06:13 PM ---------- Previous post was at 05:17 PM ----------

There's a random empty etc/audio folder, but my audio_policy_configuration.xml is in vendor/etc

I had to remove the if statement from the post-fs-data since it was checking to see if there's an audio folder (which there is, but it's empty), and since it would find the audio folder, it would not check for the file outside of it