[Nexus 5 Q&A, Help & Troubleshooting] SuicideFlasher:
Just in case it could be useful to know, I'm running this, specifically the StockOnlyDebloated one (M4B30X as of writing), with SuperSU 2.78 systemless root.

The problem I'm having is that encryption refuses to ever start if I try to activate it. Everything shuts down, the green android thing is displayed, then the screen goes blank and it stays that way. My logcat in the moments preceding the supposed beginning of the encryption process reads as follows:

11-11 23:51:20.435 989 2940 D CryptdConnector: SND -> {7 cryptfs enablecrypto inplace password [scrubbed]}
11-11 23:51:20.442 149 159 D Cryptfs : Just asked init to shut down class main
11-11 23:51:20.442 989 1289 D VoldConnector: RCV <- {651 emulated 5}
11-11 23:51:20.445 289 289 I ServiceManager: service 'android.service.gatekeeper.IGateKeeperService' died
11-11 23:51:20.447 989 1288 D MountService: Volume emulated broadcasting ejecting to UserHandle{0}
11-11 23:51:20.448 1402 1402 D MtpService: onStorageStateChanged /storage/emulated/0 mounted -> ejecting
11-11 23:51:20.448 1402 1402 E MtpService: Missing MtpStorage for /storage/emulated/0
11-11 23:51:20.456 989 1301 E NetdConnector: got -1 reading with start = 0
11-11 23:51:20.456 989 1301 E NetdConnector: closing stream for netd
11-11 23:51:20.456 989 1301 E NetdConnector: Communications error: java.io.IOException: Connection refused
11-11 23:51:20.457 989 1301 E NetdConnector: Error in NativeDaemonConnector: java.io.IOException: Connection refused
11-11 23:51:20.457 989 1320 E mDnsConnector: got -1 reading with start = 0
11-11 23:51:20.457 989 1320 E mDnsConnector: closing stream for mdns
11-11 23:51:20.458 989 1320 E mDnsConnector: Communications error: java.io.IOException: Connection refused
11-11 23:51:20.458 989 1320 E mDnsConnector: Error in NativeDaemonConnector: java.io.IOException: Connection refused
11-11 23:51:20.556 159 159 W vold : type=1400 audit(0.0:794): avc: denied { read } for name="fd" dev="proc" ino=7289 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=dir permissive=0
11-11 23:51:20.556 159 159 W vold : type=1400 audit(0.0:795): avc: denied { getattr } for path="/proc/148/cwd" dev="proc" ino=59003 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:20.556 159 159 W vold : type=1400 audit(0.0:796): avc: denied { getattr } for path="/proc/148/root" dev="proc" ino=59004 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:20.566 159 159 W vold : type=1400 audit(0.0:797): avc: denied { getattr } for path="/proc/148/exe" dev="proc" ino=59005 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:20.460 989 1035 I ActivityManager: Start proc 7171:com.android.externalstorage/u0a8 for broadcast com.android.externalstorage/.MountReceiver
11-11 23:51:20.668 289 289 I ServiceManager: service 'user.xposed.app' died
11-11 23:51:20.668 289 289 I ServiceManager: service 'user.xposed.system' died
11-11 23:51:20.669 289 289 I ServiceManager: service 'android.security.keystore' died
11-11 23:51:20.711 289 289 I ServiceManager: service 'nfc' died
11-11 23:51:20.741 288 288 I lowmemorykiller: ActivityManager disconnected
11-11 23:51:20.741 288 288 I lowmemorykiller: Closing Activity Manager data connection
11-11 23:51:20.860 149 158 E FrameworkListener: read() failed (Connection reset by peer)
11-11 23:51:20.931 289 289 I ServiceManager: service 'telecom' died
11-11 23:51:20.931 289 289 I ServiceManager: service 'user' died
11-11 23:51:20.931 289 289 I ServiceManager: service 'procstats' died
11-11 23:51:20.931 289 289 I ServiceManager: service 'meminfo' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'gfxinfo' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'dbinfo' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'cpuinfo' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'permission' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'processinfo' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'sensorservice' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'battery' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'usagestats' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'webviewupdate' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'scheduling_policy' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'telephony.registry' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'media.camera.proxy' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'account' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'content' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'vibrator' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'consumer_ir' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'alarm' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'window' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'input' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'batterystats' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'appops' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'power' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'display' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'bluetooth_manager' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'input_method' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'accessibility' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'mount' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'uimode' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'lock_settings' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'deviceidle' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'device_policy' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'statusbar' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'clipboard' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'network_management' died
11-11 23:51:20.932 289 289 I ServiceManager: service 'textservices' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'network_score' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'netstats' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'netpolicy' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'package' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'activity' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'wifip2p' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'wifi' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'wifiscanner' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'rttmanager' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'ethernet' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'connectivity' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'servicediscovery' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'updatelock' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'notification' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'devicestoragemonitor' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'location' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'country_detector' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'search' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'dropbox' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'wallpaper' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'audio' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'DockObserver' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'midi' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'usb' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'serial' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'jobscheduler' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'backup' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'appwidget' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'voiceinteraction' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'diskstats' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'samplingprofiler' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'commontime_management' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'dreams' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'assetatlas' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'graphicsstats' died
11-11 23:51:20.933 289 289 I ServiceManager: service 'print' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'restrictions' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'media_session' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'media_router' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'fingerprint' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'trust' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'launcherapps' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'media_projection' died
11-11 23:51:20.934 289 289 I ServiceManager: service 'imms' died
11-11 23:51:20.935 290 290 D SurfaceFlinger: Set power mode=2, type=0 flinger=0xb6b24000
11-11 23:51:20.935 290 290 D SurfaceFlinger: Screen type=0 is already mode=2
11-11 23:51:20.941 289 289 I ServiceManager: service 'simphonebook' died
11-11 23:51:20.941 289 289 I ServiceManager: service 'iphonesubinfo' died
11-11 23:51:20.941 289 289 I ServiceManager: service 'isms' died
11-11 23:51:20.941 289 289 I ServiceManager: service 'phone' died
11-11 23:51:20.942 289 289 I ServiceManager: service 'carrier_config' died
11-11 23:51:20.942 289 289 I ServiceManager: service 'sip' died
11-11 23:51:20.942 289 289 I ServiceManager: service 'isub' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.sound_trigger_hw' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.radio' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.audio_flinger' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.player' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.resource_manager' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.camera' died
11-11 23:51:21.004 289 289 I ServiceManager: service 'media.audio_policy' died
11-11 23:51:21.005 289 289 I ServiceManager: service 'drm.drmManager' died
11-11 23:51:21.051 7186 7186 I BootAnimation: boot animation disabled
11-11 23:51:40.717 159 159 W vold : type=1400 audit(0.0:798): avc: denied { read } for name="fd" dev="proc" ino=7289 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=dir permissive=0
11-11 23:51:40.717 159 159 W vold : type=1400 audit(0.0:799): avc: denied { getattr } for path="/proc/148/cwd" dev="proc" ino=59003 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.717 159 159 W vold : type=1400 audit(0.0:800): avc: denied { getattr } for path="/proc/148/root" dev="proc" ino=59004 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.717 159 159 W vold : type=1400 audit(0.0:801): avc: denied { getattr } for path="/proc/148/exe" dev="proc" ino=59005 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.747 149 159 E ProcessKiller: Process daemonsu:0:650 (3619) has open file /data/data/eu.chainfire.supersu/logs/L14789021760000.GRANTED.ADB shell
11-11 23:51:40.750 149 159 E Cryptfs : unmounting /data failed: Device or resource busy
11-11 23:51:40.757 159 159 W vold : type=1400 audit(0.0:802): avc: denied { read } for name="fd" dev="proc" ino=7289 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=dir permissive=0
11-11 23:51:40.757 159 159 W vold : type=1400 audit(0.0:803): avc: denied { getattr } for path="/proc/148/cwd" dev="proc" ino=59003 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.757 159 159 W vold : type=1400 audit(0.0:804): avc: denied { getattr } for path="/proc/148/root" dev="proc" ino=59004 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.757 159 159 W vold : type=1400 audit(0.0:805): avc: denied { getattr } for path="/proc/148/exe" dev="proc" ino=59005 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:51:40.783 149 159 E ProcessKiller: Process daemonsu:0:650 (3619) has open file /data/data/eu.chainfire.supersu/logs/L14789021760000.GRANTED.ADB shell
11-11 23:51:40.783 149 159 W ProcessKiller: Sending Killed to process 3619
11-11 23:51:40.864 149 159 D Cryptfs : Just asked init to shut down class main
11-11 23:51:40.864 149 159 W vold : emulated unmount requires state mounted
11-11 23:52:00.876 159 159 W vold : type=1400 audit(0.0:806): avc: denied { read } for name="fd" dev="proc" ino=7289 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=dir permissive=0
11-11 23:52:00.876 159 159 W vold : type=1400 audit(0.0:807): avc: denied { getattr } for path="/proc/148/cwd" dev="proc" ino=59003 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:52:00.876 159 159 W vold : type=1400 audit(0.0:808): avc: denied { getattr } for path="/proc/148/root" dev="proc" ino=59004 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:52:00.876 159 159 W vold : type=1400 audit(0.0:809): avc: denied { getattr } for path="/proc/148/exe" dev="proc" ino=59005 scontext=u:r:vold:s0 tcontext=u:r:logd:s0 tclass=lnk_file permissive=0
11-11 23:52:00.906 149 159 E Cryptfs : unmounting /data failed: Device or resource busy
11-11 23:52:00.906 149 159 W SocketClient: write error (Broken pipe)
11-11 23:52:00.906 149 159 W SocketClient: Unable to send msg '200 7 -1'

As you can see, the encryption begins, everything shuts down nicely, then /data fails to dismount because something in it is still in use and everything goes dead silent (it's been more than half an hour and logcat still gives no signs of activity). If I force a reboot, everything works fine and dandy as if the encryption never occured, because it didn't.

I've done the searching, and the closest thing to this issue I've found is this thread, but there's no solution there, only the exact same problem.

I strongly suspect that systemless root is the cause of this, because after the failed /data dismount attempt, /su shows up as still mounted, /su is the mount point of /data/su.img, and also gives a "device busy" error if I try to dismount it manually, probably because of the running sukernel and daemonsu processes. I could kill those, but something about them (like the fact that there's one named daemonsu:mount:master) makes me feel like that's not a good idea at all, so I haven't tried yet. Disabling root in SuperSU changes nothing.

Any clues on how to fix or at least work around this?