[Nexus 5 Q&A, Help & Troubleshooting] TJCacher:
Ok, I've done a lot more testing with this issue, and have come up with the following working theory. If any of you notice behavior inconsistent with this theory, post it up here. Maybe we can pin the exact cause down enough to let Google know the exact parameters of this glitch.

Here's my theory:

I'm going to call the situation where the Google search mic icon is hollow and the search won't respond to "Ok Google" the HM (hollow mic) mode, and the normal, full mic icon, indicating the search will properly respond to "Ok Google" the FM (full mic) mode. (HM=hollow mic is not listening, FM=full mic is listening)

The problem seems to be that the control is designed to enter HM mode whenever the system is aware that the device is playing sound and to return to FM mode when the system detects that sound has been paused or stopped, but that sometimes the system gets "stuck" in HM mode and remains there even when sound has been paused or stopped. Furthermore, in some cases HM mode is not activated when sound is started,

My theory is that the HM/FM switching only occurs correctly when the sound app itself is used to *directly* start and stop sound. If you use an alternate method to start and/or stop sound (like a linked set of media controls in the notification shade, or a set of media controls in a linked home-screen widget), then there is a near certainty that either HM mode will not be activated when sound starts and/or FM mode will not be reset when sound stops.

When the system becomes "stuck" in HM mode, even when no sound is playing, I have found two ways to reliably get it to reset to FM mode:

Method 1:

1) Stop any sound being played by the device via any means (directly from a sound app, or indirectly from a set of media controls in the notification shade or a home-screen widget)

2) Tap the word "Google" in the Google search bar at the top of any home screen

3) Once inside the search screen, cancel it by swiping right-to-left once or twice to return to the home screen.

Method 2

1) Go into the sound app which either is or was being used to play sound

2) Start the app playing sound from directly in the app itself

3) Stop or pause the sound from directly in the app itself

On my device, using Google Play as the sound app, I have had 100% success, with both of these methods, at getting HM mode "unstuck" and getting the device back into FM mode.

I can also reliably produce the "stuck" HM mode by starting music from within a music app, but stopping or pausing it from outside the app by using a widget.

On my device, if starting from FM mode, starting music playing from a widget instead of from within a music app will result in the device never entering HM mode in the first place.

If others can verify this, I think we may be able to say with some confidence that the problem is that the method being used to detect when sound starts and stops is not properly looking for sound to be started or stopped using an externally-linked set of media controls (as from a notification shade or widget), and that it only reliably works when sound is started or stopped *directly from inside the app itself, using app's built-in media controls*.

Thanks for any additional data anyone is willing to provide to prove or disprove this theory.