In these examples let's suppose function (or object) A passes the callback function C to the function (or object) B. Function B is what calls or indirectly causes to be called the callback C.

Scenario 1: Object B has a function body which looks like:

1. Look at or modify B's state.

2. Call callback C.

3. With code that assumes the state hasn't changed, do more stuff to B.

If C calls back into B (for example, if we'd like to perform the operation a second time, or if we'd like to perform a different operation), the assumption on line 3 might be incorrect. Or, some other assumption in the reentrant call might be incorrect.

(One common outcome is that the developers say, "Oh, there's a race condition," and they add a mutex. Later, they say, "Oh, there's a deadlock," and make it a recursive mutex. This is why recursive mutexes are a major code smell.)

Scenario 2: The developer realizes a safe callback has to be called after all other side effects are complete. So the developer writes this for their function body:

1. Look at or modify B's state.

2. Decide you're going to call C later.

3. Do more stuff to B.

4. Call callback C, literally the last thing you do in the function.

(The developer is an experienced C++ developer and knows to make sure object destructors for variables going out of scope happen before step 4.)

The potential problem here is on A's and C's side. What if they write:

A-1. Acquire mutex M, and look at or modify A's state.

A-2. Call into B, with callback C (which affects the state of A).

A-3. With code that assumes the state hasn't changed, do more stuff to A.

A-4. Release mutex M.

In callback C (in some helper function it invokes):

C-1. Acquire mutex M.

C-2. Do stuff assuming we have exclusive access to the state of A.

C-3. Release mutex M.

If M is non-recursive, you have a deadlock. If M is a recursive mutex (or nonexistant) the assumption on line A-3 is incorrect. Or maybe the assumption on line C-2 is incorrect.

Scenario 3: The developer is now super-paranoid about side effects of their callback. So they write all their callbacks like this:

E-1: callAsynchronously(function() {
C-1: Acquire mutex M.
C-2, C-3: as before.
}).

Well now you have no performance advantage relative to just having object B call asynchronously.

Oh, and one more.

Scenario 4: The developer's a freakin' genius. Writes out dead-perfect hypersecure code. Can run a 5-minute mile, and smells nice afterwards. Then enough synchronous callbacks happen recursively to produce a stack overflow.

----

The thing about these scenarios is that if you call the callback function synchronously every time, you're more likely to catch the bug. It'll show up in tests, or you'll just get a deadlock, or some debug-mode "DebugMutex" assertion failure.

Also, if F always invokes asynchronously, and G always invokes synchronously, you can't call F from G. And you have to add a wrapper in your callback when calling G from F.

If, for a particular function, you have inconsistent behavior, you'll have a whole bunch of undertested edge cases. And they might involve any function up the callstack -- If F1 calls F2 calls F3 calls F4, and once in a while F4 invokes synchronously, that could cause F3, F2, and F1 to have that same property (if they don't do further async operations in the callbacks they pass up).