Repository: neovim
Issue: Lua: interrupt / cancel Lua code
Labels: enhancement,needs:design,input,lua

There is a known issue both with Vim and Neovim: if you write an endless loop inside `:lua` then it can’t be interrupted. The lua itself handles the problem by simply running the following

```C
static void lstop (lua_State *L, lua_Debug *ar) {
  (void)ar;  /* unused arg. */
  lua_sethook(L, NULL, 0, 0);
  luaL_error(L, "interrupted!");
}


static void laction (int i) {
  signal(i, SIG_DFL); /* if another SIGINT happens before lstop,
                              terminate process (default action) */
  lua_sethook(globalL, lstop, LUA_MASKCALL | LUA_MASKRET | LUA_MASKCOUNT, 1);
}
```


at SIGINT handler: apparently lua authors are feeling lucky because this code is not really reentrant (handler should not be called in the middle of `signal()` call). We can do something similar:

```
/// Hook used to interrupt current lua code
///
/// @param[in]  lstate  Lua state.
/// @param[in]  ar  Debugging information, unused.
static void nlua_interrupt_hook(lua_State *const lstate, lua_Debug *const ar)
  FUNC_ATTR_NONNULL_ALL
{
  lua_sethook(lstate, NULL, 0, 0);
  luaL_error(lstate, "interrupted!");
}

/// Currently running lua interpreter state
///
/// NULL if lua is not running or if it was already interrupted.
volatile lua_State *volatile running_lstate = NULL;

/// Interrupt currently running lua interpreter
///
/// To be used from interrupt handler. Code derived from lua signal handlers.
void nlua_interrupt(void)
{
  lua_State *const lstate = (lua_State *)running_lstate;
  running_lstate = NULL;
  if (lstate != NULL) {
    lua_sethook(lstate, nlua_interrupt_hook,
                LUA_MASKCALL | LUA_MASKRET | LUA_MASKCOUNT, 1);
  }
}

/// Like lua_pcall, but set running_lstate before entering and reset it at exit
///
/// Does not have errfunc argument because it is not used.
///
/// @param  lstate  Lua interpreter state.
/// @param[in]  nargs  Number of arguments to the called function.
/// @param[in]  nresults  Number of expected results.
///
/// @return true if errored out, false otherwise.
bool nlua_pcall(lua_State *const lstate, const int nargs, const int nresults)
  FUNC_ATTR_NONNULL_ALL
{
  lua_State *const saved_lstate = running_lstate;
  assert(saved_lstate == running_lstate || saved_lstate == NULL);
  running_lstate = lstate;
  const bool ret = (bool)lua_pcall(lstate, nargs, nresults, 0);
  running_lstate = saved_lstate;
  return ret;
}
```

and call `nlua_interrupt` in `process_interrupts`, but this is not only not reentrant this is also not thread-safe (which lua authors can’t be bothered with because their handler is for single-threaded applications). Since we are apparently not handling actual SIGINT thread safety may *partially* be protected by mutexes I think.

The question: would it be fine to work with this like the following:

1. Replace all `lua_pcall` calls with `nlua_pcall`.
2. Add `nlua_interrupt` call.
3. Add a mutex to `nlua_interrupt` and `nlua_pcall`. Note: when I said “partially” I meant that while input thread is calling `nlua_interrupt()` main thread may call `debug.sethook` (this is why pointer semantically has two `volatile` there: volatile pointer to volatile memory) and I can protect only `running_lstate` (volatile pointer, not volatile memory) manipulations.

or do somebody have other ideas? Currently I may only suggest additionally mocking `debug.sethook` to use the mutex as well, but `lua_sethook` calls can’t be protected like this. AFAIR both calls should be rare though.

Note that `lua_sethook` is as simple as

```C
#define resethookcount(L)    (L->hookcount = L->basehookcount)
LUA_API int lua_sethook (lua_State *L, lua_Hook func, int mask, int count) {
  if (func == NULL || mask == 0) {  /* turn off hooks? */
    mask = 0;
    func = NULL;
  }
  L->hook = func;
  L->basehookcount = count;
  resethookcount(L);
  L->hookmask = cast_byte(mask);
  return 1;
}
```

, though I have not checked luajit sources yet.