Mel Kaye's hack was actually performed on an RPC-4000, so the story goes.

When I was doing my CS degree and had plenty of time in my hands for that sort
of thing, I had a look at the manuals for the two systems and tried to follow Ed
Nather's story.

It looks like Ed Nather misremembered the story (in his defense it had been
several decades since he'd seen it) because the whole hack rested on the machine
having an index register bit "between the address and the operation code in the
instruction word", but the RPC-4000 had the index register bit ("index tag") at
the end of the instruction word [1] and nothing between the current instruction
and operand address, which is, I think, the natural reading of "operation code"
and "address" respectively. There's also nothing between the operand address and
the next instruction address. The RPC-4000 word looks like this:

[COMMAND][OPERAND ADDRESS][NEXT ADDRESS][X]
|0 4||5 11|12 17||18 24|25 30||31|
| |
TRACK | SECTOR TRACK|SECTOR

What Nather remembers could be an overflow of the NEXT ADDRES field, through the
index tag, and into the COMMAND field. It depends on whether the RPC-4000 handled
overflows by wrapping or saturation (I can't find that in the manual).

On the other hand, Nather's account states that the hack changed an instruction
to a jump instruction- and the RPC-4000 does not have a jump instruction,
because it doesn't need it: like Nather's story says, every instruction has its
own GOTO- in the NEXT ADDRESS field.

Here's some alternative ways the hack could have played out, from what I can
tell:

1. The hack used the "Branch control" facility.

The RPC-4000 had another facility, the "Branch control", an internal flip-flop
with only one bit that was turned on when an arithmetic overflow was detected.
Kaye could have used this instead of the index tag and Nather may have
misremembered it as being the index tag.

2. The hack was actually done on the LGP-30

The hack might also have been possible to pull off on the LGP-30, that had two
bits between the opcode and the operand address [2] and an uncontrolled jump
instruction ("Unconditional transfer" in the manual). Kaye could have kept those
two bits set and caused an overflow, as told in the story.

3. The hack was done on an RPC-4000 emulating an LGP-30.

Royal McBee had an emulator written for the RPC-4000, specifically to be able to
run LGP-30 programs on the new machine [3]:

My name is James William (Bill) Bryner ... In 1960 I was hired by Royal-McBee
to write the assembler for the replacement to the LGP-30, the RPC-4000.

Mel Kaye designed the RPC-4000 assembler. It was titled ROAR (Royal-McBee
Optimizing Assembler Routine). Edward W. Dubbs and I programmed that
assembler. Following that, I wrote an LGP-30 simulator to run on the
RPC-4000. This was meant to allow all programs written for the LGP-30 to be
executed on the RPC-4000 without further programming. A drum computer
simulating a drum computer is agonizingly slow!

I bet, the blackjack program that brought everyone to the Royal McBee booth
would have been top of the line of the programs to be run on the RPC-4000.
Perhaps, then, Nather was working on Mel Kaye's "port" of the original blackjack
program, not on the RPC-4000 but on the LGP-30 emulator running on the RPC-4000.
Judging from the descriptions of Mel Kaye's programs in Ed Nather's account,
just having an emulator for the architecure would not necessarily mean that
Kaye's programs would run without any changes on the RPC-4000.

I have no clear idea how any of the above could have worked. Just guessing.

_____________

[1] http://www.bitsavers.org/pdf/royalPrecision/RPC-4000/RPC-400...

[2] Here's an example from the LGP-30 manual, as in the article:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0
(-------) (----------)(------------)
order bits track bits sector bits
= bring = 20 = 00
( address = 2000 )

[3] http://ed-thelen.org/comp-hist/lgp-30.html#Historical%20Note...