JMRI LI-USB Bugs

Summarizes bugs identified in JMRI processing when **LI-USB** interface is used with **LZV100** and possibly **LZV200**. On **NanoX** and **DR5000**, the bugs are slightly different: **NanoX** fails to operate completely, DR5000 does not suffers from "turnout interference". More on pages dedicated to NanoX and DR5000.

There are tests that prove each of the identified errors happening on JMRI code by just timing the command and replies in "unfortunate", but possible way. Some of the errors may happen separately; the diagram specifically shows as much of them as possible in that few command-reply exchanges.

The new **XNetPlus** connection stack should address these issues.

## How to read the diagram

There are several **timelines** from top to bottom, one for each object participating:

* the transmit loop that sends commands **to** the layout
* the receive thread, which receives data **from** the layout
* two turnouts: XT5 and XT6
* some other object, a Throttle, for example

Each **command** is labeled by a number **and** a colored border. The command's processing is filled by a hatch using that color. The diagram shows when the command is **posted** and when it is actually processd by the transmit thread.

Internal object states are marked on the diagram: TrafficController state is written in between transmit loop and receive thread timelines.

Turnout internal state is noted right of the turnout's timeline.

## Nitpick #1: Incorrect pairing of Reply to Command

This is just a formal difference for LI-USB/Net interface usage.

Unlike Turnouts, which get notified through TurnoutManager about (unsolicited) feedbacks, typical one-command listeners get only replies to commands they send. In the depicted case, some (for example) Throttle may receive a completely unrelated reply (feedback, in this example).

But feedbakcs that JMRI fails to identify as unsolicited cause great issues on **NanoX**.

### Cause

The TrafficController's state changes immediately when a message is pulled out from the queue, but **before** the message is sent to the port. If the reply is **already** being received at that moment, it might be assigned to a wrong **mLastSender**.

Since **mLastSender** is never reset to **null**, it retains the last-command state - for the duration of **beforePollTimeout.** If a reply comes during that time, it will be also **mis-identified** as a response to the last sender

The situation cannot be avoided because the time sync between actual buffer flush to the LI\* device, and message reception is affected by OS, network and LI\* operation itself, all out of JMRI control.

### Solution

Indvidual commands, or listeners should have a way how to **reject** a message mistakenly assigned to the listener, even such simple as a return value indicating "unexpected reply". The core could then treat the message as unsolicited.

## Error #2: Premature OFF generation

Turnout #5 generates OFF message **before its command was even sent**. The OFF message for Turnout 5 is generated in the green-hatched part at the top, while the command to layout itself **appears later than OFF (**is purple-hatched).

This bug may lead to command station and decoder **not reacting to the command**, since the delay between **ON**  and **OFF** messages may be arbitrarily short (as OFF timer starts to tick before the command) - not the 30ms coded in XNetTurnout.

### Cause

It's caused by incorrect state transition model in XNetTurnout. It transitions to *COMMANDSENT* as soon as it *hands over* the ON command to the TrafficController. But that only means the ON command enters the queue in TrafficController. Pre-existing queue items must be sent first, and ANY feedback delivered while the command just waits in the queue will reach XNetTurnout "primed" in *COMMANDSENT* state.

### Solution

The Turnout should transition to *COMMANDSENT* only when the message is really sent. Ideally **only** when the command was sent **AND** a valid reply, not a retransmittable or permanent error, was received. Only then it is valid to "prime" the Turnout to send OFF messages to the layout. The Turnout's state machine's mode is inconsistent here.

## Error #3: Duplicate OFF is sent

One OFF message is already scheduled, and other is sent. This is a behaviour **different** than the one observed with Roco and Lenz handhelds: they always send just a single message.

Other coding errors cause OFF not to be just send after more incoming messages, but also twice for each single incoming message. The number of generated OFFs vary from 1 to 4, depending on random conditions, timing and exact command station used.

It was explained duplicate packets are sent, because:

> ... we have seen too many cases where the command station didn’t respond as expected on the rails.
>
>

But with no exact details to verify. Given the amount of inconsistencies and long-standing design and implementation bugs in the code, I simply **do not trust** that the analysis was done correctly - with effects of various inter-thread synchronization bugs (still present in the code), timing issues etc eliminated. It's a behaviour different from original Lenz hardware, and most probably is here to "solve" other bugs, such as Error #8 below.

### Solution

Send exactly one OFF for each ON message, but reliably.

## Error #4: OFF is sent without delay right after ON

It was explained that **OFF** command must be sent only after a certain delay after **ON** command: the command station needs some time to react, and to repeat (3 or more times) the instruction to DCC bus. The repetition is critical so that decoders get the instruction (more) reliably. If OFF is sent quickly after ON, the command station may not repeat the DCC packet, and the decoder may happen to miss the single one.

In Error#3 we can see that OFF is queued **immediately** after **ON** command is confirmed, as the turnout is already in *OFFSENT* state.

### Cause

This is caused by a premature transition to *OFFSENT* state. The first OFF message was **scheduled**, it was not sent and even the TrafficController knows nothing about it (it's just planned in a Timer listener); so there's no indication the OFF message is still "waiting somewhere" when the 2nd reply is processed. The state machine model is inconsistent here. The code **was broken from its inception**.

### Solution

OFF messages, if duplicated at all, should queue one after another; if the prior OFF message is delayed, all others should be delayed as well.

## Error #5: Normal-priority message sent before the High-priority ones

We see that command #5 (XT5-CLOSE) was generated quite a while after command #4 (XT5-OFF) entered the high-priority message queue in XNetTrafficController. Still the **normal** message is processed before **high priority** one. This way, OFF messages may be arbitrarily delayed, if there are more turnout, throttle or other normal messages to send.

**Extended delay before OFF is sent may physically damage solenoids on the layout.**

### Cause

The bug is caused by invalid implementation of the **high-priority** queue, which uses **low-priority** poll feature. Poll is only done when the outbound queue in TrafficController is **completely empty** and yet delayed by 100ms (by default). As far as I checked GIT history, the code has neem **never** correct, it was broken from the start - unnoticed.

### Solution

Once high-priority message enters the queue, it must be processed without waiting 100ms and even if there are other **normal** messages in the queue.

## Error #6: Invalid timing of OFF message

OFF message is explicitly delayed **by 30ms**. But we see it is delayed for **at least 100ms**, despite the explicit timing in the code.

### Cause

As with Error #5, this is caused by incorrect way how **high priority** messages are delivered to TrafficController's transmit loop. Not only the **normal** queue must be exhausted, but also a "look-ahead" 100ms timeout must be elapsed before first OFF can be sent.

### Solution

Handle high-priority messages similar to normal ones: if one is available, no lookahead delay must occur.

## Error #7: Unnecessary delays between OFF messages

It's a variation of Error #6: not only the before-poll delay is applied after last normal message is handled, and before "high priority" is polled for, but also between two subsequent "high priority" messages. In combination with Error #3 (message duplication), OFFs may be delayed by hundreds of milliseconds from their originally supposed schedule.

### Cause

Full transmitLoop() is one after each high-priority message, including the before-poll delay, even though the high-priority queue is not empty yet.

## Error #8: No re-send of OFF message

If the command station **happens** to reply **BUSY** to the OFF message, the message is never repeated. If the station replies BUSY several times, **all OFF messages may be discarded silently**. Note that the OFF message was removed from the queue, so **next message** may be processed as if no error happened.

**OFF messages may be completely dropped causing continuous current to solenoids in layout.**

### Cause

Processing of "high priority" OFF messages is different code than the one handling normal messages. The duplicated "high priority" part does not handle recovery from BUSY condition.

### Solution

Unify processing to get reliable error handling.

## Error #9: Does not wait for proper OK after OFF

During Feedback processing (purple phase on the diagram), the Turnout is, or will transition into state *OFFSENT.* When the OK reply for *message #2 "XT6-THROWN"* (**not OFF** message) comes, it will be misinterpreted as OK-after-OFF and the Turnout will transition to *IDLE* state.

But at that time, OFF message was not even sent ! Still, the Turnout transitioned to IDLE, and could eventually send next command to the layout.

Note: **Lenz LZV100 with LI-USB** or LI-USB-Ethernet **always** sends 2 replies for 1 Turnout command (except the case when Command Station believes the turnout **is** already in the commanded position).

### Cause

The design does not seem to anticipate multiple responses per command. If replies come in the order

* feedback
* OK

the implementation transitions on each reply from *COMMANDSENT* to *OFFSENT* to *IDLE* even though the OFF command was not yet transmitted. Transition to *OFFSENT* is not controlled by actual send, but just when the outgoing message is *queued*.

### Solution

Transition to *OFFSENT* must be done only when OFF was really sent.