textqueue 1.1
A text-based queue tool

TextQueue
================================================================================
TextQueue is a simple human readable file-based message queue.

The five core principals of TextQueue are
* To queue text based messages
* To be easily human readable
* To be easily human editable at rest
* To function directly on files without a daemon service
* To be forwards and backwards compatible with all versions of TextQueue

TextQueue Example
--------------------------------------------------------------------------------
```
-A queued message begins with a '-' dash control character
=A processed message begins with an '=' equals sign control character
!A message that encountered a critical error begins with an `!` exclamation mark control character
-Any message can have multiple lines
 where each sequential line begins with a ' ' space
=Any message can have multiple lines
 where each sequential line begins with a ' ' space
!Any message can have multiple lines
 where each sequential line begins with a ' ' space
# A comment begins with a '#' sign
# Multiline comments are possible using a space as the starting character on
  each sequential line.
# Messages are space sensitive, so the following message will read "  I have space padding  "
-  I have space padding  

-Blank lines are ingored
=Processed messages can apear after queued messages
-Message Variables
\Message variables are attached after a message
\A message can have any number of variables
\Message variables are fixed length and editable
\Additional space can be reserved for the message variable with trailing spaces   
\A single message variable cannot span multiple lines
# Comment Variables
\Comments can also have message variables
```

* `#` A comment these lines will be completely ignored
* `-` A message that has not yet been processed
* `=` A message that has been fully processed
* `!` A message that was processed but encountered a critical unrecoverable error
* ` ` A continuation of the previous message or comment from the previous line
* `\` A variable for the previous message which can be edited

Each line must match something like the regex
```
[-=# !\\][^\n]*(?:\n|<EOF>)|\n
```

Features
================================================================================

Comments
--------------------------------------------------------------------------------
Comments are a staple for any human readable file. They do not serve a purpose
for the machine reading or writing data, but they do allow for information to
be communicated to or by humans. It is possible that a user may use comments as
a way to extend the TextQueue syntax, but the syntax is already intended to be
extended with other control characters so this issue is not enough to offset
the benefits that comments bring to the spec.


Queued Messages
--------------------------------------------------------------------------------
Queued messages are messages that have not been processed. By default TextQueue
will only return these messages. A queued message should be marked complete
once it is done being processed. 


Completed Messages
--------------------------------------------------------------------------------
Completed messages are messages that have been marked as completed and do not
need to be processed further. By default completed messages are not returned
by TextQueue.


Critical Error Messages
--------------------------------------------------------------------------------
A message that was not completed but also should not be returned by TextQueue
again because there is a critical non-recoverable error with the message. By
default critical error messages are not returned by TextQueue.


Variables
--------------------------------------------------------------------------------
Variables unlock a large amount of utility by reserving space in the file for data to be saved per message.

Variables are limited to a single line and cannot be extended with the ` ` space control character to cross multiple lines.

Some examples of things variables can be used for are:
- Timestamps
- Error Codes
- Locking Mutexes
- Actor audit history
- Retry counts
- Multi-step state


Parallelism
================================================================================
TextQueue is not intended to have more than one reader and one writer at a time.

Some concurrent access is possible. Specifically a concurrent read and write.
The current schema loses some functionality with multiple concurrent reads.
Multiple concurrent writers would require a mutex lock using a separate mutex
file. If the user is so concurrent they need message-level concurrent writes,
then they are probably better off using a daemon based message passing solution
instead. A per-message mutex file could also be possible to allow for full
once-and-only-once MQTTT QOL2 style messages by using the message offsets in
the mutex. Adding a mutex file would not change or alter the current schema.

| Readers | Writers | Stability                                                             |
|---------|---------|-----------------------------------------------------------------------|
| 0       | 0       | Stable - At Rest                                                      |
| 1       | 0       | Stable - Only Reading                                                 |
| 0       | 1       | Stable - Only Writing                                                 |
| 1       | 1       | Stable when writes are appends and reads are not reading the final message|
| 2+      | 0+      | Unstable - Requires per-message mutex variable locks                  |
| 0+      | 2+      | Unstable - Requires file locking to prevent garbled append writes     |



Multiple User Mitigation - File Splitting
--------------------------------------------------------------------------------


Multiple User Mitigation - File Mutex
--------------------------------------------------------------------------------


Multiple User Mitigation - Message Mutex
--------------------------------------------------------------------------------
In order for a message to have a mutex a [variable] must be given to that
message. This means that in order to properly use this solution, every message
must have a variable used just for the mutex.

```tq
-Hello World
\mutex=      
-How are you today
\MUTEX
-Goodbye World
\MUTEX
```


Multiple Writer Mitigation - File Splitting
--------------------------------------------------------------------------------
If you want multiple writers to be able to write jobs in parallel then the
writers must be using different files to write to, or the write operations must
somehow be serialized to prevent overwriting each other.


Multiple Writer Mitigation - Mutex
--------------------------------------------------------------------------------
You can create a mutex for the entire file to prevent more than one program
from writing to it at once. This will require changes to the library to create,
release, and respect the mutex.


Multiple Reader Mitigation

Multiple Reader Mitigation - Mutex
--------------------------------------------------------------------------------
Mutex file using the message offset to block individual messages from being
read twice. This will require changes to the reader so that it can re-check
released mutexes to see if the mutex was released because the process crashed,
errored, or timed out or because it finished processing.

A mutex file just using the queue filename to prevent multiple people from
reading at once is less good becuase you will need to keep the file locked
until the processing is done.


Other Features and Discussion
================================================================================

Message Errors
--------------------------------------------------------------------------------
TextQueue supports labeling a message as having encountered a "critical
unrecoverable error" using the `!` critical error control character. Additional
error information should be handled by the user, either writing the error info
to a log, or storing some part of the error information in a preallocated
message variable.


Inserted Timestamps
--------------------------------------------------------------------------------
Inserted timestamps seem quite useful in some use cases. Some use cases do not
have, want, or need timestamps, meaning that timestamps should not be required.
The entire body of the message is an arbitrary utf8 string. With the only
restriction being the `\n` newline character, those require a following ` `
space to be included in the message. Any user who wanted timestamps could
easily create their own sub-schema for their own messages that included
timestamps. If or when we ever find a compelling reason to add timestamps as
first-class data, we can add a new control character for it.


Prefix Syntax [Redundant]
--------------------------------------------------------------------------------
Similarly to timestamps, we might want other forms of fixed metadata. We could
easily add this in the future by creating new control sequences that apply
their content to the following message, for example, an ISO 8601 prefix or a
unix epoch timestamp prefix.

```
T2024-06-13T19:26:15+00:00-Some message with an iso 8601 timestamp as metadata
T2024-06-13T19:26:15+00:00
-Also some message with an iso 8601 timestamp as metadata
U1718306775=Some completed message with a unix epoch timestamp
U1718306775
=Also some completed message with a unix epoch timestamp
```

Prefixes might stress backwards compatability though, parsing a newer queue
on an older parser would still work. However the prefix metadata would not
be associated with the message inside the parser and instead would be ignored.

To avoid this you can trigger an error when reading messages that have control
characters you do not recognize, specificlly breaking backwards compatability.
We may be able to do something to prevent this from causing an error, instead
making the entire line one line and having the entire message ignored, not
allowing prefixes to exist on their own line the way that ` ` space suffexes do.
Or by making a new control character to control all prefixes, such as `>`.
Getting that into the spec in the earliest version.

```
T2024-06-13T19:26:15+00:00-Some message with an iso 8601 timestamp as metadata
>T2024-06-13T19:26:15+00:00
-Also some message with an iso 8601 timestamp as metadata
```
Or requiring that prefixes on their own line have to be followed by a special
control sequence such as `>` before the next line.
```
T2024-06-13T19:26:15+00:00
>-Also some message with an iso 8601 timestamp as metadata
```

The same-line-only option seems pretty good honestly compared to the others


The control characters `T`, `U`, and `>` are not finalized and merits discussion.

> [!NOTE]
> This feature was implemented as Message Variables instead.

Fixed Metadata [Redundant] -> Tags
--------------------------------------------------------------------------------
Timestamps *could* be considered a special usecase. But we could also expand on
this functionality to allow for arbitrary metadata tags on messages.

```
*tagname:tagvalue
*source:john
*timestamp:1995 July 5th at 5:30pm
*language:en-us
-My Message
 and another line of my message
```

We can add arbitrary metadata tags like this as well if we find a compelling
reason to add them.

The control character `*` or syntax is not finalized and merits discussion.

> [!NOTE]
> This feature was implemented as Message Variables instead.


Retries
--------------------------------------------------------------------------------
There is a valid usecase where the user wants to show that a job is worth
retrying, but only worth retrying so many times. Therefore marking it as
processed, error, or leaving it queued would be incorrect. In order to avoid
reserving a bunch of control characters for each quantity of retry, we can
instead reserve one, such as `!`, followed by any number of arabic numerals and
use the prefix syntax style above. Arabic numerals will always decrease in
length as the value they represent approaches zero. We can confidently decrease
this number in-place until it reaches 0 without worrying about shifting the
offset of any other characters. Every time the item is pulled from the queue,
we can automatically decrease the number of retries.

```
!12-Some Queued message with 12 retries remaining
!08-Some queued message with 8 retries remaining, that used to have 10 or more
! 8-An alternative message syntax with 8 retries remaining that used to have 10 or more
!5=A processed message that still had 5 retries remaining when it was fully processed
```

The control character `!` or syntax is not finalized and merits discussion.

> [!NOTE]
> This feature was implemented as Message Varaibles instead


Editable Metadata
--------------------------------------------------------------------------------
We could add some sort user editable in-queue data. Much like the fixed
metadata before we could add some sort of editable tag system. This would come
with some additional restrictions, though. Primarily, a tag could never
increase in size beyond its original maximum size. Examples of things we could
put in editable metadata would be statuses, processed timestamps, error codes,
or retry counts. This would require some more planning before implementation,
but just like all the previous features, we could easily add it using a new
control character.


> [!NOTE]
> This feature was implmented as Message Variables instead


File Locking
--------------------------------------------------------------------------------
Locking the file so that multiple people are not writing to it at the same time
is a little complex. Because the file format can allow for multiple people to
write a byte of the file at the same time, as well as allow for a single person
to append to the end of the file while others are reading it, we want to be sure
that a lock system can accurately restrict the file without encumbering the
parallelism afforded to the text queue format.


Other
-------------
TextQueue plays nicely with other text-based serialized systems. Is lightweight
on the disk, only updating single characters or appending lines, and is easily
human manipulable.

Due to the nature of TextQueue's syntax, we do not need to escape any control
characters inside the message because each control character will start in the
first column of the message. Any control characters that exist inside the
message will be treated as their regular utf8 characters
Any newlines inside the message are treated as the end of the message. But if
the control sequence of the next line is a ` ` space, then that newline and the
next line are added to previous message.
If any new control characters need to be added in the future they will always
start in column 0, even if they need to extend to further columns down the
line later.


TextQueue does not have any specification for a lockfile making it not fully
thread safe yet, though there are some threadsafe guarantees that arise from
the current methodology.
* Any message that is being marked as processed will never be marked as queued
* Multiple message can be marked as processed simultaniously.

However it does not protect against:
* once and only once - Two processes reading the same control caracter, picking
  up the same message, and both marking the message as handled
* Manual human editable tools accessing the file, causing full file rewrites
  that can alter the entire saved state.




FAQ
================================================================================
**Q: Why do textqueue queue files not have schema or version numbers?**  
**A:** We intentionally excluded versions, instead making sure the schema is
forwards and backwards compatible forever. We rely on the starting character of
each line to determine functionality as a “control character”. The schema today
only consumes a few of the characters. We leave all the other utf8 characters
open to extend the syntax with other future features. However, adding new
control characters to the schema should be done with intention. If not, we
could enter a situation where we have added too many features with short
control sequences, and newer features would require multi-character
control sequences.

**Q: How do I put an error message for an unprocessable message in the queue?**  
**A:** The processor program should handle error messages for any message that
fails, probably in one of their logs. Error messages do not belong in the queue
itself. If we ever changed our minds about this, then the error messages would
still need to be stored in an external log file anyway, or risk shifting the
index of later messages. A possible in-queue solution could be fixed-length
error codes using something similar to editable metadata.

**Q: Will you support other languages?**  
**A:** The plan is for TextQueue to have support for multiple languages. One
way this might happen is for TextQueue to become a library with c bindings that
any language can call into, similarly to how most implementations of sqlite work.

**Q: Why do you allow message variables on comments?**  
**A:** Comments support multi line continuations and message variables to make
it easier for users to comment out a message. If comments are allowed to have
multiple lines and are allowed to have variables then all a user needs to do to
comment out a message is change the initial control character to a `#`. A side
effect it also makes parsing slightly easier as the logic to parse comments is
the same as the logic to parse all other messages.