wagoner 1.1
A random word generator.
License: MIT
Keywords: random word generation

wagoner
=======

``wagoner`` is an anagram for rawogen, a random words generator.

``wagoner`` generates random words based on a given text. A random word is
generated incrementally, and the given text is used to know which character
should follow.


Usage
-----

``wagoner`` generates random words from tables extracted from texts. You need
first to extract such a table from a text. To extract a table, use
``wagoner.table``::

    python -m wagoner.table TEXT --output text.table

This command will extract the table corresponding to ``TEXT`` into the file
``text.table``.

Then, given the extracted table, ``wagoner.word`` can extract random words::

    python -m wagoner.word text.table

This command will generate ten words of ten characters, based on the
information in ``text.table``. To generate another number of words, use the
``--count`` option and to change their length, the ``--length`` option.

Random words can also be generated from trees. While the tables only tell
the generation which character should follow, trees also ensure that the
generated word will start like a regular word (from the text it comes from) and
will end like a regular (maybe different word). Trees are built from a
table or from a text::

    python -m wagoner.tree CONTENT --output text.tree

This command will extract a tree from ``CONTENT`` and save it into the file
``text.tree``.

Warning: trees can be very large and expensive to build; to control their
complexity, you can use the ``--prefix`` and ``--length`` options. See below
for more information.


How does it work
----------------

The main idea behind wagoner is to use a text to conduct the generation of
random words. The words in the text show what usually follows a given prefix,
and this information is used to choose the next character when building a word
incrementally.

For example, given the text composed of the two words ``ambiguous`` and
``gamblings``, and the prefix ``ig``, the first word tells us that the next
character can be ``u`` (because in ``ambiguous``, the sub-word ``ig`` is
followed by ``u``), and the second word tells us that the next character can
be ``a`` or ``s`` (because in ``gamblings``, the prefix ``g`` is followed by
``a`` or ``s``). The next character is chosen randomly among the possible next
characters and the word is incrementally generated following the same rule.

This next character is chosen randomly among the possible next characters, but
weights are used to more probably use frequent characters, and longer
prefixes. In other words, when choosing the next character for the prefix
``ig``, more weight is given to ``u`` than to ``a`` and ``s`` because ``u``
follows ``ig`` in ``ambiguous`` while ``a`` and ``s`` only follow ``g`` in
``gamblings``. Furthermore, if in the given text, a character follows more
frequently a given prefix than another, the first one will have a higher
chance to be chosen. This last characteristics of the generation can be turned
down by using the ``--flatten`` option of ``wagoner.word``. If this option is
active, two characters following the same prefix will have the same chance to
be chosen; nevertheless, a character following a shorter prefix will have less
chance to be chosen than a character following a longer prefix.

This information of the frequence of successing characters in the words of
the given text is stored in tables. Note that the ``--flatten`` option is
also present with ``wagoner.table``; this forces ``wagoner.table`` to
extract a table in which the number of occurrences of a character following a
given prefix is forgotten, giving the same result as the ``--flatten`` option
of ``wagoner.word`` at generation.

Finally, when incrementally generating a random word, the full prefix is used
to extract next character. In usual languages such as English, knowing the
full prefix is not necessary to generate a pronounceable word; instead, three
or four characters are usually sufficient to choose a next character that will
sound correctly. Restricting the generation of words based on a bounded prefix
instead of the full word can be achieved by using the ``--prefix`` option of
``wagoner.word``. Similarly, the same option can be used on ``wagoner.table``
to directly generate (smaller) tables that will not propose next characters
for longer prefixes.

Trees store more information than tables, but are also tailored to more
strict generation. Trees store the information about which character should
follow a given prefix, but only for words of a given length. They also store
information to generate words starting as words of the text and ending as them.
Because of this addition information, trees can be really complex, huge and
long to generate; they can potentially store each word of a given length
generable from the table. To tackle this complexity, trees can be limited to
looking at only a suffix of the current prefix by using the ``--prefix``
option; furthermore, by building trees for shorter words (using the
``--length`` option), these trees are smaller.


Example
-------

Let's take the list of English words available on
http://www-01.sil.org/linguistics/wordlists/english/. This list contains more
than 100000 English words and should be useful to generate random words that
sound English.

First thing to do before generating random words is to extract the table::

    python -m wagoner.table wordsEn.txt --output wordsEn.table

The table is extracted and saved in the ``wordsEn.table`` file. Then, we can
generate random words::

    python -m wagoner.word wordsEn.table

that outputs, for example::

    joggeriest
    lvinistica
    cleiadicat
    zenerousne
    ulencering
    mmanencien
    zeratenesi
    keynoteric
    encientnes
    crappinesa

You can see that some words are really like real words: ``crappinesa`` really
looks like ``crappiness``, a word of ``wordsEn.txt``. To avoid generating such
too real words, we can ask wagoner to only consider finite prefixes; this will
avoid to be trapped in the increasing proability of ressembling the word
``crappiness`` as the word is generated::

    python -m wagoner.word wordsEn.table --prefix=2

that outputs, for example::

    keyelittat
    retimcenve
    quedectrot
    fodcalitur
    xcedission
    queffliqui
    eshedlerad
    ficklapett
    quatersous
    sulationur

In this case, the words are still pronounceable, but are not words of the
``wordsEn.txt`` file.

If you want to generate words that start and end the same way words of the
text start and end, you can generate a tree from the table::

    python -m wagoner.tree wordsEn.table --output=wordsEn.tree --prefix=3

The tree can then be used to generate words::

    python -m wagoner.word wordsEn.tree

that outputs, for example::

    sinationso
    disjudging
    titualimal
    avespolybd
    prophology
    japackersc
    nonneappra
    overedefra
    oxideodore
    wordshedro


Library usage
-------------

``wagoner`` can also be used as a library. ``wagoner.table.Table`` represent
tables and a table can be extracted from a text file with::

    table = Table.from_words(wagoner.utils.extract_words(text_file))

``Table.from_words`` accept two optional arguments:

* ``prefix``: if greater than 0, the length of the prefixes to take into
  account when generating the word (default is 0).
* ``flatten``: if ``True``, the table is flattened and two successing
  characters for the same prefix will have the same weight (default is
  ``False``).

From such a table, a random word can be extracted::

    word = table.random_word(word_length)

where ``word_length`` is the length of the desired word. ``random_word``
accepts several optional arguments:

* ``prefix`` and ``flatten``, as above.
* ``start``: if ``True``, the generated word starts as a word of the text
  the table is extracted from (default is ``False``).
* ``end``: if ``True``, the generated word ends as a word of the text
  the table is extracted from (default is ``False``).
  Warning: this option should not be used because it is very time consuming.

Furthermore, ``wagoner.tree.Tree`` represent trees and a tree can be
extracted from a table with::

    tree = Tree.from_table(table, word_length)

where ``table`` is a table like the one above and ``word_length`` is the
length of the words the tree will produce. Like ``Table.from_words``,
``Tree.from_table`` supports two optional arguments: ``prefix`` and
``flatten``. In this case, the ``prefix`` argument is very important
because, if set to 0 (the default value), the tree can be really huge, even
impossible to stay in memory, and will need a lot of time to be built.

From such a te, a random word can be extracted::

    word = tree.random_word()

Unlike the tables, trees contain the complete information to extract words
(their length, whether flattening the weights) and ``random_word`` takes no
optional argument (technically, it accepts any arguments, but will ignore
them).


Changelog
---------

* 1.1: support for Python 2.7.