moonline 1.2.0
A Zipline-inspired backtesting implementation.
License: MIT
Requires Python: >=3.8,<3.10

# MoonLine
A [Zipline](https://github.com/quantopian/zipline)-inspired backtesting implementation. MoonLine utilizes the vectorized [Moonshot](https://www.quantrocket.com/docs#moonshot) interface but transforms it into an event-driven one, making it much easier to port Zipline algorithms to Moonshot.

## Disclaimer
MoonLine is in active (and early) development, so bugs are to be expected. If you have an issue that you suspect may be a bug (or a feature request), please log an issue in the [Issue Tracker](https://github.com/boosting-alpha-bv/moonline/issues).

## Development
This project uses [poetry](https://github.com/python-poetry/poetry) for development and release management.
```
$ git clone git@github.com:boosting-alpha-bv/moonline.git
$ cd moonline/
$ poetry install
```

If you do not want to use `poetry`, you can install the requirements manually via `pip` by looking them up in the [pyproject.toml](https://github.com/boosting-alpha-bv/moonline/blob/master/pyproject.toml#L11-L21) file.

### Running
To run MoonLine using `poetry`, use the following command:
```bash
$ poetry run python moonline -c <config_file>
```

Alternatively you can just run it with your regular Python installation. However, **beware of the fact that MoonLine is being developed on Python 3.7** and as such, lower version may not work (3.6 is set as the minimum required version). It is recommended to use [pyenv](https://github.com/pyenv/pyenv) to manage Python versions on your local machine.

### Generating Documentation
```bash
$ poetry run pdoc --config show_type_annotations=True --html --force moonline/
```

We are currently in the process of documenting the entire program, but for the moment several parts are undocumented.

### Working on Documentation
`pdoc` can be switched into hot-reload mode by appending the following:
```bash
$ poetry run pdoc --config show_type_annotations=True --html --force moonline/ --http :
```

This will open a webserver on [localhost:8080](http://localhost:8080) which will auto-reload whenever a change is made to docstrings in the observed modules.

## Creating A Configuration File
Since MoonLine has a lot of options, specifying them all via command line arguments quickly become tedious and inconvenient. Because of this, MoonLine instead uses a configuration file in [INI](https://en.wikipedia.org/wiki/INI_file) format. This has the added benefit of being able to quickly switch between various algorithms and configurations simply by pointing it at a different configuration file.

### Example Configuration
```ini
[Strategy]
path = strategies/
name = crypto_strategy.py

[Data]
provider = quantrocket_csv
path = data/crypto_data.csv

[Backtest]
timezone = America/New_York
trade_backlog = 3 months

[Output]
tearsheet = output/tearsheet.pdf
weights = output/weights.csv
orders = output/orders.csv

[AccountA]
allocation = 1.0
balance = 10000
currency = EUR

[AccountB]
allocation = 0.73
balance = 13751
currency = USD

[AccountC]
allocation = 0.5
balance = 5000
currency = EUR

[TradedCurrencies]
SPY = USD
RETL = USD
NUGT = USD
TMF = USD
TYD = USD
SPXS = USD
SPXL = USD
TECL = USD

[Custom]
test-value = test
```

### Sections
#### Strategy
Describes the strategy to run.
##### `path`
The path to the directory containing the strategy file(s).
##### `name`
Either the exact filename of a strategy file inside the strategy directory (i.e. `crypto_strategy.py`) or the class name of a strategy.

If given a class name, MoonLine will automatically look for and instantiate the given strategy class, so you can freely refactor your strategies and rename their files and it will still be able to pick it up.

If given a filename, only a single strategy can be contained in the given file. When a class name is given, multiple strategies can be housed in the same file.

#### Data
Describes the data to backtest with.
##### `provider`
One of:
* `quantrocket_csv`: Uses the QuantRocket CSV format of storing data for all assets in a single file
* `quantrocket_dataframe`: Similar to `quantrocket_csv` but instead accepts a preloaded Pandas DataFrame in the same format
* `feather`: Uses the Apache Arrow based [Feather](https://github.com/wesm/feather) format for reading data for all assets from a serialized Pandas DataFrame
* `pystore`: Uses the [PyStore](https://github.com/ranaroussi/pystore) database
* `csvbunch`: (WIP) Uses multiple CSV files (one per asset)
* `marketstore`: (WIP) Uses the Alpaca [MarketStore](https://github.com/alpacahq/marketstore) database
* `forex`: Uses multiple CSV files (one per Forex currency-pair) from [Genzai](https://cloud.genzai.dev/s/egNokKAaZtmGA9z)

MoonLine supports both end-of-day as well as intraday input data. These differ mainly in that intraday price data contains an additional index level called "Time".

In our experience, using the `feather` format provides the fastest load and parse times. Using this format, it is possible to load the entire American stock market using daily bars within just a few seconds, provided enough RAM is available.

For convenience and compatibility, the `quantrocket_csv` format is the best choice as it allows manual inspection and modification of the source data via any text editor. Find below a sample of the basic structure for this format. As long as it is possible to create a CSV file with the structure as shown below, MoonLine will be able to ingest it and use the data therein for backtesting.

##### End-of-Day Sample
| Field  |       Date |     265598 |
| ------ | ---------: | ---------: |
| Close  | 2009-05-20 |     73.305 |
| Close  | 2009-05-21 |     74.764 |
| Close  | 2009-05-22 |     75.845 |
| High   | 2009-05-20 |     169.74 |
| High   | 2009-05-21 |     170.65 |
| High   | 2009-05-22 |     171.34 |
| Low    | 2009-05-20 |     137.52 |
| Low    | 2009-05-21 |     138.54 |
| Low    | 2009-05-22 |     139.25 |
| Open   | 2009-05-20 |     289.41 |
| Open   | 2009-05-21 |     290.76 |
| Open   | 2009-05-22 |     291.23 |
| Volume | 2009-05-20 | 73292100.0 |
| Volume | 2009-05-21 | 73291931.0 |
| Volume | 2009-05-22 | 73234500.0 |

```csv
Field,Date,265598
Close,2009-05-20,73.305
Close,2009-05-21,74.764
Close,2009-05-22,75.845
High,2009-05-20,169.74
High,2009-05-21,170.65
High,2009-05-22,171.34
Low,2009-05-20,137.52
Low,2009-05-21,138.54
Low,2009-05-22,139.25
Open,2009-05-20,289.41
Open,2009-05-21,290.76
Open,2009-05-22,291.23
Volume,2009-05-20,73292100.0
Volume,2009-05-21,73291931.0
Volume,2009-05-22,73234500.0
```
##### Intraday Sample
| Field  |       Date |     Time |     265598 |
| ------ | ---------: | -------: | ---------: |
| Close  | 2009-05-20 | 15:45:00 |     73.305 |
| Close  | 2009-05-20 | 16:00:00 |     74.764 |
| Close  | 2009-05-20 | 16:15:00 |     75.845 |
| High   | 2009-05-20 | 15:45:00 |     169.74 |
| High   | 2009-05-20 | 16:00:00 |     170.65 |
| High   | 2009-05-20 | 16:15:00 |     171.34 |
| Low    | 2009-05-20 | 15:45:00 |     137.52 |
| Low    | 2009-05-20 | 16:00:00 |     138.54 |
| Low    | 2009-05-20 | 16:15:00 |     139.25 |
| Open   | 2009-05-20 | 15:45:00 |     289.41 |
| Open   | 2009-05-20 | 16:00:00 |     290.76 |
| Open   | 2009-05-20 | 16:15:00 |     291.23 |
| Volume | 2009-05-20 | 15:45:00 | 73292100.0 |
| Volume | 2009-05-20 | 16:00:00 | 73291931.0 |
| Volume | 2009-05-20 | 16:15:00 | 73234500.0 |

```csv
Field,Date,Time,265598
Close,2009-05-20,15:45:00,73.305
Close,2009-05-20,16:00:00,74.764
Close,2009-05-20,16:15:00,75.845
High,2009-05-20,15:45:00,169.74
High,2009-05-20,16:00:00,170.65
High,2009-05-20,16:15:00,171.34
Low,2009-05-20,15:45:00,137.52
Low,2009-05-20,16:00:00,138.54
Low,2009-05-20,16:15:00,139.25
Open,2009-05-20,15:45:00,289.41
Open,2009-05-20,16:00:00,290.76
Open,2009-05-20,16:15:00,291.23
Volume,2009-05-20,15:45:00,73292100.0
Volume,2009-05-20,16:00:00,73291931.0
Volume,2009-05-20,16:15:00,73234500.0
```
##### `path`
The path to the data required by the selected data provider. This is passed directly to the data provider during instantiation and can be any string it requires, i.e. the path to a file or directory, a URL, base64-encoded data or anything else.
##### `backfill_early`
Either `True` or `False`, which determines the filling behavior of the backtester for data that is not available. If `False` or the option is omitted entirely, data will be filled only after the first valid index. Data before that index will be `pd.NaN`. If `True`, all data will be filled non-discriminately no matter the duration.

#### Backtest
Options of the backtester.
##### `timezone`
The timezone to use as the base to interpret backtesting data in. Can be any timezone in the [tz database](https://en.wikipedia.org/wiki/Tz_database).
##### `start_date`
A valid date in one of two formats:
* `YYYY-MM-DD`: For example `2019-01-01`
* `YYYY-MM-DD HH:mm:ss` For example `2019-01-01 13:47:28`
##### `end_date`
A valid date in one of two formats:
* `YYYY-MM-DD`: For example `2019-01-01`
* `YYYY-MM-DD HH:mm:ss` For example `2019-01-01 13:47:28`
##### `trade_backlog`
A duration expressed in natural language, for example:
* `2 weeks`
* `3 months`
* `2 years`
* `1 week and 3 days`
* `2 days and 4 hours`
* `4M, 22d and 6hours`

Parsing is done via the [durations](https://github.com/oleiade/durations) library.

If this parameter is given, the `start_date` and `end_date` parameters in the configuration file are ignored. Instead, they are computed backwards from the point in time when the strategy is executed.

#### Output
Paths for output artifacts.
##### `tearsheet`
The path to a file (`PDF`). If the file exists, it will be overwritten. This will output a tearsheet in PDF format containing multiple analytical charts to assess the performance of a given strategy.
##### `weights`
The path to a file (`CSV`). This will output a table containing the percentage weights (0.0-1.0) for each asset at each time step.
| Date       | AAPL | MSFT |
| ---------- | ---: | ---: |
| 2019-11-11 |  0.3 |  0.2 |
| 2019-11-12 |  0.3 |  0.2 |
| 2019-11-13 |  0.6 |  0.4 |

```csv
Date,AAPL,MSFT
2019-11-11,0.3,0.2
2019-11-12,0.3,0.2
2019-11-13,0.6,0.4
```
##### `orders`
The path to a file (`CSV`). This will output a table containing asset-quantity pairs for orders at the last timestep of the backtest.
##### Order Sample
| ConId | Account | Account |        OrderRef | TotalQuantity | Exchange | OrderType |  Tif |
| ----- | ------: | ------: | --------------: | ------------: | -------: | --------: | ---: |
| TMF   |       A |     BUY | crypto_strategy |           217 |    SMART |       MKT |  DAY |
| TMF   |       B |     BUY | crypto_strategy |           199 |    SMART |       MKT |  DAY |
| TMF   |       C |     BUY | crypto_strategy |            54 |    SMART |       MKT |  DAY |
| TYD   |       A |     BUY | crypto_strategy |            81 |    SMART |       MKT |  DAY |
| TYD   |       B |     BUY | crypto_strategy |            74 |    SMART |       MKT |  DAY |
| TYD   |       C |     BUY | crypto_strategy |            20 |    SMART |       MKT |  DAY |

```csv
ConId,Account,Action,OrderRef,TotalQuantity,Exchange,OrderType,Tif
TMF,A,BUY,moonline-strategy-eu,217,SMART,MKT,DAY
TMF,B,BUY,moonline-strategy-eu,199,SMART,MKT,DAY
TMF,C,BUY,moonline-strategy-eu,54,SMART,MKT,DAY
TYD,A,BUY,moonline-strategy-eu,81,SMART,MKT,DAY
TYD,B,BUY,moonline-strategy-eu,74,SMART,MKT,DAY
TYD,C,BUY,moonline-strategy-eu,20,SMART,MKT,DAY
```

#### Accounts
Account data for order generation. Accounts are self-contained sections in the configuration file and there can be an unlimited amount of them. Orders are generated for each account individually based on the base percentage weights and scaled for each accounts' allocation and balance values.
##### `allocation`
A percentage value between 0.0 and 1.0 describing how much of the existing `balance` to use for this strategy.
##### `balance`
A number describing the toal capital this account has.
##### `currency`
The currency the capital is measured in, i.e. `USD` or `EUR`.

#### TradedCurrencies
Metadata for currencies per asset. For order generation, it is necessary for MoonLine to known what currency each asset is traded in. This must be specified manually using this section. Simply list each asset with the corresponding currency as a key-value pair like so:
```ini
AAPL = USD
MSFT = USD
BSCH = EUR
```

#### Custom Strategy Options
It may be the case that a specific strategy requires some custom configuration options. This is supported by adding arbitrary sections to the `config.ini` file. Any sections that are not default `moonline` sections will be parsed and put into a custom configuration object that is passed to the strategy on initialization and is available on the strategy as `self.config`.

In the example above, the section `[Custom]` would result in this value for `self.config`:
```python
{"custom": {"test-value": "test"}}
```

Please note that all section names will be forced to lowercase during loading of the config file.

As an additional nicety, `moonline` allows the overriding of custom parameters via command line options.
To overwrite the custom config value above, you would add this to the command invocation:
```
moonline -c config.ini --custom:test-value "some other value"
```

The `self.config` object would then look like this:
```python
{"custom": {"test-value": "some other value"}}
```

The syntax for specifying custom strategy parameters on the command line is:
```
--<section_name>:<param_name> <param_value>
```

In general, custom parameters are first loaded from the config file, and all parameters from the command line will be inserted after that. This means that all CLI parameters are either added to the custom config object if they don't exist yet, or they overwrite the existing value from the config file.

As such, the common pattern for using this feature would be to define default parameters for custom config options in the `config.ini` file and then override the ones that need to be changed via command line parameters to the script invocation.

## Strategy Development
MoonLine itself is a runner for trading strategies which are individual Python files that can reside anywhere on disk. To install a MoonLine algorithm, simply reference the file in the config file, after which it will automatically be loaded and used within the backtester.

While MoonLine can still be used inside QuantRocket as a Moonshot strategy, its main claim to fame is the fact that it comes with a **fully-featured standalone backtester and order generation** based on Moonshot, enabling you to run a backtest completely independently. This includes data ingestion, backtesting, tear sheet rendering as well as order generation. Find an example of a very minimal strategy below.

```python
class MyStrategy(MoonLineStrategy):

    # Specify the calendar we're generally active in
    CALENDAR = "NYSE"
    AAPL = Asset("AAPL", "NASDAQ")

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # Schedule a method to be called every day (if the market is open)
        self.schedule_interval(self.rebalance, "D", 1)

    def rebalance(self, data):
        # 'data' contains the stock data between now
        # and the last time the method was called
        print(data)
        >>> Field                                              Close    High     Low     Open    Volume
        >>> ConId
        >>> Asset(ConId=265598, Symbol=AAPL, Exchange=NA... 243.6414  31.240  29.210  29.2500  979300.0

        # Order AAPL stocks for 30% of our total cash
        self.order_target_percent("AAPL", 0.3)

    # This method is called at the highest resolution available
    # It's mostly a convenience method and can be skipped with
    # an empty implementation like the one below
    def handle_data(self, data):
        pass
```

## Asset Names
All references to assets in MoonLine are made via the `ProtoAsset` class. This is a simple data class that takes a string (the symbol of the asset) and is used for various introspective analytics while running the backtest.

They can be created either by giving a symbol name and an exchange or a ConId.\
If Forex data is used specify a currency pair here:
```python
spy = ProtoAsset("SPY")
```

## Usage
```bash
usage: moonline.py [-h] -c config.ini [-y]

optional arguments:
  -h, --help            show this help message and exit
  -c config.ini, --config config.ini
                        A file containing MoonLine configuration options
  -y, --yes             If given, automatically answers script questions with
                        'yes'
```