Use AWS Serverless to sell items in Habitica
Tags: lambda, aws, habitica
Published: 2024-06-30 08:38:00

I use Habitica daily. It is a very strong motivator to keep me on track with my daily tasks. However, I already collected all the free pets and mounts and so I don't need most of the items I collect. Selling them is a pain as you have to sell one item with each click. But Habitica has a [remarkable API](https://habitica.com/apidoc/). I decided to automate this process with AWS Lambda and EventBridge scheduled events.

The completed project can be found here:
https://github.com/ppabis/habitica-item-seller/tree/v1

Getting API key
---------------
Obviously you need a Habitica account. Once you are there (on a desktop browser), go to Settings -> [Site Data](https://habitica.com/user/settings/siteData). You will see API section where you can download the token. But be careful as the only way to reset the token is to send email to Habitica support. Also save your user ID.


![Where is API token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vvrd200v853o2wn7yg57.jpg)

Storing token in AWS Secrets Manager
------------------------------------
Secrets Manager is a great place to store such a token. Alternatively you can use SSM Parameter Store with `SecretString` to keep costs low. In a new directory for our project we will create `template.yaml`. This will be our AWS SAM template. Normally SAM doesn't have a resource for Secrets Manager. However, it is just a CloudFormation file with extras so we can use all CloudFormation compatible resources.

The user ID and API key will be provided as secret parameters (NoEcho). You will need to specify them every time you use `sam deploy --guided`.

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: A function that sells Habitica items periodically leaving only a subset of those

Parameters:
  HabiticaApiKey:
    Type: String
    Description: The API key for the Habitica account
    NoEcho: true
  HabiticaUserId:
    Type: String
    Description: The user ID for the Habitica account
    NoEcho: true

Resources:
  HabiticaSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: Secret for the Habitica API key and user ID
      SecretString: !Sub '{"${HabiticaUserId}": "${HabiticaApiKey}"}'
```

Authenticating with Habitica
----------------------------
So using the provided values we can authenticate with Habitica API by providing them as headers. We will define our functions with Python. I created the file below in a new directory under `sell_items/auth.py`. The first function is used to retrieve the user and key from Secrets Manager by specified ARN in the environment variables. If this is not specified, it tries resolving the secret and user directly from the environment variables (this can be used for testing locally).

The secret is defined in Secrets Manager as a JSON object where the key is the user ID and the value is the API token. `get_secret` function will return the first key-value pair as a tuple. The `x-client` header is based on user ID as per Habitica requirements when creating a third-party integration - see [this wiki page](https://habitica.fandom.com/wiki/Guidance_for_Comrades#X-Client_Header).

```python
import os, json
from boto3 import client

def get_secret(habitica):
    """
    Retrieves the secret from AWS Secrets Manager.
    """
    secrets = client("secretsmanager")
    secret = secrets.get_secret_value(SecretId=habitica)
    secret = json.loads(secret['SecretString'])
    return list(secret.items())[0]


def get_headers() -> dict:
    """
    Produces a list of headers that are required for Habitica API requests.
    """
    # By default try to load environment variables.
    HABITICA_USER=os.getenv('HABITICA_USER')
    HABITICA_KEY=os.getenv('HABITICA_KEY')
    
    # But if there's HABITICA_SECRET set, use Secrets Manager.
    HABITICA_SECRET=os.getenv('HABITICA_SECRET')
    if HABITICA_SECRET:
        print('Using Secrets Manager.')
        HABITICA_USER, HABITICA_KEY = get_secret(HABITICA_SECRET)

    HABITICA_CLIENT=f"{HABITICA_USER}-itemseller10"
    print(f"Client {HABITICA_CLIENT}")
    
    return {
        "x-api-user": HABITICA_USER,
        "x-api-key": HABITICA_KEY,
        "x-client": HABITICA_CLIENT
    }
```

Retrieving data from the Habitica API
-------------------------------------
In order to test the authentication, we can write a function that will get the inventory that we will later use in the application. This function will just retrieve all hatching potions, eggs and food from inventory along with gold. In case the authentication fails, we will throw an exception.

```python
import requests

HABITICA_URL="https://habitica.com/api/v3"

def get_inventory(headers: dict) -> tuple[dict, float]:
    """
    Gets user inventory in the form of eggs, potions, food and stats (gold).
    Returns a tuple of (items: dict, gold: float).
    """
    url = f"{HABITICA_URL}/user?userFields=stats,items.eggs,items.hatchingPotions,items.food"
    response = requests.get(url, headers=headers)
    code = response.status_code
    if code == 200:
        items = response.json()['data']['items']
        gold = response.json()['data']['stats']['gp']
        return items, gold
    raise Exception(response.json()['message'])
```

Next in a new file file `main.py` let us glue the two function together and print out the result. Set the values for `HABITICA_SECRET` and `HABITICA_USER` as environment variables.

```python
from auth import get_headers
from actions import get_inventory

if __name__ == "__main__":
    headers = get_headers()
    items, gold = get_inventory(headers)
    print(f"Items: {items}")
    print(f"Gold: {gold}")
```

To run it safely without exposing your API token too much, use the following bash commands. It will ask you for the token without any prompt and won't echo.

```bash
$ read -s HABITICA_USER
$ read -s HABITICA_KEY
$ export HABITICA_USER HABITICA_KEY
$ python3 main.py
Client 12345678-8b3d-48af-aabb-ccddee445566-itemseller10
Gold: 22670.394004083322
Items: {'eggs': {'BearCub': 100, 'TigerCub': 100, 'PandaCub': 100, 'Fox': 100,...
```

Another important thing in Habitica is that some of the items are not obtainable randomly but are very valuable, purchased with gems. These items we want to protect from our script. I created a function that will filter items to leave only the most common ones. It is a simple Python array operation that you can [find here](https://github.com/ppabis/habitica-item-seller/blob/main/sell_items/filters.py).

Integrate the function in your `get_inventory` function.

```python
from filters import filter_items
# ...
    if code == 200:
        items = response.json()['data']['items']
        gold = response.json()['data']['stats']['gp']
        return filter_items(items), gold
# ...
```

Functions for selling an item
-----------------------------
Now we can write a function that will sell an item. This will be similarly just a REST call to Habitica. Compared to the UI available on the web or mobile, this function allows us to set the amount of items to sell.

```python
def sell_item(headers: dict, item_type: str, item_key: str, count: int):
    """
    Sells a specific item with the given count.
    1:1 mapping of Habitica API documentation.
    """
    url = f"{HABITICA_URL}/user/sell/{item_type}/{item_key}?amount={count}"
    response = requests.post(url, headers=headers)
    code = response.status_code
    if code == 200:
        return response.json()
    raise Exception(response.json()['message'])
```

Find the most common item in your inventory and sell one of it for testing. Add the sell routine to the `main` section. This function will return a lot of data about the player including inventory after sale and current gold amount.

```python
from auth import get_headers
from actions import get_inventory, sell_item

if __name__ == "__main__":
    headers = get_headers()
    items, gold = get_inventory(headers)
    print(f"Gold: {gold:.2f}\nItems: {items}")

    sale = sell_item(headers, 'eggs', 'TigerCub', 1)
    new_gold = sale['data']['stats']['gp']
    print(f"Gold after sale: {new_gold:.2f}")
```

Run the script again and see how much money you have after a sale.

```bash
$ python3 main.py
Client 12345678-8b3d-48af-aabb-ccddee445566-itemseller10
Gold: 22824.60
Items: {'eggs': {'BearCub': 101, 'TigerCub': 99, 'PandaCub': 100, ...
Gold after sale: 22827.60
```

Iterating through the items
----------------------------
Now, we can iterate through all the items and sell them. However, we have a lot of item types and we don't want to hit Habitica's API rate threshold, which is very low. In order to avoid this, we will sell only items from one category in one run of the function. The category will be passed to Lambda in the event. The new function will take `event` as an argument.

Next thing is that we don't want to sell all the items. This fact can help us improve the function even more by filtering out items that don't need to be sold as we have less of them then the value of items to keep. The function will return how much gold we made for this function invocation.

```python
def sell_items(headers: dict, items_to_leave: int, event) -> float:
    """
    Sells items of a specific category in `event['sell_category']`.
    """
    if not is_event_valid(event):
        print("No sell_category specified!")
        raise ValueError("Missing 'sell_category' key in event. And such, ignoring this event.")
    
    category = event['sell_category']
    
    items, gold = get_inventory(headers)
    new_gold = gold     # To define it in case we don't sell anything.
    print(f"Starting with gold [{category}] {gold:.2f}.")
    items_to_sell = items[category]

    # Filter items that we have more than threshold.
    items_to_sell = {k: v for k, v in items_to_sell.items() if v > items_to_leave}
    
    # Print history of what is being done.
    string_items = ", ".join([f"{k} ({v})" for k, v in items_to_sell.items()])
    print(f"Items to sell [{category}]: {string_items}")
    
    last_sale = None
    # Sell items and store the last sale for gold calculation.
    for item, amount in items_to_sell.items():
        count = amount - items_to_leave
        print(f"Selling {count} of {item}.")
        try:
            _sold = sell_item(headers, event['sell_category'], item, count)
        except Exception as e:
            print(f"Failed to sell {item}: {e}.")
            continue
        last_sale = _sold
        print(f"Sold {count} of {item}.")
    
    if last_sale is not None:
        new_gold = last_sale['data']['stats']['gp']
        print(f"Gold after sale [{category}]: {new_gold:.2f}. Made {new_gold - gold:.2f} gold.")
    else:
        print(f"Didn't sell anything for {category}.")

    return new_gold - gold
```

Main function for Lambda
------------------------
Now we need to glue together all the parts. In `main.py` we can import required functions that we created. We will also initialize all the constants such as API key, user ID, environment configuration. Remember to remove everything under `if __name__ == "__main__":` as it is not needed in Lambda and can be called accidentally.

```python
from actions import sell_items
from auth import get_headers
import os, json

HEADERS = get_headers()
# This is the amount of each item that will be left and not sold.
ITEMS_TO_LEAVE = int(os.getenv('ITEMS_TO_LEAVE', "150"))

def lambda_handler(event, context):
    if event is str:
        event: dict = json.loads(event)
    
    try:
        gp = sell_items(HEADERS, ITEMS_TO_LEAVE, event)
    except Exception as e:
        return {
            "statusCode": 400,
            "body": str(e)
        }
    
    return {
        "statusCode": 200,
        "body": "Earned {gp:.2f} gold."
    }
```

Adding function to SAM template
-------------------------------
SAM is perfect for handling Lambda functions. It can install `requirements.txt` for us, zip everything and deploy. In the scripts I wrote above, only `requests` is needed as an extra package. If you prefer to keep your function slim, you can use `urllib3` alternatively. Otherwise, create `requirements.txt` in your code directory (my is `sell_items`) with just one line: `requests`.

We will also specify Lambda memory and execution timeout. Add lines below to your `template.yaml` under the description.

```yaml
# ...
Globals:
  Function:
    Timeout: 15
    MemorySize: 128
```

In `Resources` section we will add the Lambda. It will reference the `Secret` so it's worth putting it below the `HabiticaSecret` block. In `HABITICA_SECRET` environment variable we will have ARN of the secret that will be picked up by our function. In `Policies` we will also specify IAM permissions for Lambda's execution role so that it can read the secret. `AWSSecretsManagerGetSecretValuePolicy` is a managed policy that will put all the necessary permissions, without us needing to refer the documentation 😁. I recommend keeping architecture as `arm64` as it is more cost efficient. `CodeUri` refers to the directory where you keep all your Python files.

```yaml
Resources:
  # ...
  HabiticaSellItems:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: sell_items/
      Handler: main.lambda_handler
      Runtime: python3.12
      Architectures:
        - arm64
      Policies:
        - AWSSecretsManagerGetSecretValuePolicy:
            SecretArn: !Ref HabiticaSecret
      Environment:
        Variables:
          HABITICA_SECRET: !Ref HabiticaSecret
          ITEMS_TO_LEAVE: "100"
```

Scheduling the function
-----------------------
As you remember from previous sections, Habitica's API has a strict rate limit. To mitigate this, we split the selling routines into categories. For each category we will define a separate EventBridge schedule that will fire at a different time. Append `Events` section to you `Function` resource. For each event definition we will specify a different input to be feed to the function.

```yaml

 HabiticaSellItems:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: sell_items/
      #...
      Events:
        FoodSchedule:
          Type: Schedule
          Properties:
            Schedule: cron(10 1 * * ? *)
            Input: '{"sell_category": "food"}'
            Enabled: true
        EggsSchedule:
          Type: Schedule
          Properties:
            Schedule: cron(15 1 * * ? *)
            Input: '{"sell_category": "eggs"}'
            Enabled: true
        HatchingPotionsSchedule:
          Type: Schedule
          Properties:
            Schedule: cron(20 1 * * ? *)
            Input: '{"sell_category": "hatchingPotions"}'
            Enabled: true
```

The above configuration will run the tasks at 1:10AM UTC, 1:15 and 1:20. Next we will build the configuration using SAM and deploy it as well. You will need your AWS access keys and [SAM installed](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html#install-sam-cli-instructions). For the first run use `sam deploy --guided` to give Habitica API key and user. If you don't change the user or API key, just use `sam deploy` and it will skip asking you for parameters of the secret.

```bash
$ sam build
$ sam deploy --guided
```

Watching the results
--------------------
So I didn't specify `ITEMS_TO_LEAVE` at the beginning so it defaulted to 150. It successfully ran at night and sold all the items I had more than 150 of. For the next night I lowered the threshold to 100 and again it worked perfectly.

![Sold items every day](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sf5hf7fklzx2daff5a50.jpg)

I also checked CloudWatch to see the logs of the function and the whole process. Here everything seemed stable and behaved as expected. First selling down to 150 things in the inventory and the next day to 100.

![CloudWatch logs day 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6pq10wsfetoih0pr2sew.jpg)

![CloudWatch logs day 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ze1lua8ody0hr8g5e8xu.jpg)