thread-size-mapper 0.0.41
Thread's size mapper
Requires Python: >=3.9

# thread-size-mapper

This library translates between item size strings (as scraped from third parties) and translates into our standard sizes.

Size mapping vs size matching:
- "size mapping" is turning a `str` from a scraper into a structured representation of an item's size.
- "size matching" is about determining whether an item is the right size for a user.
- This library contains functionality for both.

Definitions of 'types of size':
- "Raw sizes"
  - type `str`
  - These are the original size strings scraped from partners' websites.
- "Item sizes"
  - type `ThreadSize`
  - These are parsed & validated sizes.
    Currently these are exported to Thimble (and present in its data-dict) to power filters, downweights and the `/v2/preferred-sizes` endpoint.
- "User-facing sizes"
  - type `UserSize`
  - These form a reduced set of sizes that we present to the user (e.g. in Registration, and in size filters in Shop).
    The aim is to simplify down the (complex!) world of sizes, for example menswear shorts are commonly specified as a waist size (e.g: "W32"), small/medium/large, or occasionally something more exotic.
    In registration we collect waists (in inches), therefore `MensBottomsWaists(waist)` represents the user-facing notion of a 'waist size', and means that a user with waist=W32 will also match the appropriate small/medium/large size bottoms too.
  - How to [add user-facing sizes](docs/Adding-user-facing-sizes.md) is documented separately.
- "Registration sizes"
  - These are the set of sizes collected in the Thread registration flow.
    They appear in this library via the classes `size_mapper.size_dict.MenswearSizeDict` and `size_mapper.size_dict.WomenswearSizeDict`.
    Since these sizes are very much user-facing, they have influenced our choices for the "User-facing sizes" `UserSize` implementations considerably.

## Usage:

### Existing usage (size-matching in Thimble)

The interface to the size mapper is a function `map_size`, which takes a size string and a parent category slug as input:

```python
>>> from size_mapper import map_size

>>> result = map_size('32 long', parent='trousers-chinos')

>>> result
WxLBottomsSize(waist=32, length=34)
>>> str(result)
'W32L34'
```

Sometimes it is possible to infer a precise size from the input, however sometimes the size is ambiguous and it is left up to the caller to select the appropriate size scheme (knowing, e.g. the third party)

```python
>>> map_size('43', parent='shoes')
ShoeSize(uk_size=9.0)

>>> map_size('10', parent='shoes')
AmbiguousSize(options={'us': ShoeSize(uk_size=9.0), 'uk': ShoeSize(uk_size=10.0)})
```

This is roughly how size matching occurs presently: the size mapper outputs `ThreadSize`s for items, and in requests Thimble takes a `size_dict` and works out which items are size-matched for the user.

```mermaid
flowchart LR
  itemsize["Item size 'W32 reg'"]
  sizedict["
    Size dict = {
        'tops': 'M',
        'shoes': 'uk9',
        'bottoms': [('W31', 'L32')],
        ...}
  "]

  subgraph ThreadSizes
    ThreadSize1["&ltWxLBottomsSize: waist=32, length=32&gt"]
  end

  itemsize --> |map_size|ThreadSizes --> Thimble
  sizedict --> Thimble
```

### Usage going forward

See [this RFC](https://www.notion.so/threadwiki/Exposing-size-mapping-data-in-Postgres-BigQuery-19bc0955b06a4444bf1e1c7b24dff7c4) for extra details.

We would like to get to the point where size-matching can be done entirely in the space of user-facing sizes.
This works as follows:

```mermaid
flowchart TB
  itemsize["Item size 'W32 reg'"]
  sizedict["
    Size dict = {
        'tops': 'M',
        'shoes': 'uk9',
        'bottoms': [('W31', 'L32')],
        ...}
  "]


  subgraph UserSizes["UserSizes (for item)"]
    A["&lt;UserSizeMatchType.EXACT: 0&gt;: [
      'menswear-bottoms-measurement:w32l32',
    ]"]
    B["&lt;UserSizeMatchType.APPROXIMATE: 1&gt;: [
      'menswear-bottoms-measurement:w32l32',
      'menswear-bottoms-measurement:w31l32',
      ...
    ]"]
  end

  subgraph UserSizes2["UserSizes (for user, from size dict)"]
    X["
      [
        'menswear-tops:m',
        'menswear-shoes:uk9',
        'menswear-bottoms-measurement:w31l32',
        ...
        'menswear-one-size',
      ]
    "]
  end

  itemsize -->|user_sizes_for_size|UserSizes
  sizedict -->|menswear_size_vector_for_user|UserSizes2

  subgraph SizeMatch["Size match? (nonzero overlap means a match at that level)"]
    SM1["&lt;UserSizeMatchType.EXACT: 0&gt;: []"]
    SM2["&lt;UserSizeMatchType.APPROXIMATE: 1&gt;: [
      'menswear-bottoms-measurement:w31l32',
    ]"]
  end

  UserSizes & UserSizes2 --> SizeMatch --> Match["(Approximate) match!" ]
```

- The function `size_mapper.user_sizes_for_size` maps a size string to the corresponding user-facing sizes which would match, at the various levels of matching precision.
- The functions `size_mapper.{wo}menswear_size_vector_for_user` convert a `size_dict` into a list of user-facing sizes.
- Size-matching is then simple: a set overlap between the user-facing sizes of the item and user.
  Importantly, this is easy to store on items in the database, making it easy to implement size-matching in Styleme & also expose this in the Data Warehouse.


```python
>>> from size_mapper import user_sizes_for_size

>>> user_sizes_for_size('uk9.5', parent='shoes')
{<UserSizeMatchType.EXACT: 0>: [],
 <UserSizeMatchType.APPROXIMATE: 1>: ['menswear-shoes:uk9'],
 <UserSizeMatchType.GENERIC: 2>: [],
 <UserSizeMatchType.NO_USER_INFO: 3>: ['menswear-shoes:unknown']}
```

(This means that a shoe with size string 'uk9.5' would be an exact match for no one, and an approximate size match for men with size uk9.)

## Installation

Use `pip install thread-size-mapper --extra-index-url <THREAD_INDEX_URL>` to install, or
`pip install -e .` to perform an editable install.

## Project structure:

There are currently two approaches internally to size-mapping: the older approach (covering menswear on Thread.com) and a newer, simpler approach covering womenswear and H&M sizes.
We expect to migrate menswear over eventually.

### Original approach: Thread.com menswear

Depending on the item's `parent category`, one of the parsers in [parsers.py](https://github.com/thread/thread-size-mapper/blob/master/size_mapper/parsers.py) will be invoked.
Each of these enables a set of tokenisers ([tokenisers.py](https://github.com/thread/thread-size-mapper/blob/master/size_mapper/tokenisers.py)), which attempt to match a range of common sizing patterns.
Multiple tokenisations are returned, and can be selected in one of two ways:

* tokenisations which exactly match a particular set of tokens.  This is necessary for trousers: the highly varied, concise, and locally ambiguous size strings necessitate this approach.
* tokenisations which span as much of the input string as possible.

There are many tests.

#### Adding new size schemes (original approach):

1. Add relevant examples to the tests.
The tests run with logging at the debug level, which provides useful information on how the input strings are being tokenised and interpretations selected.
2. Update parsers & tokenisers: tokenisers are intended to be reasonably focussed around detecting a particular piece of information (e.g. a string hinting that a number is a leg length).
There is a parser assigned to each parent category: this will probably need updating.
Generally, start with the more specific interpretations first and fall back to broader, more speculative interpretations.
3. There are many tests: these should catch unintended consequences of your changes.

### Newer approach: Thread.com womenswear and H&M sizes

The original approach took the size string for each item, looked for substrings which provided information about sizes and tried to put these together into consistent interpretations of the overall size.
The main problem was one of complexity: invariably some strings have multiple interpretations, and it's impossible to always choose the correct one.
As a particularly difficult example, '30L' on trousers could be 30 = waist 30, L = long.  Or it could be leg length 30.
Additionally, having two levels of tokenisers and interpretations proved relatively complicated to update predictably.

In the new approach, there is a single concept: a Pattern which

- contains a regex (to match the entire size string)
- matches one or more categories
- may be restricted to matching specific brands
- references a function to convert the match + extracted parameters into an object from `size_mapper.sizes`

In `size_mapper.mapping` we build up a collection of `Pattern`s to tile the space of size strings.
Some are extremely specific to deal with the idiosyncrasies of certain brands, while others are very broad and match thousands of items.
A real advantage of this approach is that we can reach arbitrarily high coverage by writing increasingly specific patterns
(should we need them).

#### Updating the size mapper (new approach)

The recommended approach is a combination of 1) updating the unit tests and 2) running the size mapper dev script, which
will run a large number of real size strings through the mapper and show you how the various rules are working together.

The [size mapper dev script](scripts/dev-run-size-mapper.py) outputs

- size strings for which there was no matching pattern
- examples of the size string that matched each pattern
- possible conflicts: item colours having item colour sizes which matched different patterns.  This often indicates
  a bug (or a problem with the item) and helps make sure that the patterns are applied in a suitable order.

Several filtering options are available:

- `--clothing-type`
- `--parent-category`
- `--brand`
- `--item-colour`

Focussing on a particular parent category at a time works well, by updating the patterns and rerunning the dev tool.

For example, running `ipython scripts/dev-run-size-mapper.py -- --db $KOBO_URL --clothing-type womenswear --parent-category womens-dresses` gave the following output:

![Dev tool output](docs/dev_tool_output.png)