Updating the mapping of an elasticsearch index
Tags: elasticsearch
Published: 2020-04-21 14:11:00

Okay, so you've set up elasticsearch. You've indexed your data. Search is super fast. All's good. But, suddenly, you have a requirement for which you need to change the mapping of your index. Maybe you need to use a different analyser, or maybe it's as simple as adding a new field to your document, which requires you to add the associated static mapping.

If you find yourself in such a situation, here are a few approaches you can take - 
- **Approach 1** - with downtime; index from external data source.
    - This assumes that you have an external data source such as a database from which you can index data all over again, as if you were doing it for the first time.
    - When to use?
        - This approach only makes sense for testing purposes in local or in staging. This should not be used in a production environment because downtime isn't really desirable.
    - Steps
        - Delete the index using the [Delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html)
        - Create the index, and set the new mapping using the [PUT Mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html)
        - Index documents from external data source. You could do this using the [Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
- **Approach 2** - without downtime; index from external data source
    - When to use?
        - You could use this approach in production, but if you have a large number of documents, indexing from an external data source like a DB can be a time-consuming process.
    - Steps
        - If not done already, create an [alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html) `index_alias` for your existing index  (`old_index`) and change your code to use the alias instead of `old_index` directly.
        - Create a new index `new_index`
        - Index documents from external data source. You could do this using the [Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
        - Move the alias `index_alias` from `old_index` to `new_index`.
    - Caveats
        - While the downtime is essentially zero, there could still be [consistency issues](https://blog.codecentric.de/en/2014/09/elasticsearch-zero-downtime-reindexing-problems-solutions/)
        - Indexing from an external data source like a DB can be a time-consuming process if you have a large number of documents.
- **Approach 3** - without downtime; index from elasticsearch
    - When to use?
        - Can be used in production when you want to [change the mapping of an existing field](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings). If you are merely adding a field mapping, prefer Approach 4
    - Steps
        - If not done already, create an [alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html) `index_alias` for your existing index  (`old_index`) and change your code to use the alias instead of `old_index` directly.
        - Create a new index `new_index`
        - use elasticsearch [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-reindex.html) API to copy docs from `old_index` to `new_index`.
        - Move the alias `index_alias` from `old_index` to `new_index`.
    - Caveats
        - While the downtime is essentially zero, there could still be [consistency issues](https://blog.codecentric.de/en/2014/09/elasticsearch-zero-downtime-reindexing-problems-solutions/)
- **Approach 4** - without downtime; update existing index
    - When to use?
        - Can be used in production when you want to merely add a new field mapping.
    - Steps
        - Update mappings of index online using [PUT mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html). 
        - Use [_update_by_query](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html) API with params
            - `conflicts=proceed`
                - In the context of just picking up an online mapping change, documents which have been updated during the process, and therefore have a version conflict, would have picked up the new mapping anyway. Hence, version conflicts can be ignored.
            - `wait_for_completion=false` so that it runs as a background task
            - `refresh` so that all shards of the index are updated when the request completes.
    - Caveats
        - Can't be used if you want to [change the mapping of an existing field](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings). Use Approach 3 instead.