Solr + Python — A Tutorial
Tags: solr, python, tutorial
Published: 2018-04-24 19:40:52

---
title: Solr + Python — A Tutorial
published: true
description: How to create a gallery page with Solr and Python
tags: solr, python, tutorial 
canonical_url: https://medium.com/@avremelk/solr-gottchas-a-tutorial-a953c8b3e775
---

__Update__: I have pushed my Python code to GitHub ([repo is here](https://github.com/avremel/Solr-Python-Tutorial)). My implementation is a tad more advanced than this tutorial. See the Readme file and code comments.

> ###Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene (Solr website).

<img src="https://thepracticaldev.s3.amazonaws.com/i/stwapm3402hoqq5vt10b.png"/>

##Goal
My goal is to demonstrate building an e-commerce gallery page with search, pagination, filtering and multi-select that mirrors the expectations of a typical user. See [this article](https://baymard.com/blog/allow-applying-of-multiple-filter-values) for a nice explanation of the multi-select filtering I am trying to implement.

Search should work for phrase queries like “mens shrt gap” or “gap 2154abc”, factoring in typos, various word forms ([stemming](https://en.wikipedia.org/wiki/Stemming)) and phonetic spelling.

##Solr Setup
Solr 7 is installed locally on my computer with an active connection to a database. Solr is using the [deltaQuery](https://wiki.apache.org/solr/DataImportHandler) feature (in `db-data-config.xml`) to detect changes in my database and import those records into Solr.

##Web Development Setup
I have a basic Django/React app with Python 3. See [this article](https://hackernoon.com/reconciling-djangos-mvc-templates-with-react-components-3aa986cf510a) for ideas on how to integrate Django with React. I recommend following [these instructions](https://lucene.apache.org/solr/guide/6_6/using-python.html) to create your own Solr client.

I was considering using [pySolr](https://github.com/django-haystack/pysolr) as a client, but it lacks good documentation and seems to have been neglected since 2015 (like most Solr libraries). Nevertheless, pySolr can work if you are ready to comb through the GitHub issues and codebase.

If you are using pySolr:

1. Paste `export DEBUG_PYSOLR=’true’` into your terminal before running your server, and you will be able to view the URL generated by pySolr.

2. The URL you see in your terminal doesn’t seem to be clued in about URL encoding issues, so a query like `Dolce & Gabbana` will work on your website, but break when you paste the URL into a browser.

##Facets & Facet Pivots
Facets are synonymous with product categories or specs. Solr has an option to return the available facets with their respective counts for a specific query. You can control the minimum number of products required in a facet by setting `facet.mincount=<number>`.

For example, if you are selling brand named clothing, `facets` might refer to gender, style and material. If the search was for “mens casual gap”, the facets returned would look like this (notice the constraints on `gender` and `style`):
```python
"facet_fields" : {
   "gender" : [
       "Men" , 25,
       "Women" , 0 
    ],
   "style" : [
       "casual", 10,
       "dress", 0
    ],
   "material" : [
       "wool", 15,
       "cotton", 10
    ],
   }
```

##Example Query
Let’s run through an example:
```python
from urllib.request import urlopen
import urllib.parse
import simplejson

def gallery_items(current_query):

    solr_tuples = [
        # text in search box
        ('q', "mens shirt gap"),
        # how many products do I want to return
        ('rows', current_query['rows_per_page']),
        # offset for pagination
        ('start', current_query['start_row'] * current_query['rows_per_page']),
        # example of a default sort, 
        # for search phrase leave blank to allow 
        # for relevancy score sorting
        ('sort', 'price asc, popularity desc'),
        # which fields do I want returned
        ('fl', 'product_title, price, code, image_file'),
        # enable facets and facet.pivots
        ('facet', 'on'),
        # allow for unlimited amount of facets in results
        ('facet.limit', '-1'),
        # a facet has to have at least one 
        # product in it to be a valid facet
        ('facet.mincount', '1'),
        # regular facets
        ('facet.fields', ['gender', 'style', 'material']),
        # nested facets
        ('facet.pivot', 'brand,collection'),
        # edismax is Solr's multifield phrase parser
        ('defType', 'edismax'),
        # fields to be queried
        # copyall: all facets of a product with basic stemming
        # copyallphonetic: phonetic spelling of facets
        ('qf', 'copyall copyallphonetic'),
        # give me results that match most fields
        # in qf [copyall, copyallphonetic]
        ('tie', '1.0'),
        # format response as JSON
        ('wt', 'json')
    ]

    solr_url = 'http://localhost:<port>/solr/<core>/select?'
    # enocde for URL format
    encoded_solr_tuples = urllib.parse.urlencode(solr_tuples)
    complete_url = solr_url + encoded_solr_tuples
    connection = urlopen(complete_url)
    raw_response = simplejson.load(connection)

```
Phrase search will be discussed in the next section — Schema Modeling.
* I would suggest using tuples for each key-value pair as it will be easier to `urlencode`. It will also be easier to manipulate, particularly when you have a complicated `fq` with a ton of `AND`, `OR` logic (which will happen very soon if you are doing filtering).
* Each facet group will have its own `fq` field. This ensures that `AND` logic is applied across filter groups. Here is code for applying `OR` logic within a facet group:
```python
    def apply_facet_filters(self):
        if self.there_are_facets():
            for facet, facet_arr in self.facet_filter.items():
                if len(facet_arr) > 0:
                    new_facet_arr = []
                    for a_facet in facet_arr:
                        new_facet_arr.append("{0}: \"{1}\"".format(facet, a_facet))
                    self.solr_args.append(('fq', ' OR '.join(new_facet_arr)))

```
* `facet.pivot.mincount` allows you to control the minimum number of products required for a `facet.pivot`, but beware, if you set it to 0, [your server will likely crash](https://stackoverflow.com/questions/49837742/solr-facet-pivot-mincount-0-crashes-server).
* I’ve found that field values needed to be formatted in quotes: `‘fq’: “brand: \”{0}\””.format(current_query[‘current_brand’])`
* `facets` are returned in arrays like `[‘brand’, ‘gap’]`, not a `dict()` which I find inconvenient. Here is one way to format them:
```python
import more_itertools as mit
facets = {}

for k,v in raw_response['facet_counts']['facet_fields'].items():
    spec_list = [list(spec) for spec in mit.chunked(v, 2)]
    spec_dict = {}
    for spec in spec_list:
        spec_dict[spec[0]] = spec[1]
    facets[k] = spec_dict

raw_response['facet_counts']['facet_fields'] = facets
```
* By default, if a user selects a facet in a facet group, Solr will return that facet group with only the selected facet, since the search has been narrowed down. But many times, a user would like still like to view the unselected facets and associated counts, to enable multi-select. To allow this functionality, use [tagging and excluding](https://lucene.apache.org/solr/guide/6_6/faceting.html#Faceting-TaggingandExcludingFilters). See [my repo](https://github.com/avremel/Solr-Python-Tutorial) for a possible implementation.
* To create price ranges as a filter with custom intervals, copy `price` to a new field with one of Trie `fieldTypes`. The new field should have `indexed` and `stored` set to `false`, and `docValues` set to `true`. Then follow the [instructions](https://lucene.apache.org/solr/guide/6_6/faceting.html#Faceting-IntervalFaceting) to add custom ranges. See the next section on schema modeling. See [my repo](https://github.com/avremel/Solr-Python-Tutorial) for a possible implementation.

##Schema Modeling
> *If you can get past the idea that fields exist simply to store properties of data, and embrace the idea that you can manipulate data so it can be found as users expect it, then you can begin to effectively program relevance rules into the search engine.* (Relevant Search, Chapter 5)

We are ready to modify fields in our document schema to conform to the users’ perception of our products.

Take a look at the [documentation](https://lucene.apache.org/solr/guide/6_6/schema-api.html) about how to update the schema, particularly the sections on [tokenizing](https://lucene.apache.org/solr/guide/6_6/tokenizers.html) and [filtering](https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html). Learn about [stemming filters](https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html). Ask yourself which tokens/filters are relevant for your situation, and whether it should be apply at query or index time.

I will be following a [recommendation in the documentation](https://lucene.apache.org/solr/guide/7_2/putting-the-pieces-together.html#working-with-text) to copy all fields a user might be interested in into a single `copyall` field. This solves the [albino elephant issue](http://lucene.472066.n3.nabble.com/The-downsides-of-not-splitting-on-whitespace-in-edismax-the-old-albino-elephant-prob-td4327440.html), as well as signal discordance:

> _As we’ve stated, when users search, they typically don’t care how documents decompose into individual fields. Many search users expect to work with documents as a single unit: the more of their search terms that match, the more relevant the document ought to be. It may surprise you to know that search engine features that implemented this idea were late to the party. Instead, Lucene-based multifield search depended on field-centric techniques. Instead of the search terms, field-centric search makes field scores the center of the ranking function. In this section, we explore exactly why field-centric approaches can create relevance issues. You’ll see that having ranking functions centered on fields creates two problems:_

> _**The albino elephant problem** — A failure to give a higher rank to documents that match more search terms._

> _**Signal discordance** — Relevance scoring based on unintuitive scoring of the constituent parts (title versus body) instead of scoring of the whole document or more intuitive larger parts, such as the entire article’s text or the people associated with this film. (Relevant Search, Chapter 6)_

We will be using the [Schema API](https://lucene.apache.org/solr/guide/6_6/schema-api.html) through the [Admin UI](https://lucene.apache.org/solr/guide/6_6/overview-of-the-solr-admin-ui.html). You cannot edit the schema file manually ([why](https://lucene.apache.org/solr/guide/6_6/schema-api.html)). Here is the recipe for creating the `copyall` field:

* Step 1: Create a `fieldType` for the field. I am using the same `fieldType` for both index and query time. I have kept the stemming light to ensure that brand names stay intact.
```xml
      <fieldType name="facets" class="solr.TextField">
        <analyzer>
          <tokenizer class="solr.ClassicTokenizerFactory"/>
          <filter class="solr.LowerCaseFilterFactory"/>
          <filter class="solr.StopFilterFactory"/>
          <filter class="solr.ClassicFilterFactory"/>
          <filter class="solr.EnglishMinimalStemFilterFactory"/>
        </analyzer>
      </fieldType>
```

* Step 2: Create a `copyall` field with a facets as the `fieldType`. Set `multiValued=true` to allow multiple values in the field (as an array). Set `omitNorms=true` since users don’t care about the length of each field ([docs](https://lucene.apache.org/solr/guide/6_6/field-type-definitions-and-properties.html#field-default-properties)), and we don’t want Solr to care either.

* Step 3: Create copyFields for every field in the data source that you want to be copied. Remember, [there is no chaining of copyField’s](https://stackoverflow.com/questions/9632019/copyfield-chain).
```python
    {
       "add-copy-field":{
           "source":"brand",
           "dest":"copyall"
        }
    }
```

* Step 4: Repeat steps 1–3 if you want to create a `copyall` for phonetic spelling. Use an appropriate fieldType. I am using the [Beider-Morse Filter](https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-Beider-MorseFilter).

* Step 5: Add a tie breaker of 1 to get a most-fields functionality. [The docs](https://lucene.apache.org/solr/guide/6_6/the-dismax-query-parser.html#TheDisMaxQueryParser-Thetie_TieBreaker_Parameter) provide a nice explanation.

####**Some ideas**:
* Add [index time boosts](https://wiki.apache.org/solr/SolrRelevancyFAQ#index-time_boosts) for products that are more popular and you want them to rank higher in the search results. You could also do a query time boost, something to the effect of `bf='div(1,popularity)'`.

* Use [function queries](https://lucene.apache.org/solr/guide/6_6/function-queries.html) to customize anything about your the relevancy scoring of your search results.

* Consider the [N-Gram](https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-N-GramFilter) filter for typo tolerance.

* Consider the [Edge-N-Gram](https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-N-GramFilter) filter for autocomplete.

* Consider using the `text_en` `fieldType` for regular English words (it is one of the many `fieldTypes` [which come with Solr](https://lucene.apache.org/solr/guide/6_6/field-types-included-with-solr.html#field-types-included-with-solr)):
```xml
  <fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EnglishPossessiveFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
  </fieldType>
```

##Debugging and Workflow

* Check analysis in the Admin UI for how particular terms are analyzed at index or query time.
* Add a `console.log` in your code to print the url for every query. Set `debugQuery=true` and read the `parsedQeury` and `explain`. All the math fun is lurking in the `explain` (see Relevant Search, Chapter 2).
* After re-configuring the schema, [make sure to delete](https://www.google.com/search?rls=com.microsoft%3Aen-US%3A%7Breferrer%3Asource%3F%7D&rlz=1I7TSNF&ei=ohjaWuuHIKWG5wKm7peoCA&q=delete+all+docs+solr&oq=delete+all+docs+solr&gs_l=psy-ab.3..0.2373.2778.0.2996.4.4.0.0.0.0.81.299.4.4.0....0...1c.1.64.psy-ab..0.4.298...0i7i30k1j0i8i7i30k1j0i13i30k1.0.RCVeecp6NmU) all docs in your index and do a fresh full-import from your database. This can be done in the Admin UI.
* If you need to debug the database import, use the Debug-Mode with verbose output.

![Solr Admin UI](https://thepracticaldev.s3.amazonaws.com/i/87cxvsrmhb6z91soxbpm.png)

##Further Reading

![Relevant Search Book](https://thepracticaldev.s3.amazonaws.com/i/qgtxlxqpp38fnrnrefa6.jpg)

[Relevant Search](www.manning.com%2Fbooks%2Frelevant-search) by Doug Turnbull and John Berryman

The examples in the book use ElasticSearch, but Appendix B provides mappings to Solr. If the book is too long, read chapters 5 & 6. These chapters tackle which strategy to use for matching multi-field (phrase) search with the most relevant results.