Index my bio
Tags: softwareengineering, elasticsearch, python
Published: 2018-02-10 22:35:57

---
title: "Index my bio"
published: true
tags: software-engineering,elasticsearch,python
canonical_url: 
---

I want to make life of job-changers and human-researchers easier. &nbsp;I want to make easily searchable projects view. &nbsp;I want to have it on Elasticsearch, cause I want to proof, you can easily do something small with it, so it’s not only for very BIG data projects.

Following piece of code available is on [Github](https://github.com/lukaszkuczynski/search_my_bio).

## Run on your Linux machine

It’s easy as piece of cake. &nbsp;You only need `docker` and it’s younger brother `docker-compose`. &nbsp;Then just this little script allows me to run whole the machine.

```
version: "2"
services:
  kibana:
    image: kibana
    ports:
    - 5601:5601
  elasticsearch:
    image: elasticsearch
```

## Write in YAML, index to ES

Thinking of domain right now, what do I need when thinking of my Bio? &nbsp;I will simplify it to `project` object. &nbsp;This could be exemplary bio of someone..

```
Projects:
  Commercial:
    Truck browser:
      started at: 2013-01
      finished at: 2013-06
      tasks:
      - writing code
      - testing my code
      learned:
      - JDK8
      - Junit Test version 5 is cool
      challenges:
      - dealing with frontside, CSS :(
      - hard to deploy on prod server
      technologies:
      - Spring 3.2
      - JDK8
      - handmade JS and CSS
      measure of success:
      - 90% of test coverage
      - 2 hundred of happy users in Polish workshops
  Private:
    Mini REST service for my CD Collection:
      started at: 2013-02
# still working on it
      tasks:
      - care about whole app
      learned:
      - NodeJS + Express = fast web or Rest app written in JS!
      challenges:
      - which library on npm to choose?!
      technologies:
      - NodeJS
      - Express
      - javascript
      measure of success:
      - REST app in 6 days
```

How to index it? First, make a JSON out of it.

## Python in data wranigling and ES I/O

Data wrangiling or conversion is so easy. &nbsp;Let’s make JSON out of Yaml notation here. &nbsp;With [yaml library](http://pyyaml.org/wiki/PyYAMLDocumentation) it’s easy.

```
import yaml
fname = "projects.yml"
with open(fname) as f:
    doc = yaml.load(f)
```

Then let’s proof we can connect to ElasticSearch. &nbsp;After importing official library, we can see this line

```
luk@luk-UX410UAK:~/prj/searchmybio$ python indexmyprojects.py
--Listing commercial projects--
Truck browser
--Listing side-projects--
Mini REST service for my CD Collection
{'name': 'HmNOtOU', 'cluster_name': 'elasticsearch', 'cluster_uuid': 'tRdhrV3gR0OnUfZMkPrpqQ', 'version': {'number': '5.5.0', 'build_hash': '260387d', 'build_date': '2017-06-30T23:16:05.735Z', 'build_snapshot': False, 'lucene_version': '6.6.0'}, 'tagline': 'You Know, for Search'}
```

Good, connection is done!

## Index time!

Before we’ll start to index docs, we have to arrange place for them. &nbsp;Any schema? &nbsp;No.. Elasticsearch can do it for us. &nbsp;We just have to create index, easily with some py script. &nbsp;After that..

```
res = es.indices.create("searchmybio_luke")
```

..we can query ES in Kibana tools. So let’s now fill it! After few refactors we are ready to insert or PUT docs to their place. &nbsp;Indexing from within a script is done by this one-liner: All docs are&nbsp;there. &nbsp;Any proof? &nbsp;Query it:

```
GET searchmybio_luke/project/_search
{
  "query": {
    "match": {
      "technologies": "nodejs"  
    }
  }
}
```

Even you can write test and run it against this index.

```
query_find_nodejs = {
    "query": {
        "match": {
            "technologies": "nodejs"
        }
    }
}
res = self.es.search(index='searchmybio_luke', doc_type='project', body=query_find_nodejs)
hits = res['hits']['hits']
self.assertEquals(len(hits), 1)
```

That’s all. &nbsp;In less then one hour we were able to run Elasticsearch on our machine and index docs from yaml directly to index.

Here is screenshot from my Kibana

![](https://i0.wp.com/lukcreates.pl/wp-content/uploads/2017/07/Selection_001.png?resize=660%2C769)