SEO Part1: Help Search Engine Understand Our Website
Tags: seo
Published: 2020-08-13 10:06:41

<blockquote>
This is the second article in this SEO series. In this article, I will explain how to help the search engine understand our website better.
</blockquote>

Search Engine robots analyze our website in a very different way. Although the most important criteria for a good website is still the user experience. But since the search engine needs to automate the analyzing process, it needs to use lots of extra information to understand our website. It's important to make sure that extra information matches our content. Otherwise, the search engine might get confused about the content we want to provide to our users.
Now, let's dive into the details.

## Agenda
  * [robots.txt](#agenda-1)
  * [sitemap](#agenda-2)
  * [URL](#agenda-3)
  * [HTTP status code](#agenda-4)
  * [meta tag](#agenda-5)
  * [link tag](#agenda-6)
  * [Sematic HTML tags](#agenda-7)
  * [HTML tags attributes](#agenda-8)
  * [Structured Data](#agenda-9)
  * [Tools](#agenda-10)

### robots.txt <a name="agenda-1"></a>
`robots.txt` is a guideline for search engine robots, it's used to keep part of our pages private.
If we provide the wrong setting, it might accidentally block the search engine and make our website fail to display on the SERP (Search Engine Results Pages).

We can provide the `robots.txt` file for the entire website.
```
User-agent: *
Disallow:
```

Or we can also use `meta` tag to specify the setting for a specific page.
```html
<meta name="robots" content="noindex">
```

### sitemap <a name="agenda-2"></a>
We can provide the `sitemap` to notice the search engine to index our pages.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> 
  <url>
    <loc>https://www.oahehc.com</loc>
    <priority>0.90</priority>
    <changefreq>always</changefreq>
    <lastmod>2020-07-18T12:59:15.983Z</lastmod>
  </url>
  ...
</urlset>
```

Once we finish our sitemap, we can submit the result to the search engine. Take Google as an example, we can send an HTTP request or handle it on [google search console](https://search.google.com/search-console).
```
http://www.google.com/ping?sitemap=<complete_url_of_sitemap>
```

Another thing worth mentioning is that there is a budget when the search engine crawls a website. So if we have tens of thousands of pages on our website. We can put the most important pages at first or set `priority` probably.


### URL <a name="agenda-3"></a>
The `URL` is another criterion that the search engine might reference. Keeping the `URL` in a clean directory structure not just makes it easier to manage, but also lets the search engine be able to understand.
If it's possible, including keywords and context on the `URL` to make it match our website content.


### HTTP status code <a name="agenda-4"></a>
Now the search engine has our URL, it can send the request and get the content. But before it starts analyzing the content, it will check the `HTTP status code` that our server responds to. Providing the correct status code can prevent the search engine from getting confused. There are two common mistakes:
1. 301 (permanent redirect) vs 302 (temporary redirect)
  - replace old website should use 301
  - if we have different URLs for desktop and mobile websites and redirect our users based on their device. Then we should use 302.
2. 404 page should return 404
  ![404](https://i.imgur.com/mxRDaG7.png)


### meta tag <a name="agenda-5"></a>
In this step, the search engine already has the content of our website. It will check the HTML head first.
![html head](https://i.imgur.com/2ORipsS.png)

### title and description
```html
  <title>title</title>
  <meta name="description" content="description">
```
The search engine reads the title and description to have a basic understanding of our website. So it's important to provide the proper information. For example:
  (a) prevent using stop words
  (b) should match the website content
  (c) include the keywords but not overuse
  (d) multiple tabs page should provide different title/description for each tab

  
### social media
If we want to optimize our website on social media, we should provide proper meta tags. Taking Facebook as an example, we should including that information inside the head tag.
```html
  <meta property="og:url" content="https://www.oahehc.com.tw" />
  <meta property="og:type" content="article" />
  <meta property="og:title" content="title" />
  <meta property="og:description" content="description" />
  <meta property="og:image" content="https://cdn.oahehc.com.tw/logo.png" />
```
  And don't forget to use their tool to make sure everything is working as we expected.    
  [FB Sharing Debugger](https://developers.facebook.com/tools/debug/)


### link tag <a name="agenda-6"></a>
It's important to prevent duplicate content on our website because this makes the search engine hard to decide which page they should provide to their users. Therefore, another information we might need to provide in the head tag is `link`.  
If we have different URLs for desktop and mobile websites. Then we can provide `canonical link` & `alternate link` to tell the search engine to treat them as the same page.

```html
<link rel="canonical" href="https://www.oahehc.com.tw">
<link rel="alternate" href="https://m.oahehc.com.tw" media="only screen and (max-width: 640px)">
```


### Sematic HTML tags <a name="agenda-7"></a>
```html
<div>
  <div>1. xxx</div>
  <div>2. ooo</div>
  <div>3. ...</div>
</div>

<ol>
  <li>xxx</li>
  <li>ooo</li>
  <li>...</li>
</ol>
```
With proper CSS styling, the above two examples will look exactly the same by the human eye. But it's quite different in the search engine's aspect.
There are a few basic guidelines about how to choose HTML tags properly:

1. Using `h1`~`h6` for heading, and we should only have one `h1` tag for each page
2. Structure the page by semantic elements like `header`, `main`, `aside`, `footer`, `section`, `nav`, ...
3. Using `table/th/td/td` for table
4. Using `ul/ol/li` for list
5. Don't mix up `button` & `a`

Use semantic HTML tags not just help the search engine understand our webpage better, sometime we might gain an extra bonus from that.
Google search provides [featured snippet](https://support.google.com/websearch/answer/9351707) if there is a proper answer for the search question. If our content is summarized in the table or list, then it will have a higher chance to be choosing as one of them.

![featured snippet](https://i.imgur.com/XINxVUR.png)


### HTML tags attributes <a name="agenda-8"></a>
Except choose proper HTML tags, sometime we might have to provide extra attributes to add more information.  
For example, add `alt` for `img` tag to explain the image not just help the search engine understand better, but also make our website more friendly for people who use a screen reader to browse our website.
```html
<img src="https://cdn.oahehc.com.tw/dog.png" alt="dog" />
```

When we use an image as the content for a hyperlink, add `title` attribute to provide extra information is also a good practice.
```html
<a href="./dog" title="dog list">
  <img src="https://cdn.oahehc.com.tw/dog.png" alt="dog" />
</a>
```


### Structured Data <a name="agenda-9"></a>
If we want to provide more information for the search engine, we can add `structured-data` on our website.
There are a few different categories of `structured-data` that Google will display in richer features in search results.
![Bloomberg](https://i.imgur.com/LAqMMP3.png)
We can check this article to know more detail - [Explore the search gallery](https://developers.google.com/search/docs/guides/search-gallery).

To add `structured-data` on our website, we can directly add into the HTML tags:
```html
<ol itemscope itemtype="http://schema.org/BreadcrumbList">
  <li itemprop="itemListElement" itemscope
      itemtype="http://schema.org/ListItem">
    <a itemprop="item" href="https://example.com/dresses">
    <span itemprop="name">Dresses</span></a>
    <meta itemprop="position" content="1" />
  </li>
  <li itemprop="itemListElement" itemscope
      itemtype="http://schema.org/ListItem">
    <a itemprop="item" href="https://example.com/dresses/real">
    <span itemprop="name">Real Dresses</span></a>
    <meta itemprop="position" content="2" />
  </li>
</ol>
```

Or we can create a script tag and set all the information in json format:
```html
<script type="application/ld+json">
{
 "@context": "http://schema.org",
 "@type": "BreadcrumbList",
 "itemListElement":
 [
  {
   "@type": "ListItem",
   "position": 1,
   "item":
   {
    "@id": "https://example.com/dresses",
    "name": "Dresses"
    }
  },
  {
   "@type": "ListItem",
  "position": 2,
  "item":
   {
     "@id": "https://example.com/dresses/real",
     "name": "Real Dresses"
   }
  }
 ]
}
</script>
```

Once we finish the structured data, don't forget to test it and make sure the format is correct - [Structured Data Testing Tool](https://search.google.com/structured-data/testing-tool/u/0/).


### Tools <a name="agenda-10"></a>
You might feel overwhelmed after seeing so many extra errands we have to deal with. Luckily, there are tools that can help us.
- [ESLint](https://eslint.org/)
  ESlint not just can help us prevent syntax error, adding a proper extension, it can also point out the missing attribute on our HTML tags.
  ![eslint](https://i.imgur.com/X2fcK6t.png)

- lighthouse
  `lighthouse` is a built-in feature in Chrome, we can use it to identify common problems on our website. Now we can focus on `Accessibility` & `SEO`. We just need to click `Generate report`, then `lighthouse` will let us know how to fix the problems on our website.
  ![lighthouse](https://i.imgur.com/xbrnF6A.png)

- [google search console](https://search.google.com/search-console/about)
  Once we publish our website, definitely start using `google search console`. It doesn't just provide the information to measure search traffic. Moreover, it points out all the problems that Google finds out when they are trying to crawl our website. Fixing those problems can make the search engine understand our website better.
  ![google search console](https://i.imgur.com/ir9hjeU.png)

---

## Conclusion
Now we know how to make the search engine easier to understand our website. In the next article, I will focus on the most important question - how to improve the user experience for our real users - [SEO Part2: Improve User Experience & SEO](https://dev.to/oahehc/seo-part2-improve-user-experience-seo-11cd)

## Reference
  * [Introduction to robots.txt](https://support.google.com/webmasters/answer/6062608)
  * [Learn about sitemaps](https://support.google.com/webmasters/answer/156184)
  * [HTTP response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
  * [HTML Semantic Elements](https://www.w3schools.com/html/html5_semantic_elements.asp)
  * [How Google’s featured snippets work](https://support.google.com/websearch/answer/9351707)
  * [Explore the search gallery](https://developers.google.com/search/docs/guides/search-gallery)
  * [Structured Data Testing Tool](https://search.google.com/structured-data/testing-tool/u/0/)
  * [ESLint](https://eslint.org/)
  * [google search console](https://search.google.com/search-console)
  * Cover Image by <a href="https://pixabay.com/users/Tumisu-148124/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=896175">Tumisu</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=896175">Pixabay</a>