WORDS

Static website search

2019-05-26

A couple of months back, I added site search to this blog, created as a static website. Such a website consists of just HTML, CSS, Javascript and media, with no backend code or database for fetching data that the browser requests. Instead the browser simply requests files on disk, which the server returns without any further processing.

Search indexes

When searching for some text string within a larger text source, you could employ a number of different algorithms. I'm going to mention two naive methods and a better method.

Complete search. The easiest way is to simply go through all of the text and return those items that contain the search term. If the source text isn't too large, it might be efficient enough.
Partial search. A slightly quicker way might be to extract only the key terms that users should be able to search for. That way you might be able to enable searching through a larger source text.
Efficient search index. The previous methods work for small amounts of source material, but if you have more than a few megabytes, it might be worth it to use a ready-made solution. For instance, you could use your database's built-in full-text search, or perhaps even something like Elastic Search or an external service like DuckDuckGo or Algolia, depending on your use case.

Search without backend code?

You don't necessarily need a database in order to search. You will, however, need some kind of search index, but you could transfer it to the client browser. Then you can let the client perform the search and display the results without consulting the server.

This works wonderfully on smaller data-sets. For this blog, I've opted to do a partial search. For each page, I've manually entered a generous amount of keywords (that I've called topics). You can see a complete list of pages and topics on the site map. Only when I have over 1,000 pages will the search index be too large to transfer to the client (I'm setting a max size of 500 kB).

Show me the code

This website uses Hugo, a static website generator. Each blog post is written in Markdown. When navigating to a page, a script is loaded that displays the search box, and if the user types something, the search index is fetched. Then the script filters the pages which match the search term, displaying the result below the search box.

Here's a slightly simplified version of the code as of May 2019, used with Hugo 0.55.

Markdown frontmatter

Each blog post is represented as its own Markdown file. It begins with some metadata called frontmatter.

static-website-search.md

---
title: "Static website search"
date: 2019-05-26
languageCode: "en-us"
tags: ["English", "Programming", "Meta"]
topics:
  [
    "Static websites",
    "Search",
    "Search indexes",
    "Full-text search",
    "Backend",
    "...",
  ]
---

A couple of months back...

Search index template

The search index is generated using a Hugo template. The file extension is .json since that the file type to be generated.

themes/ananke-custom/layouts/home.searchindex.json

{{- $pages := .Site.Pages -}}
{{- $result := slice -}}
{{- range $pages -}}
  {{- $p := .Params -}}
  {{- $page := dict "title" .Title "url" .RelPermalink "tags" $p.Tags "lang" $p.LanguageCode "topics" $p.Topics -}}
  {{- $result = $result | append $page -}}
{{- end -}}
{{- $result | jsonify -}}

Config file

In order to enable search index creation, you need to add it as an output format in your site config.

config.toml

# ...

[outputFormats.SearchIndex]
mediaType = "application/json"
baseName = "search"

[outputs]
home = ["HTML", "SearchIndex"]

Search index JSON

Using the search index template and the config, a search index JSON file will be generated. Here's what that can look like:

/search.json

[
  //...
  {
    "lang": "en-us",
    "tags": ["English", "Programming", "Meta"],
    "title": "Static website search",
    "topics": ["Static websites", "Search", "Search indexes"],
    "url": "/words/static-website-search/"
  }
  //...
]

When all of the infrastructure is in place, we can create a search widget. Here, I've used modern Javascript, but that's okay since I don't need to support Internet Explorer.

themes/ananke-custom/static/dist/js/search.js

(() => {
  function $(selector) {
    return document.querySelector(selector);
  }

  function elem(tag, attrs, ...inner) {
    const e = document.createElement(tag);

    if (attrs) {
      for (const key in attrs) {
        e.setAttribute(key, attrs[key]);
      }
    }

    if (inner) {
      e.append(...inner);
    }

    return e;
  }

  const menu = $("#main-menu");
  const searchItem = menu.firstElementChild.cloneNode();
  searchItem.innerHTML = `
    <form class="f5 relative">
      <div class="flex bg-black-70 white ba br1 b--gray overflow-hidden">
        <input
          type="search"
          placeholder="Search"
          accesskey="f"
          class="bg-transparent ba bw0 pl2 pv1 color-inherit"
        /><button title="Search" class="bg-transparent bw0" tabindex="-1">
          🔍
        </button>
      </div>
      <ul id="search-results" class="absolute w-100 bg-near-white ba bt-0 b--gray pv1 ph0 ma0 list f6" hidden></ul>
    </form>
    <style>
      #search-results a {
        color: inherit;
        display: block;
        padding: 0.25rem 0.5rem;
        text-decoration: none;
      }

      #search-results a:hover,
      #search-results a:focus,
      #search-results .active a {
        background: #06f;
        color: #fff;
      }
    </style>
  `;

  const form = searchItem.firstElementChild;
  const input = form.querySelector("input");
  const results = form.querySelector("ul");

  form.onsubmit = async e => {
    e.preventDefault();

    const target = results.querySelector(".active a");
    if (target) {
      location.href = target.href;
    } else {
      search();
    }
  };

  window.addEventListener("click", e => {
    if (!form.contains(e.target)) {
      closeSearch();
    }
  });

  input.oninput = search;
  input.onfocus = loadSearchIndex;
  input.onkeydown = e => {
    if (e.key === "Escape") {
      closeSearch();
    } else if (e.key === "ArrowDown" || e.key === "ArrowUp") {
      e.preventDefault();
      const curActive = results.querySelector(".active");
      if (curActive) {
        curActive.classList.remove("active");
      }

      const nextActive =
        (e.key === "ArrowDown" &&
          (curActive
            ? curActive.nextElementSibling
            : results.firstElementChild)) ||
        (e.key === "ArrowUp" &&
          (curActive
            ? curActive.previousElementSibling
            : results.lastElementChild));

      if (nextActive) {
        nextActive.classList.add("active");
      }
    }
  };

  menu.appendChild(searchItem);

  async function search() {
    const term = input.value.toLowerCase();
    if (!term) {
      closeSearch();
      return;
    }

    const index = await loadSearchIndex();
    const matches = index.filter(x => x.search.includes(term)).slice(0, 10);

    const elems = matches.map((x, i) =>
      elem(
        "li",
        i === 0 ? { class: "active" } : undefined,
        elem(
          "a",
          {
            href: x.url,
            tabindex: -1
          },
          x.title
        )
      )
    );

    if (elems.length) {
      results.textContent = "";
      results.append(...elems);
    } else {
      results.innerHTML = `<li class="i gray ph2">No results</li>`;
    }
    results.hidden = false;
  }

  function closeSearch() {
    results.hidden = true;
  }

  let searchIndex;
  async function loadSearchIndex() {
    if (!searchIndex) {
      searchIndex = getSearchIndex();

      try {
        await searchIndex;
      } catch (e) {
        alert("Failed to load search index");
        console.error(e);
        searchItem.remove();
      }
    }

    return searchIndex;
  }

  async function getSearchIndex() {
    const response = await fetch("/search.json");
    const obj = await response.json();

    for (const item of obj) {
      item.search = [item.title, ...(item.topics || [])]
        .filter(x => x)
        .map(x => x.toLowerCase())
        .join(" ");
    }

    return obj;
  }
})();