My Ideal Service
Published: 2021-04-17 19:33:34

---
title: My Ideal Service
published: true
date: 2021-04-15 20:58:00 UTC
tags: 
canonical_url: https://kevincox.ca/2021/04/15/my-ideal-service/
---

<main><p>This is a set of features that I like to see from services, whether they are personal services to run on a home server or production services powering a business. The common theme of this list is providing enough hooks and documentation so that a lot of standard requirements can be handled by a layer in front of the service. This ensures that these key requirements are handled consistency by trusted tools and removes the requirement to re-implement them in every service.</p>

<h2 id="use-http"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#use-http">Use HTTP</a></h2>

<p>Where possible use HTTP. There are times for custom protocols but HTTP is widely understood, has great tooling, is accessible from the web and is quite performant (with HTTP 2 and 3). Unless there is a really strong justification for a custom protocol it is best to use HTTP. A lot of items later on this list assume HTTP, however if a custom protocol is used these features are still wanted, they will just need to be implemented in the service itself as they can’t be implemented in the reverse-proxy.</p>

<h2 id="listen-on-unix-sockets"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#listen-on-unix-sockets">Listen on UNIX Sockets</a></h2>

<p>UNIX sockets support filesystem permissions, allowing me to restrict which users and groups can access the service. IP sockets can be accessed by all services on the system. Even on a single-user system where all services are trusted this goes against <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">the principle of least privilege</a> and raises the impact if any service is compromised. This is especially important when doing authentication in a reverse-proxy as local services can bypass it by accessing localhost directly.</p>

<p>I would even argue that most services shouldn’t except to be exposed directly to the internet anyways, it is better to expect a local reverse-proxy to handle rate limiting, access control and logging as well as providing a full-featured battle-hardened and performant HTTP implementation before sending clean HTTP to the service.</p>

<p>NGINX example:</p>

<div><pre><code><span>location</span> <span>/</span> <span>{</span>
	<span>proxy_pass</span> <span>http://unix:/run/ipfs-api.sock:</span><span>;</span>
<span>}</span>
</code></pre></div>
<h2 id="socket-passing"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#socket-passing">Socket Passing</a></h2>

<p>Services should support accepting their socket from the caller instead of binding the socket themselves. This provides a number of advantages.</p>

<ul>
  <li>Arbitrary socket options can be set without any support from the service. (For example see the <a href="https://www.freedesktop.org/software/systemd/man/systemd.socket.html#Options">socket options supported by systemd</a>, not all apply to HTTP services but many do)</li>
  <li>Support for hitless restart as the socket will buffer connections while the service is restarting.</li>
  <li>Support for parallel startup of dependent services.</li>
  <li>Support for lazily starting services on first use.</li>
  <li>Services can exit on inactivity to save resources and they will be restarted for the next connection.</li>
  <li>You can listen on both IP and <a href="https://kevincox.ca/2021/04/15/my-ideal-service/#listen-on-unix-sockets">UNIX sockets</a> with no change to server code.</li>
  <li>Capabilities to bind ports is not required, allowing running the app as a non-root user without any special capabilities.</li>
</ul>

<p>If the service only needs one socket (which is the most common case) the simpler <a href="https://en.wikipedia.org/wiki/Inetd">inetd protocol</a> can be used. The <a href="https://www.freedesktop.org/software/systemd/man/sd_listen_fds.html">systemd protocol</a> can also be used and supports any number of <a href="https://www.freedesktop.org/software/systemd/man/systemd.socket.html#FileDescriptorName=">named</a> sockets.</p>

<p>Example of socket-passing via systemd.</p>

<div><pre><code><span>[Socket]</span>
<span>ListenStream</span><span>=</span><span>/run/ipfs-api.sock</span>
<span># Only users in the ipfs group may access the IPFS API.
</span><span>SocketGroup</span><span>=</span><span>ipfs</span>
<span>SocketMode</span><span>=</span><span>660</span>
</code></pre></div>
<h2 id="header-based-auth"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#header-based-auth">Header-based Auth</a></h2>

<p>It is unreasonable for every service to implement every authentication method that a user might want, that would be an <a href="https://shendrickson.com/nxm/">NxM</a> problem. Instead it should support relying on a reverse-proxy to provide the authentication information.</p>

<p>The simplest option is passing the username in a request header. This can be set by the reverse-proxy once it has authenticated the user or cleared if the user is not authenticated.</p>

<p>It is also convenient if users are automatically created when a new username is received, however assigning default permissions can be difficult so not all services can offer this without significant effort.</p>

<p>NGINX example:</p>

<div><pre><code><span>auth_basic</span> <span>"Restricted</span> <span>Access"</span><span>;</span>
<span>auth_basic_user_file</span> <span>"users.htaccess"</span><span>;</span>
<span>proxy_set_header</span> <span>X-User</span> <span>$remote_user</span><span>;</span>
</code></pre></div>
<p>For services that support public access a login URL should be clearly documented such that that URL can be intercepted by the reverse-proxy and the auth provided via header. For services that are entirely private (either by design or by administrator choice) the reverse-proxy can be be relied upon to intercept all URLs.</p>

<p>For API-based access it is difficult for the reverse-proxy to intercept the request and present login info. In that case the best approach is allowing users to generate auth-tokens. These auth tokens then should be passed in a uniform way across your API. Good options are a <code>Authorization</code> header containing either a <code>Bearer</code> token or a <code>Basic</code> username and token. Personally I prefer <code>Basic</code> as it allows proxies to determine the username for rate-limiting or blocking users if they are removed. Another option is passing the token in a URL query parameter but this is not recomended because URLs are generally logged and auth tokens shouldn’t be! However no matter how the tokens are passed they should be validated by the service and <strong>the request must be rejected</strong> if the auth token is invalid or the provided username doesn’t match the token. This should happen even for URLs where the auth token isn’t needed! Otherwise the reverse-proxy can’t assume that requests with a token are authenticated.</p>

<h2 id="proper-cache-headers"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#proper-cache-headers">Proper Cache Headers</a></h2>

<p>Many services don’t set reasonable cache headers by default. Some services have an “operating guide” describing how to configure a reverse-proxy to apply the correct headers, but why make every user do this when you can just do it once in the service itself. Not only is this configuration service-specific but the service will always have more accurate information about what can be cached and for how long, not only can this be used by clients but it can also be used to apply caching in the reverse-proxy if desired. Services should also generate and check <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag">ETag</a>s and <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified">Last-Modified</a> headers when it makes sense.</p>

<p>Services <strong>should not</strong> perform their own caching (or should provide a way to turn it off) and rely on a reverse-proxy. This makes it easier to manage the cache across services if desired.</p>

<p>One complication here is controlling local caching vs browser caching. In some cases resources are cacheable, but cheap enough to generate that it is not worth caching on the same machine. NGINX uses the <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid"><code>X-Accel-Expires</code></a> header to override its caching but not all reverse-proxies have a similar option. The best option here is to simply provide a list of URLs that administrators should exclude from local caching.</p>

<h2 id="structured-logging"><a href="https://kevincox.ca/2021/04/15/my-ideal-service/#structured-logging">Structured Logging</a></h2>

<p>Structured logging is great because it is easy to monitor and alert on a service. Good structured logging can even replace the need to emit metrics for a large number of use cases. Some tips for structured logging are below.</p>

<ul>
  <li>Use a common format, for example JSON lines to stdout. Supporting multiple formats is always a plus.</li>
  <li>Give important messages a stable message ID. Generally using a random number like a UUID is a good option. This way alerts and monitoring will continue to work even if the message text is changed between versions. Consider the removal of messages or fields to be a breaking change and avoid if at all possible.</li>
</ul>