galdecoa.com

Using a service worker with streamed partial HTML contents

📅

This is the tale of how and why I copied the ideas of two engineers.

It begins a long, long time ago, roughly a couple of years back. For all the headaches and frustration that web development can cause, it can also bring moments of interest and cautious confidence, of going from seeing what somebody else has done and thinking “cool” to thinking “I think I can do that, too”. That is what happened when I came across Philip Walton’s article “Smaller HTML Payloads with Service Workers”.

The ingenious engineer Walton describes a method to reduce the HTML payload while navigating a site. In essence, the site caches the parts that are common to the documents being served and then, when there is a request for a certain document, a service worker intercepts it, fetches the part of the document without those common partials, and composes the response.

A smaller payload means reduced network traffic but also an earlier First Contentful Paint (FCP), by streaming the content from the cache. Not only that, storing the partial content in the cache allows repeat queries to a certain article to be served without any network connectivity.

Sounds good, let’s do it

Two weeks ago I decided that I wanted to try my hand at it. Walton’s solution uses Workbox, a very convenient set of service worker libraries and tooling. I preferred using vanilla JavaScript, if only to try to understand a bit better what is going on. This is what I came up with.

Generating document partials

In this site, all documents start as a markdown file, then are converted to a static HTML file. Using the main open and close tags, any of such HTML documents can be split into three parts. Taking a simplified version of a static document, with less-than-adequate indentation, the first part of the markup contains the “header”:

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="utf-8" />
	<title>Using service workers with partial HTML contents @ galdecoa.com</title>
	<meta name="description" content="Speeding up a site's navigation and providing offline fallback" />
	<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
	<link rel="stylesheet" href="css/main.min.css" />
	<link rel="icon" href="favicon.ico" sizes="any" />
	<link rel="icon" href="favicon.svg" type="image/svg+xml" />
</head>
<body>
<header>
	<p><a href="index.html">galdecoa.com</a></p>
</header>
<main>

The second part has the article’s contents:

<header>
<h1>Using service workers with partial HTML contents</h1>
</header>
<!-- Contents go here -->
<footer>
<p>© <time>2022</time> Gonzalo Álvarez de Aldecoa</a></p>
</footer>

And the “footer” closes the document:

</main>
<aside>
	<p><a href="index.html" title="Back to home page"><span role="img" aria-label="An icon representing a house"></span></a></p>
	<p><a href="#top" title="Back to top"><span role="img" aria-label="An arrow pointing up"></span></a></p>
</aside>
<script type="module" src="js/main.min.js"></script>
</body>
</html>

I choose to store partial files in their own directory so, for any document.html I also keep partials/document.html. I save partials/_header.html and partials/_footer.html, as well, although these are generated only if there is a change in the common structure.

There is only one minor inconvenience: splitting the HTML file leaves some document specific content out of the main partial. Namely, the meta tags. The template could generate them twice but that would be redundant in the complete HTML file, which is still used. So, after segmenting the markup, those tags can be read from the original file and inserted in the partial, assuming each tag is in a separate line by itself:

#!/bin/bash

SLUG="document";
HTML_FILE_PATH="$SLUG.html"
PARTIAL_HTML_FILE_PATH="partials/$SLUG.html";

# Copy everything between <main> and </main>
sed -n '/<main>/,/<\/main>/{/<main>/d; /<\/main>/d; p}' "$HTML_FILE_PATH" > "$PARTIAL_HTML_FILE_PATH";

# Create temporary file
PARTIAL_TMP_FILE_PATH=$(mktemp --suffix=.html);

# Dump everything in partial before </header> to tmp file
head -n $(($CLOSE_HEADER_TAG_POS-1)) "$PARTIAL_HTML_FILE_PATH" > "$PARTIAL_TMP_FILE_PATH";
# Copy `description` and `keywords` meta tags to tmp file
grep -o '<meta name="\(description\|keywords\)".*$' "$HTML_FILE_PATH" >> "$PARTIAL_TMP_FILE_PATH";
# Dump everything in partial after </header> to tmp file. `printf` prevents 
# having a new line at the end of the partial.
printf "%s" "$(tail -n +$CLOSE_HEADER_TAG_POS "$PARTIAL_HTML_FILE_PATH")" >> "$PARTIAL_TMP_FILE_PATH";

# Substitute partial with tmp file
mv "$PARTIAL_TMP_FILE_PATH" "$PARTIAL_HTML_FILE_PATH";

The previous code is only valid when the manipulated tags have each their own line, and only one line. In this case, partials/document.html will look something like this:

<header>
	<h1>Using service workers with partial HTML contents</h1>
	<meta name="description" content="Speeding up a site's navigation and providing offline fallback" />
</header>
<!-- Contents go here -->
<footer>
	<p>© <time>2022</time> Gonzalo Álvarez de Aldecoa</a></p>
</footer>

If partials/_header.html has been saved at this stage, a couple of sed commands should get rid of the elements that are not common to every document: the content of title and any meta tags that are stored in the main partial.

Finally, I keep a partials/_offline.html that contains a descriptive message of the scenario it might be used in:

<header>
	<h1>Content unavailable</h1>
	<meta name="description" content="Unable to restore a previous copy of the document" />
</header>
<section>
<p>This happens when your browser is offline and a previously stored copy could not be found.</p>
<p>Make sure you have connectivity and <a href="#">reload this page</a>.</p>
</section>
<script type="module">(function(d){d.querySelector("main a").addEventListener("click",function(e){e.preventDefault();d.location.reload();})})(document)</script>

Deploying a service worker

In a bout of unbridled originality, I usually put the service worker in a file called sw.js. Initially, it only handles the install event. Knowing that, at some point, the “header”, “footer”, and “offline” partials are going to be used, they can be cached at that stage.

const cacheName = "v1";
const requiredFilesPaths = [
	"partials/_header.html",
	"partials/_footer.html",
	"partials/_offline.html"
];

/**
 * Handle the service worker's `install` event. Caches required files.
 * @param {ExtendableEvent} installEvent 
 */
async function onInstall( installEvent ) {
	try {
		const cache = await caches.open( cacheName );
		installEvent.waitUntil( cache.addAll( requiredFilesPaths ) );
	} catch { }
}

self.addEventListener( "install", onInstall );

A (very verbose) way to register the service worker is adding, in the main script:

( 
/**
 * Register the service worker.
 * @param {Navigator} navigator 
 * @param {String}	serviceWorkerLabel 
 * @param {Object}	console 
 */
function registerServiceWorker( navigator, serviceWorkerLabel, console ) {
		
	if ( 
		   !( serviceWorkerLabel in navigator ) 
		|| !( navigator[ serviceWorkerLabel ] instanceof ServiceWorkerContainer )
		|| ( typeof navigator[ serviceWorkerLabel ].register !== "function" )
	) { 
		console.info( "Service worker cannot be registered" );
		return; 
	}

	/**
	 * Handle the service worker's registration.
	 * @param {ServiceWorkerRegistration} registration
	 */
	function afterServiceWorkerRegisters( registration ) { 
		console.log( "Service worker registered with scope " + registration.scope ); 
	}

	/**
	 * Handle an error in the service worker's registration.
	 * @param {TypeError} error 
	 */
	function catchServiceWorkerRegisterError( error ) {
		console.warn( "Error registering service worker:" );
		console.warn( error.message );
	}

	navigator[ serviceWorkerLabel ].register( "./sw.js" )
		.then( afterServiceWorkerRegisters )
		.catch( catchServiceWorkerRegisterError )
	;

} 

)( navigator, "serviceWorker", console );

The closure is not required and the benefits of using name aliases in this case are probably negligible, but what is life without whim?

Defining a network and cache strategy

Going back to sw.js, the next thing to do manage the fetch requests. The service worker acts as an intermediary between the browser and the network and, as such, can relay resources in different ways:

  1. Pass-through. Basically, do nothing. For example, images and/or videos might not be cached, so their requests go straight to the network.
  2. Prioritise network. Try to fetch from network if possible and then cache the response, falling back to the cached resource if the network is unavailable.
  3. Prioritise cache. Restore the content from the cache, queuing an asynchronous update of the resource from the network.
  4. Compose. Synthesise a response from different resources. There can be more than one way to build a response but I will stick to using partials.

These possibilities can be enumerated:

/** An enumeration of ways of relaying requests */
const RelayType = Object.freeze( {
	"PassThrough": 1,
	"PrioritiseNetwork": 2,
	"PrioritiseCache": 4,
	"Compose": 8
} );

The values of the enumeration are powers of two so, should it be required, they can be used as flags with bitwise operations. It is not needed in this case, but this is a good a place as any to make a note of it:

const RelayTypeValues = Object.values( RelayType );
/**
 * Determine if `relayType` is valid and has been used to create `flags`.
 * @param   {Number}  relayType A `RelayType` value.
 * @param   {Number}  [flags=0] An OR result of several `RelayType` values.
 * @returns {Boolean}		   True if `relayType` is valid and has been used
 *							  to create `flags`.
 */
function isRelayTypeFlagged( relayType, flags = 0 ) { 
	return (
		   ( RelayTypeValues.indexOf( relayType ) > -1 )
		&& ( ( relayType & flags ) == relayType )
	);
}

When to use one mechanism or another takes into consideration which contents are going to be cached, and how. For example:

  • Resources that are bulky and/or do not change often may be served from the cache, with an asynchronous update. E.g., CSS or JS files.

  • Leaner contents or those that are preferably served as they are on the server may be fetched from the network, even though they will be cached as a fallback measure. E.g., partial HTML documents.

  • Documents that can be composed should be processed accordingly. E.g., HTML requests that are not partials and are in the service worker scope.

  • Any other request ought to be passed on without manipulation.

Some of these criteria can be checked with regular expressions tested against the adequate URL strings, so it is useful to define once both the RegExps and the method that gives a URL object that takes into account the default document:

/** Looks for '/partials/', ending with '.html' */
const isHTMLInPartialsPathRE = new RegExp( /\/partials\/.*\.html$/, "i" );
/** Starts with the SW scope and ends with '.html' */
const isHTMLInScopeRE = new RegExp( 
	"^" + self.registration.scope.replace( "/", "\\/" ) + ".*\\.html$", 
	"i" 
);

/**
 * Get a `URL` object from `request`'s URL string.
 * @param   {Request} request
 * @returns {URL}
 */
function getRequestURL( request ) {
	const requestURLString = request.url.endsWith( "/" )
		? request.url + "index.html"
		: request.url
	;
	const requestURL = new URL( requestURLString );
	return requestURL;
}

Using these, the logic described previously can be translated as:

/**
 * Determine how a request should be handled.
 * @param   {Request} request 
 * @returns {Number}		  A `RelayType` value.
 */
function getRelayType( request ) {

	const destination = request.destination;
	if (
		   ( destination === "style" )
		|| ( destination === "script" )
		|| ( destination === "manifest" )
	) {
		return RelayType.PrioritiseCache;
	}

	const requestURL = getRequestURL( request );
	const requestURLBareString = requestURL.origin + requestURL.pathname;

	if ( isHTMLInPartialsPathRE.test( requestURLBareString ) ) {
		return RelayType.PrioritiseNetwork;
	}

	if (
		   ( request.mode === "navigate" )
		&& ( isHTMLInScopeRE.test( requestURLBareString ) )
	) {
		return RelayType.Compose;
	}

	return RelayType.PassThrough;
	
}

This method only determines how requests ought to be handled, but does nothing to actually dispatch them.

How to fetch and cache, and viceversa

Having created a cache on install, the way to store responses for later use is conceptually simple: bind a method to the fetch event and, when pertinent, fetch any requested resource, put a copy of the response on the cache, and synchronously call the FetchEvent’s respondWith method with a promise that resolves to the response.

There are a couple of things to point out. First, when having to deal with promises and produce synchronous results I currently favour aync/await, but it might not be the right choice always. Second, the previous outline does not cover all the strategies that are being considered, although the basis is the same for all of them.

With this in mind, a basic method to fetch and store a response can look like this:

/**
 * Fetch `request`'s response from the network and put it in the cache before
 * returning it.
 * @param   {Request}   request 
 * @returns {Response?}
 */
async function fetchAndCache( request ) {
	if ( !navigator.onLine ) { return; }
	try {
		const response = await fetch( request );
		const cache = await caches.open( cacheName );
		await cache.put( request, response.clone() );
		return response;
	} catch {}
}

This can be used to implement the different strategies defined in the previous section. A method that handles the “network-first” scenario only requires adding the fallback if there is no response:

/**
 * Try to fetch the requested resource from the network and store it in the
 * cache. If unavailable, fall back to the cached response.
 * @param   {Request}   request 
 * @returns {Response?}
 */
async function fetchResponseAndCacheOrFallback( request ) {
	let response = await fetchAndCache( request );
	if ( response ) { return response; };
	const cache = await caches.open( cacheName );
	response = await cache.match( request );
	return response; 
}

A “cache-first” function can use it, too:

/**
 * Look for a cached response to `request`. When offline, return the result from 
 * querying the cache. When connected, if the resource was not found in the
 * cache, try to fetch it from the network, store it, and return it. Otherwise,
 * return the response after asynchronously calling for an update of the cached 
 * resource.
 * @param   {Request}   request 
 * @returns {Response?}
 */
async function getResponseFromCacheAndUpdate( request ) {
	const cache = await caches.open( cacheName );
	let response = await cache.match( request );
	if ( !navigator.onLine ) { return response; }
	if ( !response ) { 
		response = await fetchAndCache( request ); 
		return response;
	}
	fetchAndCache( request );
	return response;
}

These methods can be used to call FetchEvent.respondWith() for the aforementioned cases, but they can also be used to implement the fourth strategy.

Working with partials

For any request that should have a partial, finding out said resource’s URL is trivial:

/**
 * Get the URL of a document's partial content.
 * @param   {Request} request
 * @returns {URL}
 */
function getPartialURL( request ) {
	const requestURL = getRequestURL( request );
	const partialURL = new URL( requestURL.href.replace( 
		/(\/[^\/]*\.html)/, 
		"/partials$1"
	) );
	return partialURL;
}

One way to respond to one of such requests would be to retrieve partials/_header.html and partials/_header.html from the cache, fetch the partial of the document _offline.html, or retrive if there is not network connectivity, and concatenate them adequately:

/**
 * Compose a document's complete HTML from the pertinent partials.
 * @param   {URL}	 partialURL The URL of a document's partial content.
 * @returns {String} 
 */
async function generateResponseBody( partialURL ) {

	const headerResponse = await getResponseFromCacheAndUpdate(
		new Request( "partials/_header.html" )
	);
	const headerHTML = await headerResponse.text();

	let partialResponse = await fetchResponseAndCacheOrFallback( 
		new Request( partialURL )
	);
	if ( !partialResponse ) {
		partialResponse = await getResponseFromCacheAndUpdate( 
			new Request( "partials/_offline.html" )
		);
	}
	const partialHTML = await partialResponse.text();

	const footerResponse = await getResponseFromCacheAndUpdate(
		new Request( "partials/_footer.html" )
	);
	const footerHTML = await footerResponse.text();

	return headerHTML + partialHTML + footerHTML;

}

The drawback is that fetching the document partial creates a bottleneck. If the network is slow, the whole response is delayed until the partial has been retrieved.

Fortunately, the Response body parameter can be a ReadableStream object, as Jake Archibald described it in 2016. I must confess to not fully understanding how this works precisely, but the logic is simple enough to follow:

/**
 * Queue a readable `stream`'s contents in `controller`.
 * @param   {ReadableStream}				  stream 
 * @param   {ReadableStreamDefaultController} controller 
 * @returns {Promise}
 */
async function pushStream( stream, controller ) {
	const reader = stream.getReader(); // Get a lock on the stream
	return reader.read().then( function process( result ) {
		if ( result.done ) { return };
		// Push the value to the combined stream
		controller.enqueue( result.value );
		// Read more & process
		return reader.read().then( process );
	} );
}

/**
 * Generate a `start` function for a `ReadableStreamUnderlyingSource` that 
 * composes a document from cached partials.
 * @see {@link https://jakearchibald.com/2016/streams-ftw/#crossing-the-streams}
 * @param   {URL}	 partialURL The URL of a document's partial content.
 * @returns {Promise} 
 */
function generateReadableStreamUnderlyingSourceStart( partialURL ) {

	return async function( controller ) {

		const headerResponse = await getResponseFromCacheAndUpdate(
			new Request( "partials/_header.html" )
		);
		await pushStream( headerResponse.body, controller );

		let partialResponse = await fetchResponseAndCacheOrFallback( 
			new Request( partialURL )
		);
		if ( !partialResponse ) {
			partialResponse = await getResponseFromCacheAndUpdate( 
				new Request( "partials/_offline.html" )
			);
		}
		await pushStream( partialResponse.body, controller );

		const footerResponse = await getResponseFromCacheAndUpdate(
			new Request( "partials/_footer.html" )
		);
		await pushStream( footerResponse.body, controller );

		controller.close();

	};
	
}

/**
 * Create a stream for a complete document, generated from its partials.
 * @param   {URL}			partialURL The URL of a document's partial content.
 * @returns {ReadableStream}
 */
function generateResponseBody( partialURL ) {
	const composedResponseStream = new ReadableStream( {
		"start": generateReadableStreamUnderlyingSourceStart( partialURL )
	} );
	return composedResponseStream;
}

Finally, this can be used in a method that composes the response for a document that can be streamed, in this manner:

/**
 * Create a `Response` from the partials stored in `cache` and the document
 * partial associated to `request`'s URL.
 * @param   {Request}	   request 
 * @returns {Response}						  
 */
function composeResponseFromPartials( request ) {
	const partialURL = getPartialURL( request );
	const responseBody = generateResponseBody( partialURL );
	const responseOptions = { headers: { "Content-Type": "text/html" } };
	const composedResponse = new Response( responseBody, responseOptions );
	return composedResponse;
}

All together now

After all this, the fetch event handler will reflect the four planned possibilities:

/**
 * Handle the service worker's `fetch` event.
 * @param   {FetchEvent} fetchEvent 
 */
function onFetch( fetchEvent ) {

	const request = fetchEvent.request;
	const relayType = getRelayType( request );

	if ( relayType === RelayType.PassThrough ) { return; }

	if ( relayType === RelayType.PrioritiseNetwork ) {
		fetchEvent.respondWith( fetchResponseAndCacheOrFallback( request ) );
		return;
	}

	if ( relayType === RelayType.PrioritiseCache ) {
		fetchEvent.respondWith( getResponseFromCacheAndUpdate( request ) );
		return;
	}

	if ( relayType === RelayType.Compose ) {
		fetchEvent.respondWith( composeResponseFromPartials( request ) );
		return;
	}

};

self.addEventListener( "fetch", onFetch );

Finishing touches

Going back to the generation of the partials, partials/_header.html was left with an empty title at the end of the process. In order to show the document title in the browser window, the following line of code can be included anywhere after closing the main element, in a script tag or inside the main script that’s loaded at the end of the body:

document.title = document.querySelector( "main h1" ).innerText;

Now comes an absurd bit. Some people might “audit” this solution with browser tools like Lighthouse and take whatever result comes out at face value. Composed responses do not have the meta tag with the description in the head of the document and it may raise an “issue” regarding SEO best practices.

At the time of writting, search engine crawlers do not fire up the service worker so they, consequently, use the full static HTML files that do have all the elements in their expected positions. Sadly, sometimes, arguing that the tool used to generate the report is not taking that into account is pointless. Some people find it easier to believe without understanding, and will require a “fix”. So, even though there is no technical benefit, here is a solution:

( 
/**
 * This code's only use is to conform to Lighthouse's SEO metric criteria.
 */
function( document, querySelectorFn, metaSelector ) {
	const head = document[ querySelectorFn ]( "head" );
	const meta = document[ querySelectorFn ]( metaSelector );
	if ( head[ querySelectorFn ]( metaSelector ) || !meta ) { return; }
	head.appendChild( meta );
} )( document, "querySelector", "meta[name=description]" );

If it were an issue, I might rather generate JSON-LD file with the required data following schema.org/Article, and use the service worker fetch it and inject the meta tags after recovering _header.html from the cache.

What next?

The first thing to do would be to revise and test all the previous code. Hopefully it is not too buggy but I am certain it is far from perfect.

Then would come improvements on the functionality, that could start with something small. For example, the requiredFilesPaths array could include more files that are going to be used: main.min.css, favicon.ico, favicon.svg, and main.min.js. After that, one might consider revisiting getRelayType so these mostly static contents are served from the cached storage. Also, if they are considered quasi-static, always trying to fetch from the network is probably not the best idea. As a matter of fact, the whole network/cache strategy could be developed further to reduce network traffic as much as possible. All this leads, in my opinion, to consolidating an “offline first” approach.

In this case I would be the one to decide on how to proceed, but most of the time that is not the case. The previous paragraph can be a bit overwhelming, especially if it has to be justified to a project manager/boss/client. These objectives can seem a bit too “technical”. My usual approach in these stances is trying to find out which “non-technical” objective can be aligned with the desired features. If there are none, then I consider that the value I see in something might not be seen as worth the effort.

In this case, I would argue how these improvements can be part of an effort to deploy a Progressive Web App, which would enable the same code to exist as an app in different platforms. A potential cost reduction, through eliminating the time and effort to develop and deploy different versions of the same program, can be very enticing, and rightly so.