Testing

ogygia/testing turns the devtools and the profiler into primitives for your own tests. Your tests click through the app as they always do. ogygia measures what happened, and your test asserts on it.

• page(): the page load as the browser measured it. Vitals, each island’s wake, and the findings of the Page tab.
• hydration(): each island’s hydration status, with the difference for a changed one (the Hydration tab).
• record(actions): what your actions did. Slow interactions and the function behind each, dead and repeated clicks, requests with the server’s own split, long tasks, shifts, errors (the Record tab).
• profile(path): the SSR profiler’s report of a path. Score, forecast, findings, and weight.

With Playwright

ogygia does not import Playwright. You hand it your own test and expect:

import { test as base, expect as baseExpect } from '@playwright/test';
import { withOgygia } from 'ogygia/testing';

export const { test, expect } = withOgygia(base, baseExpect);

import { test, expect } from './fixtures';

test('checkout stays fast', async ({ page, ogygia }) => {
  await page.goto('/cart');

  expect(await ogygia.page()).toMeetBudget({ lcp: 2500, cls: 0.1 });
  expect(await ogygia.hydration()).toHydrateCleanly();

  const session = await ogygia.record(async () => {
    await page.getByRole('button', { name: 'Buy' }).click();
  });
  expect(session).toHaveNoFindings({ severity: 'warn' });
  expect(session).toMeetBudget({ maxInteractionMs: 200, errors: 0 });

  expect(await ogygia.profile('/cart')).toMeetBudget({ score: 80, renderMs: 200 });
});

The server under test needs ogygia({ devtools: true }): the dev server, or a preview build with devtools on (see Devtools in a build). The fixture turns devtools on for your baseURL (the og_devtools cookie), or with no baseURL for the origin of the first page the test opens, so each page is measured from its first load. profile() needs ogygia({ profiler: true }). On a build, give it the profiler’s key:

export default defineConfig({
  use: { baseURL: 'http://127.0.0.1:4173', ogygiaOptions: { profilerKey: process.env.OGYGIA_PROFILER_SECRET } },
});

A recording can span pages: actions that do a full page load (page.goto, a form post, a plain link) keep recording on the next page, and the report covers all of them (session.counts.pages). If the actions leave for a document with no testing API (another site, a page without devtools), the recording is lost and record() says so, with your actions’ own error as its cause.

To keep the page instead, record with blockNavigation. Every document load the actions cause (a reload in a handler, a stray link) is answered 204, so the browser stays where it is, and each one is listed:

const session = await ogygia.record(() => page.getByRole('button', { name: 'Filter' }).click(), { blockNavigation: true });
session.navigations; // [{ url: 'https://…/products', after: 'button.filter "Filter"', island: 'ProductFilter', t: 4120, approx: false }]

after is what started the navigation, seen as it began: the link or form, or the input being handled then (a reload inside a click handler names that button). Where the browser cannot say (no Navigation API, which is Chromium-only) or the site runs an older engine, approx is true and after is only the last input before it. navigations is the list to count and compare. Each entry is also a navigation-blocked finding, for reading, which compare leaves out.

Each island in hydration() has a key for its instance, so a diff can say which carousel broke: Carousel#2, the second island of that name in page order. That number moves when an island is added earlier on the page. For a key that survives that, give the fixture ogygiaOptions: { islandKey: 'data-block-id' }: each island is keyed by that attribute on itself or its nearest ancestor that has it, across shadow roots (an island inside a web component finds the block around the component). Two islands under the same keyed element are id and id#2. Each island also carries t0 and done, when it began to wake and when it could answer, in ms on the page’s clock, so one report tells which step of a session woke it. Session findings carry names, the islands they point at, like page findings.

Changed, but how?

A changed island says what kind of change it was. attributesOnly: true means the same elements and text, with only attributes different, listed in attributes as { tag, name, from, to }. That is often not the app rendering differently: a web component setting its own tabindex or aria-* as it upgrades, or an inline style set and cleared. inputDuringHydration: true means a pointer, a key press or focus landed on the island while it hydrated, so its DOM may have answered that input. Let either through in a matcher, or leave chosen attribute changes out of the report altogether:

expect(await ogygia.hydration()).toHydrateCleanly({ allowAttributeChanges: true, allowInput: true });

// playwright.config.ts: these read `clean`, with what was left out in `ignored`
use: { ogygiaOptions: { ignoreAttributes: [{ tag: /^qds-/, name: /^(tabindex|aria-)/ }, { name: 'style', value: '' }] } }

A rule matches the element’s tag, the attribute’s name and its new value (its old one when it was removed). A string matches exactly, and a RegExp by test.

What measuring costs, and lean testing

Measuring isn’t free. Devtools checks each island’s markup as it hydrates, builds its report, and loads the testing engine. A visitor with devtools off pays none of that. page().self says what it cost: script_ms is the main-thread time of ogygia’s own measuring so far (by phases), and bytes is the decoded size of the testing engine’s files. Subtract both from a CPU or JS-weight measurement.

Lean testing goes further. With ogygiaOptions: { lean: true } (the og_testing=lean cookie beside og_devtools=on, or ?og-testing=lean), the page runs the testing API and nothing else of devtools: no launcher, no dock, no watcher of its timers, no event side-channel in its HTML. It loads what it loads with devtools off, plus the testing engine when a test first asks. Findings that need the server’s events are not there in that mode. Every report says which mode ran (engine.lean), and an engine before version 3 ignores the cookie and runs full devtools.

When the site runs an older ogygia

The reports are built by the page’s testing engine, which ships with the site’s build, not with your test’s ogygia package. A check that measures an older deploy (a baseline) next to a new preview gets older reports from it. Every report says which engine answered (report.engine.version; 3 is current), and the first older one logs one warning per site for the whole run. What the test side knows is filled in: navigations from the loads it held back (with approx: true), names: [] on findings. Instance keys and counts.cls are absent there, and compare then tallies hydration by name and skips the session’s CLS rather than report a false move.

What each engine version detects:

So an island that reads clean on one engine and changed on another differed in that run. Input landing while it hydrated is the usual cause, and version 3 flags it.

The server, over the same actions

record(actions, { server: true }) also records the server while your actions run: the profiler’s CPU across every request they made, their outbound calls, and the requests themselves. It lands on the session as server:

const session = await ogygia.record(() => page.getByRole('button', { name: 'Buy' }).click(), { server: true });

session.server.window.requests; // [{ method: 'POST', path: '/cart/checkout', ms: 412, status: 200 }, …]
expect(session.server).toHaveNoFindings({ severity: 'warn' });
expect(session).toMeetBudget({ serverBusyMs: 300 });

Under the hood this is the profiler’s window: POST /__profiler/window/start, then POST /__profiler/window/stop answers the report. One window runs at a time. It waits a few seconds for a background sample to finish, and stops itself after a minute.

On a serverless deployment, start, the session’s requests and stop can each land on another instance. Give the profiler a store (redis, postgres or SQLite, see the profiler’s storage docs) and it works anyway:

• Start answers a signed window id, and the driver adds it to each of the session’s requests to the site (x-og-window).
• Every instance writes such a request, with its own cpu_ms, to the store before it answers.
• Stop, on any instance, gathers them. window.instances says how many served the session.
• The CPU profile is the starting instance’s, since sampling is per process. A stop that lands elsewhere returns the requests without it.

A window that cannot run never costs you the session: session.server is then { error }.

Profiles for a CI check

• engine.version: a profile names the ogygia that produced it, the same number as the page reports. Its detectors come and go with it, so compare profiles only across one version. One from 0.8.2 has no engine.
• profile(path, { fields: ['requests', 'network', 'findings', 'assets', 'summary'] }): returns only those sections. A full report is hundreds of KB, mostly stacks, hot_functions and drill.
• ?weigh=full: weighs every file the page loads with no time budget (on a serverless host, what the request has left), so assets.totals holds run to run. Each file it could not weigh says why in assets.missed_why: budget, timeout, status 404, file cap or error.
• Document-Policy: js-profiling: a page with devtools switched on is sent it, so the browser’s CPU sampler runs without the app sending it.
• page().route: the SvelteKit route id the server rendered, on a build with devtools on.
• More CPU functions: ogygiaOptions: { cpuFunctions: 50 } keeps more than 15 of page().cpu.fns.
• Session lists: session.failed lists every request that answered an error status. session.by_element gives the slowest interaction on each element the session touched, at 0 for a click too quick for the browser to time.

The matchers

A failing matcher says what missed and where: Clock: changed — −rendered on the server +rendered on the browser.

Budget lines, by report:

checkBudget(report, budget) returns the same lines as data, for your own logic.

Reports in CI

Every report a test takes is attached to its result as ogygia.json. Playwright’s HTML reporter shows it, and the JSON reporter carries it. Two helpers turn two runs (a base branch and a head branch) into a PR comment:

import { compare, toMarkdown } from 'ogygia/testing';

// each run: the parsed `ogygia.json` attachments ({ test, reports }[])
const diff = compare(baseRun, headRun);
console.log(toMarkdown(diff)); // what moved, per test

Per test, the diff carries:

• Findings: new and fixed, by code.
• Profiles: score, server render and JS at start, with the files behind a JS move.
• Hydration: each island instance that stopped or started hydrating cleanly (Carousel#2: clean → failed).
• Sessions: the counts that moved (errors, unexpected shifts, slow interactions, blocked navigations, the session’s CLS).
• Vitals: only those that moved past a run’s noise. Defaults: LCP and FCP 150 ms, TTFB 100 ms, INP 50 ms (each also 10 % of the base), CLS 0.02. Set your own with compare(baseRun, headRun, { noise: { lcp: 300 } }).

Several runs a side

Real pages are noisy, and one run against one run turns that noise into false regressions: a scripted session’s errors, failed requests and interaction shifts depend on the path it happened to take. Give a test several runs a side, { test, runs: [reports, reports, reports] }, and compare splits the signals:

• Load signals (page vitals and findings, hydration, the profile’s score, render time and JS) are compared on the runs’ median. A finding or a broken island counts when most runs have it.
• Session signals (its findings, errors, shifts, slow interactions, blocked navigations) count only when every head run has them and no base run does. A count moves only when every head run is above every base run, or below.

Three runs a side is a good start, for the pages whose first pass shows a candidate.

Other runners and drivers

createOgygia(driver) gives the same four primitives over any driver that can run a function in the page (evaluate) and, for profile(), make a request (get). Puppeteer’s Page fits. The matchers are plain expect.extend matchers:

import { expect } from 'vitest';
import { createOgygia, ogygiaMatchers } from 'ogygia/testing';

expect.extend(ogygiaMatchers);
const ogygia = createOgygia({ evaluate: (fn, arg) => page.evaluate(fn, arg) });

Holding Playwright pages yourself (two browser contexts in one test, say a preview and a baseline) without the fixture? playwrightDriver(page) is the driver the fixture builds, blockNavigation included, and turnOnDevtools(page, baseURL) sets the cookie. onNavigationHeld is called as each load is held back, so you can note which step of your script was running:

import { createOgygia, playwrightDriver, turnOnDevtools } from 'ogygia/testing';

await turnOnDevtools(previewPage, previewURL);
const preview = createOgygia(playwrightDriver(previewPage, { baseURL: previewURL, onNavigationHeld: (n) => log(step, n.url) }));