What does it mean when a performance check fails your pull request and every score on the report is fine?
That happened to me recently on demo.miguelcarino.com, the companion site where I build a working example for each post I write here. Every ready pull request that touches its main app has to pass a Lighthouse gate before I merge. Lighthouse is Google's free page audit, the same one in the Lighthouse panel of Chrome's developer tools. It loads a page, then scores it out of 100 in four categories: performance, accessibility, best practices and search engine optimization (SEO). The gate is a script in my build pipeline that runs it and fails the check when a page misses its bar.
This time the scores were clear. Performance ran 89 to 94 on mobile, and the other three categories were 100. The gate failed anyway, on a line further down: three pages, /performance, /platform and /defer, shipped 335 to 339 KB of JavaScript against a 300 KB budget. Every page with a "View source" panel, the expandable code listings the demo uses, was downloading every snippet in the app from one 1.1 MB generated module. Splitting it into one module per snippet brought those pages to about 182 KB.
None of the four scores had flagged it.
A Lighthouse report holds a lot more than its four circles, and most of what makes it useful sits below them. I start with the small print.
Check the conditions before the numbers
Every report says, near the bottom, how it was measured. Read that first, because a score means little without it.
The default run is mobile. Lighthouse emulates a mid-range phone with a 4x CPU slowdown on a network of 150 ms round trips and about 1.6 Mbps, which its source describes as roughly the 75th percentile of 4G connections. The desktop run assumes a fast connection and no CPU slowdown. The same page can score 100 on desktop and 80 on mobile, and both numbers are honest.
The slowdown is usually simulated. Lighthouse loads the page at full speed, records every request and every task on the main thread, then models how long the same load would take on the slower phone. That's why a report from a fast laptop and one from a busy continuous integration (CI) machine, the shared server that runs a project's checks, can disagree. The model starts from what the machine really did.
All of this is lab data: one synthetic visit under fixed conditions, which makes it good for comparing one build with the next and says little about what visitors on real phones get.
Performance is five metrics, weighted
The performance score blends five timing metrics. In the current version the weights are:
- Total blocking time (TBT), 30%: how long the main thread was too busy to respond to input during load
- Largest contentful paint (LCP), 25%: when the biggest thing on screen finished drawing
- Cumulative layout shift (CLS), 25%: how much the content jumped around as it loaded
- First contentful paint (FCP), 10%: when anything at all first appeared
- Speed index, 10%: how quickly the visible area filled in
Each metric gets its own score first, read off a curve built from real sites in the HTTP Archive, a public project that regularly crawls the web and records how pages are built. A value that only a quarter of those sites beat scores 50. A value only 8% beat scores 90. The curve flattens near the top, so the last points are expensive: Chrome's docs say going from 99 to 100 takes about as much improvement as going from 90 to 94.
The color bands follow the same scale. Red is 0 to 49, orange is 50 to 89 and green is 90 to 100.
So the first question about a performance score is which metric moved it. A TBT problem and an LCP problem have different fixes, and the score alone can't tell them apart. Below the metrics, the report lists insights and diagnostics, such as render-blocking requests, unused JavaScript and a breakdown of how long the LCP element took and why. Those carry no weight in the score, and they explain why a metric came out the way it did.
Accessibility, best practices and SEO are checklists
The other three categories run audits that pass or fail, and the score is a weighted average of the passes.
Accessibility runs a set of rules from axe-core, the open-source accessibility engine from Deque. Each rule's weight comes from axe's impact rating, from 1 for minor up to 10 for critical. There's no partial credit. A page with one unnamed button among 50 named ones gets zero for the button-name audit. Manual checks, like a logical focus order, are listed but not scored. So a 100 means none of the automated checks failed, and nothing more. The demo runs a stricter axe sweep of its own on every route, with the WCAG 2.2 rules up to AAA, because Lighthouse's subset stops well short of that.
Best practices is a mix of security and hygiene. It checks for HTTPS, deprecated browser APIs, third-party cookies, errors logged to the console, a doctype, a character set and images shown at the wrong aspect ratio.
SEO checks whether a search engine can crawl and understand the page: a title, a meta description, a successful status code, descriptive link text, links a crawler can follow, a valid robots.txt, image alt text, hreflang and a canonical URL. One audit outweighs the rest. A page that blocks indexing is weighted so it can't score above 69, however clean everything else is. The structured data check is manual and scores nothing.
The version I run also lists a fifth category, Agentic Browsing, shown as a count of checks passed instead of a score. My gates leave it out.
Why the score moves
Run Lighthouse three times on the same commit and you'll get three performance scores. Lighthouse's own docs list the reasons: network jitter, other work on the machine, ads and A/B tests that change the page and the browser itself. They single out free CI machines as "typically quite volatile."
I measured that before I trusted my gate. On GitHub's shared runners, the same commit of my portfolio, miguelcarino.com, scored anywhere from 87 to 97 on mobile. My first floor was 90, and it would have failed good builds at random.
The other three categories don't drift that way. An image either has alt text or it doesn't. Byte counts hold still too, since the same build sends the same files.
A gate built on what holds still
That's the design of the demo's gate, which you can read in full in the Lighthouse section of the demo. It runs four pages three times each, on mobile and desktop, and judges the run with the median score. Accessibility, best practices and SEO must be 100. Performance fails only below a floor, 85 on mobile and 95 on desktop, and anything under 100 is a warning. A page that misses on timing alone gets three more runs before it fails.
The precise work is done by budgets, in scripts/lighthouse-budget.mjs, trimmed here:
export const RUNS = 3;
export const FORM_FACTORS = {
mobile: {
performance: { fail: 0.85 },
metrics: {
'largest-contentful-paint': 2800,
'cumulative-layout-shift': 0.1,
'total-blocking-time': 500,
},
},
};
export const TRANSFER_BUDGET_KB = {
script: 300,
image: { mobile: 120, desktop: 160 },
document: 40,
stylesheet: 10,
};The gate script reads Lighthouse's network-requests audit, which the report doesn't display but its JSON output holds. It keeps the requests to the site's own origin, adds up the bytes by resource type and compares each total with its budget. This line in scripts/lighthouse-gate.mjs wrote the failures that day:
fixed.push(`${where}: ${label} ${kb(type).toFixed(0)} KB over ${limit} KB`);On the worst page it read script 339 KB over 300 KB. The fixed list holds the failures that don't change between runs, so they fail on the first set of runs. The same script also fails any JPEG or PNG over 20 KB, with a message to add a modern-format source.
A score is a smoke alarm. A budget tells you which room.
Field data, WebPageTest and Lighthouse CI
Field data answers a different question. PageSpeed Insights shows Lighthouse's lab run next to data from the Chrome User Experience Report (CrUX), collected from real Chrome users over the previous 28 days and reported at the 75th percentile. That's the data behind the Core Web Vitals: good means LCP within 2.5 seconds, interaction to next paint (INP) within 200 ms and CLS of 0.1 or less. INP needs real taps and clicks, so a page-load run can't measure it, and TBT is its stand-in. For a live site with traffic, CrUX is the better guide to what people actually get. It can't gate a pull request, though. It covers code that shipped weeks ago, and a page without enough visitors has no data of its own.
WebPageTest runs real browsers from chosen locations and devices, with filmstrips and request waterfalls that make a slow load easy to see. It's the tool I'd reach for to investigate a regression. As a check on every pull request, it adds an outside service and its queue to every build.
Lighthouse CI is Google's packaged version of what my script does. It runs three times per URL by default, asserts on scores, audits and budgets, and can post a status check to GitHub and store reports for comparison. For a team starting out, I'd start there. I wrote my own because the rules I wanted were specific: bytes from the site's own origin only, a re-measure before a timing failure, the format check and analytics blocked so Google's servers don't time my build. It's one script with no extra service, and Lighthouse CI could cover most of it with some configuration.
Gate on the Numbers That Hold Still
When the gate failed that day, its message named three pages, a resource type and a number of kilobytes, and that pointed straight at the cause. A score can't do that. It sums up one simulated visit, and it moves a few points between runs of the same commit. I still read it and let it warn. The build fails on what holds still: the three checklists and the bytes.
