refactor(e2e): verify PerfKit at boundary instead of produce path#3613
Draft
fredericoo wants to merge 3 commits intomainfrom
Draft
refactor(e2e): verify PerfKit at boundary instead of produce path#3613fredericoo wants to merge 3 commits intomainfrom
fredericoo wants to merge 3 commits intomainfrom
Conversation
PerfKit's CDN build (shopify-perf-kit-spa.min.js) contains isObviousBot() which checks navigator.webdriver, window.__playwright__binding__, and window.__pwInitScripts. When any of these are truthy, PerfKit silently suppresses all metric publishing — no sendBeacon, no XHR, no produce requests at all. Playwright always sets navigator.webdriver=true in automation mode, which causes the bot check to trigger. This means verifyPerfKitRequests() can never find any produce requests, causing consent-tracking-accept and privacy-banner-accept tests to fail. The fix uses CDP's Page.addScriptToEvaluateOnNewDocument to override these automation signals before any page scripts (including PerfKit) execute. CDP injection avoids the chicken-and-egg problem that page.addInitScript would cause (it sets __pwInitScripts which PerfKit also detects). Additionally, goto() now awaits the CDP setup promise to ensure the bot signal override is registered before the first navigation. Note: This addresses bot detection only. There may be a secondary issue with web-vitals metrics not being collected in headless mode, which would prevent PerfKit from having data to flush even with bot detection bypassed.
Contributor
|
Oxygen deployed a preview of your
Learn more about Hydrogen's GitHub integration. |
PerfKit uses sendBeacon() which Playwright's page.on('request') doesn't
intercept. Add CDP-level capture for /v1/produce requests so
perfKitProduceRequests is populated even when PerfKit sends via beacon.
This complements the bot signal hiding from the previous commit — with
both changes, PerfKit can publish AND we can observe its requests.
fredericoo
commented
Mar 23, 2026
e2e/fixtures/storefront.ts
Outdated
Comment on lines
+139
to
+154
| source: ` | ||
| Object.defineProperty(navigator, 'webdriver', { | ||
| get: () => false, | ||
| configurable: true, | ||
| }); | ||
| Object.defineProperty(window, '__playwright__binding__', { | ||
| get: () => undefined, | ||
| set: () => {}, | ||
| configurable: true, | ||
| }); | ||
| Object.defineProperty(window, '__pwInitScripts', { | ||
| get: () => undefined, | ||
| set: () => {}, | ||
| configurable: true, | ||
| }); | ||
| `, |
Contributor
Author
There was a problem hiding this comment.
im sorry this is disgusting
Replace verifyPerfKitRequests assertions that depended on PerfKit's internal publish pipeline (bot detection + web-vitals PerformanceObserver + sendBeacon flush) with boundary-level assertions that verify what Hydrogen actually controls: 1. PerfKit script element is present with correct data attributes 2. window.PerfKit is initialized (SPA navigation API available) 3. Tracking cookies (_shopify_y, _shopify_s) match expected values This eliminates 5 independent failure points that were outside our control: bot detection timing, web-vitals callback scheduling in headless Chromium, PerfKit flush heuristics, sendBeacon reliability, and CDP request capture for beacons. Also removes the finalizePerfKitMetrics + waitForTimeout(500) dance from test specs since we no longer need to trigger PerfKit's flush. Reverts the CDP bot-signal-hiding and produce-request-capture code since the refactored assertions don't depend on PerfKit publishing.
fredericoo
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces PerfKit produce request assertions with boundary-level assertions that verify what Hydrogen actually controls, eliminating flaky dependencies on PerfKit's internal publish pipeline.
Problem
PerfKit's CDN build added `isObviousBot()` (Shopify/perf-kit#156, deployed to CDN ~March 18-19) which silently blocks all metric publishing when it detects Playwright automation signals. Even after bypassing bot detection via CDP, tests remained flaky because `web-vitals` PerformanceObserver callbacks don't fire reliably in headless Chromium, leaving PerfKit's internal queue empty.
The previous `verifyPerfKitRequests` depended on 5 independent systems outside our control:
Approach
Test at Hydrogen's boundary instead of PerfKit's produce path. The new `verifyPerfKitRequests` asserts:
This verifies the contract Hydrogen owns — providing correct configuration and tracking data to PerfKit — without depending on PerfKit actually publishing.
Changes
Net: -61 lines (removed more code than added).
Local test results
Test plan