A One-Second Static Check for E2E Selector Drift

Every E2E test run has a price tag, and it's paid in feedback-loop minutes. Here's a classic example. Someone renames a data-test attribute during a refactor, not knowing ten specs reference it. Best case: a 15-minute (or more) pre-merge run fails near the end and tells them. Worst case: the rename slips through the merge and the nightly job fails at 3:30am. The next morning, someone burns an hour establishing that the "failure" is not a bug, not an outage, not flake, just a selector that was renamed three days ago. Either way, you paid minutes or hours of test execution to learn something that was knowable in one second, before a browser ever launched. Our team decided to finally stop paying that tax. The solution? I wrote a ~500-line Node script that cross-checks every selector our specs reference against every selector our app source can produce, and fails the offending PR in about a second. No browser, no app boot, no AI. We built it for our Cypress suite (Vue 3 frontend, ~200 spec files, ~6,300 selector call sites), but nothing about the idea is Cypress-specific. It works for Playwright, WebdriverIO, or anything else because the thing being checked isn't the test framework. It's a naming contract. And here's the best part: on its first run against our fully green test suite, it found eight real automation defects. Selector Drift A selector like cy.getByDataTest('orders-table') is a reference to something defined elsewhere, exactly like a function call. When someone deletes the function, the compiler tells you immediately. In contrast, when someone deletes an attribute, the tooling says nothing; instead you find out whenever a test that happens to touch that element happens to run. When discussing this issue, we explicitly did not want runtime "self-healing" selectors, where AI guesses which element you probably meant. We believe self-healing can be useful but typically hides drift, so we wanted the opposite: fail the PR that caused the drift, loudly and deterministically. We wanted to understand why and fix it on our terms. Prerequisite: One Convention This static check is only possible because of a convention we already enforced: every element access in every spec goes through a single helper call, cy.getByDataTest('...'), a one-line wrapper around our framework's selector call (in Cypress that's cy.get('[data-test="..."]'), registered as a custom command). Then on the app side, components expose exactly one attribute (data-test) as the test hook. No raw CSS selectors, no XPath, no text matching for element targeting. If your suite selects elements ad hoc, CSS classes here, test-ids there, text content elsewhere, there is no contract to check. The convention is the contract; the script just enforces it. If you don't have this discipline yet, adopting it is step zero, and it pays off even if you never build the checker. I know everyone has their own opinion on which selectors are "best". This isn't an argument that data-test attributes are perfect, just that in order to have a deterministic static check, you need a deterministic pattern. How It Works Strip away the details and the whole idea is comparing two lists. List one: every selector name the tests use. List two: every selector name the app can provide. Any name a test uses that the app can't provide is drift, a reference pointing at nothing. That's why the script is built in three phases: build list one, build list two, compare them. Phase 1: Collect Every Selector the Tests Use. We need to find every cy.getByDataTest(...) call across every spec file and figure out what string it's asking for. Your first instinct might be a text search, but grep can't reliably tell a real call from a comment, a string that merely mentions the name, or a selector assembled from pieces. So we use the tool that already understands the code perfectly: the TypeScript compiler itself. If your project is written in TypeScript, the compiler is already installed, it's the dev dependency literally named typescript in your package.json, and it doubles as a library. Your script can require("typescript"), hand it a file, and ask it to parse it. Zero new dependencies. Parsing produces an AST (abstract syntax tree): instead of a wall of text, every piece of syntax becomes a labeled node, "this node is a function call named getByDataTest; its first argument is a string literal whose value is orders-table." Walking that tree means you find every call with certainty, regardless of formatting, and inspect each argument as structured data instead of pattern-matching on characters. With the tree in hand, each call's argument resolves one of three ways: A plain string — cy.getByDataTest('orders-table'). Exact name; goes on the list as-is. A template with dynamic parts — cy.getByDataTest(`${bid.id}-bid-card`). We can't know what bid.id will be at runtime, but we do know the name ends in -bid-card. The dynamic span becomes a wildcard and the pattern *-bid-card goes on the list. The static skeleton is the contract. Fully dynamic — cy.getByDataTest(someVariable). Nothing statically knowable, so we skip it…but we count every skip and print the total on each run, so the blind spot is never invisible. For us: 16 calls, about 0.25%. Pretty negligible overall, which is nice. Phase 2: Collect Every Selector the App Can Provide. Now the other list: every data-test value the app source is capable of putting on the page. This side doesn't need the compiler, attribute values sit in plain sight in the markup, so a couple of regexes over the source files do it. Static attributes like data-test="orders-table" go on the list as exact names. Dynamically bound attributes like :data-test="`${bid.id}-bid-card`" get the same wildcard treatment as the test side and go on as patterns; skeletons will match skeletons. The genuinely hard part on this side is composition: wrapper components that assemble names at runtime, like a shared Button component that appends -button to whatever base name it's given, or sometimes a base derived from an i18n label. Every judgment call there got resolved in the same direction: when in doubt, make the app's list more generous. A too-generous list can let some exotic drift through; a too-strict list fails PRs over selectors the app genuinely renders. Phase 3: Compare. For every name from phase 1, ask: could anything on the phase-2 list produce it? An exact name needs an exact entry or a pattern that could generate it (42-bid-card is producible by *-bid-card). A pattern matches a pattern when their fixed fragments line up. Every miss is reported with the spec file, line number, original expression, and the resolved name, and the script exits non-zero. Total runtime for us: about one second for ~6,300 call sites checked against ~1,600 source files. Show Me the Code Below is a stripped-down but fully working version of the checker, about 120 lines, one dependency: that same typescript package from Phase 1, which any TypeScript project already has installed. It's deliberately minimal so the shape of the idea stays visible, and the comments explain what's happening and where you'd adapt it to your own helper name, attribute, directories, and framework syntax. Our production version layers on the extras mentioned above (factory inlining, constant resolution, the more-generous handling of composed names), but the architecture is identical, and even this minimal version genuinely catches drift. The three code excerpts below map one-to-one onto the three phases: resolve what the tests use, harvest what the app provides, compare. The heart of Phase 1 is resolving each helper argument into a checkable string, wildcarding whatever is only knowable at runtime: // Resolve a helper-call argument to a checkable string. // Dynamic spans become "*" wildcards - the static skeleton IS the contract. // Returns null when nothing is statically knowable (a bare variable). function resolveArg(node) { // cy.getByDataTest('submit-button') -> "submit-button" (exact) if (ts.isStringLiteral(node)) return node.text; // cy.getByDataTest(`plain-template`) -> "plain-template" (no ${}, still exact) if (ts.isNoSubstitutionTemplateLiteral(node)) return node.text; // cy.getByDataTest(`${bid.id}-bid-card`) -> "*-bid-card" // We can't know the runtime value of ${bid.id}, but we CAN verify the // app renders *something* ending in "-bid-card". if (ts.isTemplateExpression(node)) { let s = node.head.text; for (const span of node.templateSpans) s += "*" + span.literal.text; return s; } // cy.getByDataTest(someVariable) - fully dynamic. A fuller version resolves // constants, enums, and inlines selector-factory functions here before // giving up; this minimal one just reports it as skipped (skipped, but // counted - the blind spot must never be invisible). return null; } Enter fullscreen mode Exit fullscreen mode Phase 2 is the two attribute-harvesting regexes. These are the most repo-specific lines in the script, they encode your framework's syntax (Vue here; for JSX you'd match data-test="..." and data-test={`...`}): // Static attribute: data-test="orders-table" for (const m of src.matchAll(/(? "*-bid-card" // Same wildcard treatment as the spec side - skeletons match skeletons. for (const m of src.matchAll(/:data-test="`([^`]+)`"/g)) { patterns.add(m[1].replace(/\$\{[^}]*\}/g, "*")); } Enter fullscreen mode Exit fullscreen mode Phase 3 is matching. Wildcards become regexes anchored at both ends, so *-bid-card matches `${id}-bid-card` but refuses x-bid-card-list: // Escape regex metacharacters in the fixed fragments. const esc = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); // "*-bid-card" -> /^.*-bid-card$/ const toRegex = (pat) => new RegExp("^" + pat.split("*").map(esc).join(".*") + "$"); function isProducible(sel) { if (!sel.includes("*")) { // Literal reference: exact literal match, or a composing pattern // (e.g. the app-side pattern "*-bid-card" can produce "42-bid-card"). return literals.has(sel) || patternRegexes.some((r) => r.test(sel)); } // Pattern vs pattern: replace the spec side's wildcards with a sentinel // character that the app side's ".*" wildcard will absorb - the static // fragments still have to line up. const sentinel = sel.replace(/\*/g, "\u0000"); return patternRegexes.some((r) => r.test(sentinel)) || [...literals].some((l) => toRegex(sel).test(l)); } Enter fullscreen mode Exit fullscreen mode Run against a fixture with one typo'd selector and one stale absence assertion, the output looks like this — file, line, original expression, resolved selector, non-zero exit: Checked 4 selector call site(s); skipped 1 fully dynamic. 2 selector(s) not found in src/** : cypress/tests/orders.cy.ts:5 cy.getByDataTest("order-table") -> "order-table" cypress/tests/orders.cy.ts:6 cy.getByDataTest("delete-button") -> "delete-button" Selector contract check FAILED. Enter fullscreen mode Exit fullscreen mode These excerpts plus the glue you'd expect (walk the directories, collect the results, print the report) are the entire script. Treat what's on this page as your baseline: the architecture, the resolution rules, and the tricky bits are all here, and the remaining boilerplate is exactly the kind of thing your favorite AI coding assistant will fill in correctly with this article as the spec. Point it at your repo, tell it your helper name and attribute, and adapt the two harvesting regexes to your framework, that's the whole port. The Check Goes Both Ways Everything so far frames this as catching app-side drift: the app renames an attribute and tests break. But look at what the comparison actually does, it flags any test reference the app can't fulfill, without knowing which side moved. Sometimes the app renamed an attribute out from under a healthy spec. Sometimes the spec itself has rotted: it's still asserting things about UI that was redesigned or removed long ago. Both are broken references, and we wanted both from the start. Test-side rot has one especially nasty form: absence assertions. Consider cy.getByDataTest('delete-button').should('not.exist') guarding that viewers can't see a delete action. The day delete-button is renamed to remove-button, that assertion doesn't fail, it can't fail. It's checking that a name which no longer exists anywhere... doesn't exist. It will pass forever, green and useless, while the thing it was guarding goes untested. We call these vacuous assertions, and unlike positive-selector drift, no amount of runtime testing will ever surface them. So the checker validates absence assertions too: the element shouldn't be on the page right now, but the name still has to be real, something the app source could actually produce. The surprise wasn't that the check covered this; we designed it to. The surprise was how much it found. On its first run against our fully green suite: four stale names, a wrong-page selector, two flagged lines inside a four-assertion block guarding a requirement about a UI section that no longer existed in the app at all, plus one trailing-whitespace typo in a positive selector. Eight defects flagged in a suite that was passing every night. Why It's Not 100% We classified every one of our ~1,300 unique checked selector names by what actually protects it. The short version: hard rename protection covers about 15% of names, and we estimate a third to half of real-world drift incidents get caught. If that number sounds low, the examples below show exactly where the leaks are and why the check is still worth it. Fully Protected: Literal-Anchored Names. The spec says cy.getByDataTest('orders-table'), and the only thing in the source that can produce it is the literal data-test="orders-table". Rename or delete that attribute and the check goes red with a file:line hit. About 7% of our names live here, but they're disproportionately the distinctive, hand-written, page-level attributes that people actually rename during refactors. Protected at the Suffix: Pattern Contracts. A spec pattern like *-bid-card is checked against the bound attribute :data-test="`${bid.id}-bid-card`". Rename the binding to ${bid.id}-bid-tile and the check fails, the suffix contract is enforced. What it can't prove is that the runtime ids on both sides agree; they do in practice because both interpolate the same domain value, but that's a bet the runtime suite backstops. The Big Leak: Composed Attributes. Here's the miss, concretely. Say a shared Button wrapper component appends -button to whatever base it receives, so the app-side list contains only the pattern *-button — there is no save-button literal anywhere in the source to anchor against. A spec references cy.getByDataTest('save-button'). Someone renames the base from save to submit. The stale literal save-button still matches the pattern *-button, the check stays green, and the drift ships to runtime. The same applies to dialogs, form fields, and any attribute whose base is assembled across wrappers or derived from an i18n label at runtime: only the suffix convention is statically verifiable. Roughly 85% of our names sit in this composed class, which is why the headline number isn't higher. So why do we still estimate a third to half of real incidents caught when only ~15% of names are hard-protected? Because real drift skews favorable. Deleting a component removes its names and patterns from the app-side list too, so every spec still referencing them fails. And renames cluster on exactly the distinctive literal-anchored attributes the check fully protects, while the composed *-button names are the ones nobody touches individually. Meanwhile one guarantee holds: a name that doesn't exist in the source at all — a typo, a fabrication, a reference to a deleted component — can never get in through any statically resolvable call, which for us is 99.75% of them. Where It Runs CI, as early as possible. First step after dependency install, before the frontend build, before the Docker image, before ten parallel test shards spin up. This placement is the whole economic argument in one line: drift is caught before anything expensive starts, the failure lands red and deterministic on the PR that caused it instead of in a nightly run three days later. Pre-commit, as a complement. At one second, the check is cheap enough that the developer who typed the drift can catch it before it ever leaves their machine. We hooked it into the repo's existing husky + lint-staged setup, using the function form, lint-staged appends staged filenames to string commands, so a plain string entry would pass a pile of file paths as arguments to a script that doesn't expect them: // lint-staged.config.js module.exports = { "*.{ts,vue}": () => "node scripts/check-selectors.js", }; Enter fullscreen mode Exit fullscreen mode Summary Step zero is the convention: one helper function, one attribute. That's the majority of the work and valuable on its own. The rest is the three phases you just read, extract what the tests use, harvest what the app provides, compare, adapted to your helper name and your framework's attribute syntax. Run it against your green suite first and budget a small cleanup PR for what it finds; if our experience is any guide, it will find something. Ours took roughly a day to build and tune at ~500 lines, including cleaning up the defects it caught. One second per run. One day to build. Eight defects in a suite that was passing every night. I look forward to hearing how you adapt this approach. And as always, happy testing.