SPEC-OPS-006: Public Request Monitoring

Reference Number	Version	Effective Date	Supersedes	Authority Level
SPEC-OPS-006	1.6	2026-10-23	1.5	Subordinate implementation contract

Revision 1.6 (2026-10-23; owner ruling 2026-10-23): the collector probes the gate first and, while the gate is closed, reads nothing from the application: no request telemetry and no health read; the endpoint and database checks are UNKNOWN under the new closed diagnostic NOT_COLLECTED (§VIII Independent platform checks). The collector carries no Cloudflare Access credential: the health endpoints are public, and the Access redirect is the maintenance signal. The service token existed only to reach production during maintenance.

Revision 1.5 (2026-10-21; owner ruling 2026-10-21): the endpoint check is FAIL (HTTP_UNAVAILABLE) only on an HTTP 5xx response. Any other non-redirect, non-authentication HTTP error, such as 429 (rate limited) or 404, means the check could not observe health: it is UNKNOWN under the new closed diagnostic HTTP_REFUSED, for the endpoint and the database alike, and so can never make the hero Mostly unavailable (§VIII Independent platform checks). Prompted by 2026-10-20, when the collector’s health read was rate limited and the hero read Mostly unavailable while the app was up.

Revision 1.4 (2026-10-02; ratified by the owner 2026-10-02, under DOM-OPS-001 2.12): adds the hall_pass component and its route group (§V, §VI); adds http_404_failure_count and the closed list of failed-request routes, which at 1.4 is the hall-pass approve, reject and cancel routes only (§V, §VI); the failed-request rule — at least 3 in the window and more than 2% of the component’s requests (owner ruling 2026-10-02) — governing classification, the card and the hero (§VI, §VIII); schema request-telemetry-v2, with request-telemetry-v1 snapshots still read (§V, §VII).

I. Purpose

Define objective public request measurements, independent operator interpretation and historical measured-window presentation authorized by DOM-OPS-001 §1. A request observation never establishes business correctness or universal availability.

II. Scope

The closed monitoring sampler, transport, snapshot persistence and public display. No application business changes, synthetic users, domain reads or mutation probes. Internal feature integrity remains governed by SPEC-OPS-005.

III. Authority Level

Subordinate to INV-CORE-000, INV-CORE-001, INV-ARC-005, INV-ARC-007, INV-ARC-009, INV-ARC-020 and DOM-OPS-001. DOM-OPS-001 authorizes the runtime behavior; this specification supplies its bounded implementation details.

IV. Dependencies

• DOM-OPS-001_OPERATIONS_DOMAIN.md
• SPEC-OPS-002_EXTERNAL_STATUS_PERSISTENCE_MODEL.md
• SPEC-OPS-003_APPLICATION_OBSERVABILITY_CONTRACT.md
• SPEC-OPS-005_FEATURE_HEALTH_EVIDENCE.md
• SOP-OPS-001_SERVICE_STATUS_AND_INCIDENT_COMMUNICATION.md

V. Closed numerical schema

The source is fixed by this schema as GRAFANA_TELEMETRY; it is not a caller-supplied field and cannot be relabeled as an independent feature verifier.

A snapshot contains exactly:

• schema_version: request-telemetry-v2;
• sampled_at: source-generated UTC ISO timestamp for the shared query evaluation time;
• window_seconds: integer 300;
• source_latest_at: newest nginx log EVENT UTC ISO timestamp (not ingestion time), or null;
• components: one object for each of service, login, attendance, hall_pass, payroll, roster, classroom_economy, with no duplicate, missing or additional keys.

Each component contains exactly:

• key: one closed component key;
• request_count, http_1xx_count, http_2xx_count, http_3xx_count, http_4xx_count, http_404_count, http_500_count, http_5xx_count, http_404_failure_count: nonnegative integers or null;
• p80_ms, p95_ms: finite nonnegative numbers or null;
• collection_state: OK or UNAVAILABLE.

For OK, all counts are integers and the 1xx/2xx/3xx/4xx/5xx counts partition total. 404 is a subset of 4xx and 500 a subset of 5xx. http_404_failure_count counts the 404s on the component’s failed-request routes (§VI) and is a subset of 404; a component with no such routes carries 0. Zero requests requires null quantiles. A nonempty window requires usable ordered quantiles (p80_ms <= p95_ms). For UNAVAILABLE, every numerical value is null. Unknown fields, arbitrary labels, booleans used as numbers, nonfinite numbers and unbounded values are rejected. Counts are bounded at 1,000,000,000 per window and latency at 600,000 milliseconds. Rates are derived from counts; an absent denominator never means zero percent. Only this reduced schema leaves the monitoring boundary, never log lines or labels.

A request-telemetry-v1 snapshot, written before 1.4, has the six components without hall_pass and components without http_404_failure_count, under the same rules. Readers accept it and treat it as carrying no hall-pass evidence and no failed-request 404s; it can carry neither. A snapshot of either version with the other version’s keys or fields is rejected. This lets the host sampler and the status service be upgraded in either order, and keeps retained activity readable (§VIII).

VI. Collection, freshness and classification

The sampler queries local Loki with fixed reviewed route expressions and a common UTC evaluation time. It aggregates away all labels. Caller-defined queries are forbidden. Request counts and latency come from the same 300-second window. The service group observes upstream application requests. Route groups are proxies, not a complete registry of business operations. Source evidence may overlap groups. The hall_pass group is the hall-pass API (/api/hall-pass/...), the teacher’s Hall Pass page (/admin/hall-pass) and the verification page office staff use (/verify/hallpass/<token>). No other group matches these routes, so adding it changes no other component’s numbers; service already counted them.

Failed-request routes are closed and fixed by this specification. At 1.4 they are, for hall_pass only, /api/hall-pass/request/<id>/approve, .../reject and .../cancel: the teacher’s Hall Pass page approves or rejects, and the student’s page cancels, only a request id the application has just shown, so a 404 there means the request the user was looking at could not be found. Every other route, on hall passes or elsewhere, has none. The sampler counts their 404s with one fixed query on the same stream, window and evaluation time as the component’s counts, reduced to a single number; no route or status label leaves the host.

Sample once per minute. A snapshot older than 300 seconds is stale; source lag older than 180 seconds makes monitoring unavailable/stale rather than normal. Missing source freshness cannot prove current monitoring. Source future timestamps must not refresh evidence. The static telemetry endpoint polling request can keep the nginx stream active. Source recency measures nginx log-source activity only; it cannot certify complete ingestion, application readiness or feature correctness. No new application health endpoint is required; the existing bounded platform health read remains authorized independently below. A successfully queried empty route group with a fresh source is NO_TRAFFIC; an unavailable source is MONITOR_UNAVAILABLE. A transport retry must preserve source timestamps. Receipt time is persisted separately and never renews evidence. Collection or validation failures produce bounded unavailable evidence, never invented zero counts or a fabricated source sampled time. The persistence API is append_snapshot(snapshot_or_none, received_at, diagnostic=None). A null snapshot records a collector failure using receipt time and one of TRANSPORT_UNAVAILABLE, ACCESS_DENIED, INVALID_SNAPSHOT, or STALE_SNAPSHOT, separately from genuine source-minute snapshots, so a failed attempt cannot suppress a valid sample from the same minute. Transport failure is not a source measurement.

Initial policy thresholds are strict greater-than comparisons:

• 5xx / requests > 2%: elevated server-error responses;
• failed-request 404s at least 3 in the window and failed-request 404 / requests

  2%: requests not found when acted on (only a component with failed-request routes; at 1.4, hall-pass requests not found when approved, rejected or cancelled). Both conditions must hold, so one or two stray 404s (a double click, a stale page) never qualify. This is the failed-request rule; the card, the hero and operator drafts use it too;

• 404 / requests > 5%: elevated not-found responses;
• p95 > 1500 ms: high latency.

404 is a descriptive anomaly signal, not SYSTEM_FAILURE: expected resource absence may account for it. Failed-request 404s that meet the failed-request rule are a signal of a possible problem on their component; they are still not SYSTEM_FAILURE, an outage or an incident. Except for that rule’s minimum, all threshold crossings remain visible regardless of sample size. Any nonempty window without a threshold crossing is NORMAL; there is no other minimum count. This numerical classification does not certify successful classroom actions. Distinct states are NORMAL, ELEVATED_ERRORS, HIGH_LATENCY, NO_TRAFFIC, MONITOR_UNAVAILABLE, and STALE. Show all applicable reasons; where a single state is required, unavailable/stale takes precedence, followed by elevated errors, high latency, no traffic and normal. Elevated errors must distinguish server-error responses, failed-request 404s and other not-found responses in visible wording. These numerical thresholds are disclosed publicly and changed only with a versioned code/contract amendment. They do not diagnose user impact or create incidents.

VII. Storage and history

telemetry_snapshots stores append-only genuine source snapshots with a deterministic source-UTC- minute ID and separate received_at. A transaction appends each minute once, updates a monotonic telemetry_current/current pointer, and updates that source UTC day’s telemetry_days/YYYY-MM-DD rollup once. All transactional reads precede writes. Duplicate delivery cannot increase coverage; old delivery cannot move the current pointer backward. Reject stale source snapshots at collector ingestion.

Daily per-component counters are sampled, measured, normal, elevated_errors, high_latency, and other. sampled counts unique persisted minute windows; measured counts eligible windows classified NORMAL, ELEVATED_ERRORS or HIGH_LATENCY. Empty and unavailable/stale windows do not establish a normal operating range and are excluded from that denominator. normal / measured is the share of eligible observed windows within thresholds. With no eligible windows this percentage is unavailable, never 100%.

Display 90 UTC calendar dates including today. Gaps are explicit. Coverage is eligible measured windows divided by scheduled elapsed minute windows for each UTC day, not request success or application uptime. Today’s denominator stops at the observation time. Sampled and eligible counts remain available for explanation. Do not sum overlapping five-minute request counts into daily traffic totals. A daily single-state tally follows classification precedence while current cards preserve all applicable threshold reasons.

Historical rollups retain the policy applied at collection; older windows may have required 20 requests. hall_pass history starts with the first request-telemetry-v2 window; days before it have no hall_pass counters, shown as gaps. This distinction is disclosed beside history; old counters are not recomputed or relabeled as uptime.

Detailed snapshots expire after seven days; daily rollups after 90 days. Use separate Firestore TTL expiry fields/provisioning for these collections. Do not alter existing observation or notice retention. The current pointer is replaceable and still subject to read-time freshness. Old integrity observations are retained under their existing policy and never transformed into request history.

VIII. Public interface and operator interpretation

The public pages are written for teachers and students. Labels use plain language and describe what a reader can rely on; protocol names, enum values and investigation references never serve as headings. Public layout follows the shared public-page design in this order: overall status hero, operator incident/update banner, simple teacher/student service cards, and platform status with database reachability plus request measurements/history.

The hero asks “Is Classroom Token Hub Working?” and answers with exactly one state. Precedence is top to bottom; the first condition that holds decides:

State	Public label	Condition
maintenance	Under maintenance	A fresh gate check reports the Application Availability Gate closed (§Independent platform checks)
unavailable	Mostly unavailable	A fresh endpoint or database check is FAIL
degraded	Detected problems	A current operator notice, a fresh nonzero HTTP 5xx count, or a fresh window meeting the failed-request rule (§VI)
available	No known issues	Both endpoint and database checks are fresh PASS, including during idle request traffic
unknown	Unknown	Anything else: missing, unknown, stale or future checks

Under maintenance is gate messaging under DOM-OPS-001 §Application Availability Gate. It communicates an access restriction and makes no claim about application health; it overrides the other states because readers cannot enter while the gate is closed, whatever the checks report. No known issues claims only the absence of a known problem: reachability establishes connectivity, not business correctness. Detected problems means signals of possible issues were observed. It does not declare an outage, open an incident or create a notice; the wording that accompanies it on the page names what was seen and never says or implies an outage. The hero’s timestamp is the older of the two fresh endpoint/database PASS/FAIL check timestamps, never stale/future evidence or the request snapshot time. Each state carries a visible word as well as its colour (INV-ARC-020).

Feature cards describe recent activity, not yes/no functionality estimates. While the hero is Under maintenance, every card reads Closed for maintenance: nobody can enter, whatever requests from behind the gate show. Otherwise a current operator notice for the card’s capability takes precedence: an AWARE notice reads Checking reports, any other active notice Having problems. Otherwise any fresh 5xx, or a fresh window meeting the failed-request rule (§VI), is Checking: errors seen — an automatic observation at the AWARE level, which never creates or implies a notice; p95 above 1,500 ms is Slower than usual; a window containing 2xx/3xx is Working. Windows with only remaining response classes say Nothing to report, claiming neither success nor outage. Apart from the failed-request rule’s minimum, these labels have no minimum count. Quiet windows say Quiet; missing or stale collection says Status unknown. All cards disclose their scope in visible text.

The current telemetry projection retains last_activity, a map of the closed component keys to their latest nonempty, valid, fresh source window’s sampled_at, source_latest_at and validated component. The collector updates it atomically with the current pointer only when that pointer advances. Idle or failed attempts preserve historical activity without refreshing its timestamp; duplicate or delayed samples cannot overwrite it. Readers validate retained data and exclude future or older-than-seven-day entries; advancing writes prune expired entries. No new raw fields, tenant labels or business data are retained. Absence means no retained activity. Historical outcomes use neutral styling during inactivity or monitoring loss, visibly labeled Last observed with the original five-minute window end. They never establish present health or conceal a monitoring failure. Public GET remains read-only.

A fresh window meeting the failed-request rule makes the hero Detected problems, as a fresh 5xx does; failed-request 404s below the rule, and every other 404, change neither the hero nor any card. The operator console offers a window meeting the rule, not covered by a notice, as a draft, as it does a fresh 5xx; a draft is not a notice.

Counts, 404/500/5xx percentages (and, for a component with failed-request routes, its failed-request percentage), p80/p95, source timing and 90-day history appear in the lower platform section, collapsed by default behind a visible disclosure, and the 90-day history may also be summarised beside the main column. Explain their scope and thresholds there, without long technical qualifications on every teacher card. The hero uses the public site’s landing composition and the brand wordmark under SPEC-DES-001 §IX; the footer is the public site’s footer. Historical bars expose full text and keyboard/touch disclosure, not color or hover alone. Operator notices are independent, explicitly attributed human interpretation, with optional snapshot links and an investigation evidence note or reference when no automated snapshot is linked. They do not overwrite metrics.

Public GET reads persisted projections only. The browser never receives Loki, Grafana or monitoring credentials or a raw-query interface. The sampler writes bounded validated JSON atomically to /var/lib/cth-status/telemetry.json; /health/telemetry is an exact static nginx endpoint permitting GET/HEAD with Cache-Control: no-store. It is public, like the application health endpoints. The collector reads it with no credential, a timeout, a response-size limit and no redirects, and does not read it while the gate is closed. Grafana operator authentication is unchanged. Installing the sampler and endpoint is a separate reviewed deployment operation.

Independent platform checks

Platform connectivity is persisted separately from request telemetry. Its record has exactly schema_version, UTC ISO received_at, and checks. A platform-check-v2 record contains exactly one endpoint, one database and one gate result; a platform-check-v1 record, written before v2, contains exactly the first two and is read as carrying no gate evidence. Each result has key, outcome (PASS/FAIL/UNKNOWN), checked_at (UTC ISO or null), and a closed diagnostic. Endpoint HTTP_OK is PASS; HTTP_UNAVAILABLE is FAIL. Actual database DATABASE_REACHABLE is PASS and DATABASE_UNAVAILABLE is FAIL. ACCESS_DENIED, TRANSPORT_UNAVAILABLE, HTTP_REFUSED, INVALID_RESPONSE and NOT_COLLECTED are UNKNOWN for either check; STALE_SOURCE is additionally permitted for database UNKNOWN. UNKNOWN carries null checked_at; proven checks preserve their source timestamp, never later than receipt. Current results expire after 300 seconds; future or malformed evidence is unknown.

The health read’s HTTP status maps to exactly one endpoint diagnostic. A 2xx body that validates is HTTP_OK. HTTP_UNAVAILABLE is an HTTP 5xx response and nothing else: a server failure is the only response that establishes the application is failing. A redirect (301, 302, 303, 307, 308), 401 or 403 is ACCESS_DENIED. Any other HTTP error status, for example 429 (rate limited) or 404, is HTTP_REFUSED: the request was refused before health was observed, so the check is UNKNOWN with null checked_at, never FAIL, and the database result carries the same diagnostic. A refusal is not executed evidence of failure (INV-ARC-017). A bounded read of the existing health endpoint may obtain these two actual checks without collecting its feature placeholders. This is connectivity, not domain proof.

The gate check observes whether the Application Availability Gate (DOM-OPS-001) is closed. Outside operational work the application hostname has no Cloudflare Access policy in front of public pages, so the check sends one bounded GET for a static public asset on that hostname with no service token or other credential, follows no redirect, and reads only the status line and Location header. A redirect whose Location host is a Cloudflare Access login host (*.cloudflareaccess.com) is GATE_CLOSED (FAIL). Any other HTTP response — including an application error, which the endpoint check reports separately — is GATE_OPEN (PASS). An HTTP 401 or 403 is UNEXPECTED_DENIAL (UNKNOWN), because a public asset is never denied when the gate is open and the gate’s own response is not otherwise established. Transport failure is TRANSPORT_UNAVAILABLE (UNKNOWN). The health read cannot stand in for this check: a redirect there is ACCESS_DENIED, which remains UNKNOWN and never implies the gate is closed.

The collector probes the gate before anything else. While the gate is GATE_CLOSED, it reads nothing from the application: it makes no health read and stores no request telemetry, and the platform record carries the endpoint and database as UNKNOWN NOT_COLLECTED with null checked_at beside the gate result. The hero is Under maintenance from the gate alone. A closed gate is an access restriction, not an application failure, and never contributes FAIL to the endpoint or database results.

platform_observations retains append-only records for seven days using expires_at; platform_current/current is a monotonic replaceable projection. Reads are pure. These records do not contribute to request-window history or service-card proofs.

IX. Validation

Focused tests cover strict schema/redaction, query failure, malformed/multi-series responses, no/low traffic, thresholds and their boundaries, 404 versus semantic failure, failed-request 404s on the designated routes against 404s elsewhere, both schema versions, every hall-pass route falling in the hall-pass group and no other, stale/future timestamps, retry idempotency, monotonic current pointer, UTC rollover, history denominators/gaps, independent notices, auth/CSRF and pure GET. Endpoint tests cover 5xx as FAIL and a 429 or 404 as UNKNOWN HTTP_REFUSED, which leaves the hero Unknown, not Mostly unavailable. Gate tests cover an Access redirect, an open response, an application error, 401/403, transport failure, a redirected health read with an open gate, and a closed gate that reads nothing from the application. Render populated/idle/stale history and cards for keyboard/touch access. A deployed claim requires source → collector → Firestore → rendered evidence.

X. Amendment

Increment version, update date and documentation index, and review changes against DOM-OPS-001. No amendment may relax internal verification or privacy authority.