Evidence-backed monitoring systems
Evidence-backed monitoring systems
A monitoring system earns trust by preserving the path from observation to decision. A digest lists events. A decision system maintains a baseline, records what it checked, separates evidence from inference, changes state only when the evidence warrants it, and keeps enough history to revisit the call.
Notto Competitor Watch made that pattern concrete, but it applies to regulatory watch, vendor risk, security advisories, account health, incident follow-up and any other repeated evidence review.
Planetary Escape company management adds the compliance version: notifications identify questions, authoritative records settle them, deadlines create state transitions, and preserved proof closes actions. See Notifications are leads, not source of truth.
The operating loop
- Define what would count as a material change before searching.
- Record the sources, identities, aliases and time range checked.
- Store the event date separately from the date it was observed.
- Separate verified fact, source or company claim, and internal inference.
- Compare the result with the retained baseline.
- Change the decision state only when the evidence changes risk or action.
- Publish the decision with evidence beside each factual claim.
- Retain both the source record and the decision history.
This loop makes No action a legitimate result. Monitoring should protect a roadmap from noisy reactions as well as surface genuine threats.
Time has three meanings
An event date says when the outside world changed. An observation date says when the monitor found it. A decision date says when the team changed its view or action.
An old event found today is a baseline correction. It can change today's decision without becoming today's market event. Collapsing those dates creates false momentum and makes recurring scans rediscover the same story as new.
No change requires coverage
GREEN is an evidence claim. It means the required search ran and found no verified material change. A missing scan is an observation gap, so the honest result for that interval is unknown. See Evidence of Absence.
The practical test is simple: would this search have found the change if it had happened? If coverage, identity resolution or source access was too weak, record the limitation instead of upgrading silence into evidence.
Severity is decision state
Alert levels should describe the decision the evidence creates:
- immediate action with an owner;
- a material baseline or narrative change; or
- no verified material change.
The highest active state remains in force until later evidence explicitly reduces it. A quiet day does not erase yesterday's unresolved threat. This is the state-transition rule described in Monitors should alert on state transitions.
Evidence must sit beside the claim
An appendix helps navigation. It does not show which source supports which sentence. Put the link beside the fact, score or definition it supports, and keep source claims separate from the team's inference.
Link presence is only the first check. A validator can reject unsafe or invented links and still miss a real source that does not support the sentence. Provenance and support remain separate tests; see Citations Required, Validator Enforces.
Publication is part of the evidence model
The current page, retained history and evidence library are views over durable local records. A small pointer can identify the current decision while the retained records rebuild the archive. The remote directory stays a deployment target, not the only surviving copy.
A fresh current-state page does not imply a state transition. A successful GREEN run can close the monitoring loop without producing an executive report. This is the monitoring version of Zero Change Can Be Success: completion, publication and activity are separate facts.
That distinction becomes easier to enforce when the monitoring record and decision brief are separate artifacts. The record preserves coverage, rejected leads and unchanged baselines. The brief exists only when the evidence changes a decision. See Monitoring records and decision briefs are different products.
That design depends on Mirror Deployments Require Complete Artifacts, Archives Need a Durable Source of Truth and Authentication Redirects Do Not Prove Content Health. The release must contain every route that should survive, and health checks must test both the gate and the content behind it.
The reading surface is part of the contract too. A recurring report should keep one visual grammar so readers can compare evidence without relearning the interface. Name the reference shell, then test the rendered page against it. See Recurring Reports Need a Stable Visual Contract.
The documentation contract
Monitoring docs contain executable behaviour: search scope, thresholds, ownership, publisher commands and failure rules. Validate examples from the documented working directory. A command that names the right script with the wrong path is still a broken operating contract.
Generated checks can protect path safety, required files and access-control invariants. They cannot keep the surrounding explanation honest by themselves. Periodic code-truth sweeps still matter; see Generated Docs as Drift Defense.
Failure modes
- Treating news volume as threat level.
- Reporting an old event as new because it was found today.
- Calling an unobserved interval quiet.
- Mixing source claims with internal conclusions.
- Keeping sources in an appendix without claim-level links.
- Rebuilding a mirrored release from only the files that changed.
- Testing the authentication redirect while ignoring the protected page.
- Letting each run invent a new stylesheet for the same report series.
- Publishing the monitoring apparatus as if it were an executive decision.