Stream catalog and bounded incremental data¶
P6 exposes read-only structured data events through five registry-defined tools:
stream_catalog, stream_peek, stream_collect, stream_refresh, and
stream_subscribe. CLI subscription is a single cursor/poll window; REST and MCP
use the same finite result contract. None of these tools creates a callback,
webhook, notification, schedule, trading trigger, or human console.
The built-in topic and event records are local contract fixtures. Production
registers reviewed topics derived from the mounted EODHD websocket sources
(us.equity.quote, us.equity.trade) in the durable PostgreSQL event store
and projects already-captured provider quote/trade observations into durable
stream events; stream_refresh additionally runs one bounded real provider
capture under a database refresh lease. A topic without persisted events
returns a complete empty window with its cursor, never a fixture row. The
protocol rules below describe the supported bounded contract; they do not
imply coverage for every covered security or every session window.
Under requirements 3.1, the EODHD real-time quote and
websocket trade connectors are disabled in config/connectors.yaml.
Production builds no topic from a disabled source, so no new quote or trade
events are captured. The stream tools are advanced tools that need an explicit
grant, and config/consumer_rights.yaml hides these sources from consumer
connections.
Catalog contract¶
Each topic declares its topic id, event schema version, provider, license grant, TTL, watermark, latest sequence, opaque cursor protocol, retention time/event limits, maximum subscription duration, and maximum events per request. Catalog and event models are closed schemas.
Every event carries a stable event_id for deduplication plus sequence,
event_time, observed_at, known_at, provider, schema version, quality codes,
license, evidence, lineage, and structured data. Payload fields associated with
prompts, commentary, recommendations, signals, alerts, notifications, scores,
orders, or trade actions are rejected.
Cursor and replay rules¶
Cursors are opaque HMAC-authenticated records bound to authenticated caller,
institution and topic. Request-supplied identity fields never establish this
binding. They
contain the next sequence and an expiry derived from topic retention. Tampering,
cross-tenant/cross-topic use, invalid sequences, expiration, or a sequence older
than retained data fails with cursor_invalid or cursor_expired.
Repeated reads with the same cursor return the same retained event ids. A client
persists the returned cursor only after processing the window; after disconnect it
reuses that cursor. It deduplicates by event_id. Sequence holes return gap and
explicit ranges. Data beyond TTL returns stale without events.
An initial read without a cursor starts at the earliest retained sequence visible
at the requested as_of. An empty retained set starts at the latest visible
sequence plus one. Retention can include sequences older than the event TTL, so
the initial window can be stale while the catalog also reports recent events.
Use the returned continuation cursor and a bounded stream_collect request to
advance through the retained windows. The catalog watermark describes the topic;
the watermark in a stale window describes that window's original observations.
Bounds and failure states¶
The protocol caps each response at 500 events, subscriptions at 60 seconds,
provider batches at 500 events, and pending retained events at 2,000. Same
tenant/topic refreshes share one in-flight provider request. A full topic buffer
rejects refresh before calling the provider. There is no unlimited history replay
or subscription. A stale retained window omits expired event payloads and
returns a continuation cursor past that window. Resume with stream_collect
to reach later retained sequences; the stale watermark keeps its original time.
Stable statuses are ok, stale, gap, partial, out_of_order,
cursor_expired, cursor_invalid, license_denied,
resource_limit_exceeded, and provider_failure. Provider failure is retryable;
if retained usable events exist, refresh returns them as partial and retains the
provider failure code. Resource violations fail closed. License violations fail
closed only when the server's license-enforcement switch is enabled; the default
disabled mode preserves source terms as metadata without granting source rights.
Monitoring¶
The durable implementation stores topic state and event windows in
PostgreSQL. Topic row locks allocate sequences atomically; compact event
identities survive payload expiration so replay cannot manufacture fresh data.
Provider refresh leases expire after 60 seconds and use ownership tokens to
reject writes from a stale leader. Clients retry a coalesced partial response
after the active leader commits. This storage implementation is not yet evidence
that a production supplier/topic is registered or that its data coverage passed.
Production fixtures remain isolated. Supplier capture and persisted event
projection require separate business acceptance for their actual topics,
securities and observation windows.
Operator-configured minimum refresh intervals persist with each topic, so releasing a leader does not reset the provider request budget for other workers. Topics with the same provider share a transaction lock, in-flight lease and the strongest persisted interval across sources, tenants and workers. A restarted reader or newly registered topic cannot reset this budget. Increasing an interval also respects the last request already made. Distinct providers have independent budgets. This conservative grouping does not establish a supplier subscription or authorize provider I/O: the real-provider bridge still must set intervals from verified supplier limits and enforce any additional shared account/product quota. A full retained window rejects refresh before provider I/O; ordinary read calls continue to consume its events.
The migration is additive. A production application rollback preserves these tables and their audit/data history; destructive schema downgrade is restricted to isolated restore rehearsals. Payload retention follows each configured topic; deduplication identities must be retained until a separately approved cleanup policy can demonstrate that upstream IDs will not be replayed.
Prometheus exposes argus_stream_operations_total,
argus_stream_provider_requests_total, and argus_stream_flow_control_total.
The last metric uses bounded kinds such as coalesced, backpressure, and
cursor_rejected; event values and cursor contents never appear in labels.
Duplicate captures and immutable events¶
Provider events for production quotes and trades can appear again in later
bounded refreshes. Persistence identifies duplicates by the originally captured
observed_at, known_at and a full-content checksum. A new retrieval time does
not replace the original clocks and does not create a new sequence. A change
in price, quantity, subject, currency, source, license or resolution mapping
still cannot reuse the same event identity.
The check runs under the database lock for the same topic. Restarts and
concurrent refreshes reuse the persisted identity. After the retention period
ends, a duplicate capture does not reinsert the old event or restore an expired
payload. The compact identity keeps its original clocks and checksum. Ordinary
StreamEvent writes still require the full content to match exactly. These
technical rules do not grant source licenses or use authorization.