Skip to content

Stream catalog and bounded incremental data

P6 exposes read-only structured data events through five registry-defined tools: stream_catalog, stream_peek, stream_collect, stream_refresh, and stream_subscribe. CLI subscription is a single cursor/poll window; REST and MCP use the same finite result contract. None of these tools creates a callback, webhook, notification, schedule, trading trigger, or human console.

The built-in topic and event records are local contract fixtures. Production registers reviewed topics derived from the mounted EODHD websocket sources (us.equity.quote, us.equity.trade) in the durable PostgreSQL event store and projects already-captured provider quote/trade observations into durable stream events; stream_refresh additionally runs one bounded real provider capture under a database refresh lease. A topic without persisted events returns a complete empty window with its cursor, never a fixture row. The protocol rules below describe the supported bounded contract; they do not imply coverage for every covered security or every session window.

Under requirements 3.1, the EODHD real-time quote and websocket trade connectors are disabled in config/connectors.yaml. Production builds no topic from a disabled source, so no new quote or trade events are captured. The stream tools are advanced tools that need an explicit grant, and config/consumer_rights.yaml hides these sources from consumer connections.

Catalog contract

Each topic declares its topic id, event schema version, provider, license grant, TTL, watermark, latest sequence, opaque cursor protocol, retention time/event limits, maximum subscription duration, and maximum events per request. Catalog and event models are closed schemas.

Every event carries a stable event_id for deduplication plus sequence, event_time, observed_at, known_at, provider, schema version, quality codes, license, evidence, lineage, and structured data. Payload fields associated with prompts, commentary, recommendations, signals, alerts, notifications, scores, orders, or trade actions are rejected.

Cursor and replay rules

Cursors are opaque HMAC-authenticated records bound to authenticated caller, institution and topic. Request-supplied identity fields never establish this binding. They contain the next sequence and an expiry derived from topic retention. Tampering, cross-tenant/cross-topic use, invalid sequences, expiration, or a sequence older than retained data fails with cursor_invalid or cursor_expired.

Repeated reads with the same cursor return the same retained event ids. A client persists the returned cursor only after processing the window; after disconnect it reuses that cursor. It deduplicates by event_id. Sequence holes return gap and explicit ranges. Data beyond TTL returns stale without events.

An initial read without a cursor starts at the earliest retained sequence visible at the requested as_of. An empty retained set starts at the latest visible sequence plus one. Retention can include sequences older than the event TTL, so the initial window can be stale while the catalog also reports recent events. Use the returned continuation cursor and a bounded stream_collect request to advance through the retained windows. The catalog watermark describes the topic; the watermark in a stale window describes that window's original observations.

Bounds and failure states

The protocol caps each response at 500 events, subscriptions at 60 seconds, provider batches at 500 events, and pending retained events at 2,000. Same tenant/topic refreshes share one in-flight provider request. A full topic buffer rejects refresh before calling the provider. There is no unlimited history replay or subscription. A stale retained window omits expired event payloads and returns a continuation cursor past that window. Resume with stream_collect to reach later retained sequences; the stale watermark keeps its original time.

Stable statuses are ok, stale, gap, partial, out_of_order, cursor_expired, cursor_invalid, license_denied, resource_limit_exceeded, and provider_failure. Provider failure is retryable; if retained usable events exist, refresh returns them as partial and retains the provider failure code. Resource violations fail closed. License violations fail closed only when the server's license-enforcement switch is enabled; the default disabled mode preserves source terms as metadata without granting source rights.

Monitoring

The durable implementation stores topic state and event windows in PostgreSQL. Topic row locks allocate sequences atomically; compact event identities survive payload expiration so replay cannot manufacture fresh data. Provider refresh leases expire after 60 seconds and use ownership tokens to reject writes from a stale leader. Clients retry a coalesced partial response after the active leader commits. This storage implementation is not yet evidence that a production supplier/topic is registered or that its data coverage passed. Production fixtures remain isolated. Supplier capture and persisted event projection require separate business acceptance for their actual topics, securities and observation windows.

Operator-configured minimum refresh intervals persist with each topic, so releasing a leader does not reset the provider request budget for other workers. Topics with the same provider share a transaction lock, in-flight lease and the strongest persisted interval across sources, tenants and workers. A restarted reader or newly registered topic cannot reset this budget. Increasing an interval also respects the last request already made. Distinct providers have independent budgets. This conservative grouping does not establish a supplier subscription or authorize provider I/O: the real-provider bridge still must set intervals from verified supplier limits and enforce any additional shared account/product quota. A full retained window rejects refresh before provider I/O; ordinary read calls continue to consume its events.

The migration is additive. A production application rollback preserves these tables and their audit/data history; destructive schema downgrade is restricted to isolated restore rehearsals. Payload retention follows each configured topic; deduplication identities must be retained until a separately approved cleanup policy can demonstrate that upstream IDs will not be replayed.

Prometheus exposes argus_stream_operations_total, argus_stream_provider_requests_total, and argus_stream_flow_control_total. The last metric uses bounded kinds such as coalesced, backpressure, and cursor_rejected; event values and cursor contents never appear in labels.

Duplicate captures and immutable events

Provider events for production quotes and trades can appear again in later bounded refreshes. Persistence identifies duplicates by the originally captured observed_at, known_at and a full-content checksum. A new retrieval time does not replace the original clocks and does not create a new sequence. A change in price, quantity, subject, currency, source, license or resolution mapping still cannot reuse the same event identity.

The check runs under the database lock for the same topic. Restarts and concurrent refreshes reuse the persisted identity. After the retention period ends, a duplicate capture does not reinsert the old event or restore an expired payload. The compact identity keeps its original clocks and checksum. Ordinary StreamEvent writes still require the full content to match exactly. These technical rules do not grant source licenses or use authorization.