Stage Four Technology Introduction Gate¶
This document is the step 51 design record and ADR gate for stage four data infrastructure. It is a planning contract only: it does not introduce Kafka, Elasticsearch, Airflow, a new orchestration runtime, a human console, a dashboard, a chat surface, or any prohibited financial capability.
Stage four technologies may extend throughput, replay, indexing, orchestration,
history depth, and source governance only after the current PostgreSQL, Redis,
Celery, MinIO, and batch processing baseline has been measured and shown
insufficient for a verified requirement. New technology cannot replace the shared
DataPackage, SourceEvidence, license, permission, audit, provenance,
tenant-isolation, or quality contracts.
ADR Gate¶
Every proposal for Kafka, Elasticsearch, Airflow, or another platform component must be recorded as an architecture decision before implementation.
| ADR field | Required content | Gate id |
|---|---|---|
proposal_id |
Stable identifier and owner. | adr-proposal-identity |
verified_requirement |
Product requirement, workload, or failure mode proven by test data. | adr-verified-requirement |
measured_bottleneck |
Current PostgreSQL, Redis, Celery, MinIO, or batch baseline limit with reproducible evidence. | adr-measured-bottleneck |
baseline_alternatives |
Tuning, schema changes, indexes, partitioning, caching, retry policy, worker scaling, object layout, or batch redesign considered first. | adr-baseline-alternatives |
operational_cost |
Deployment, monitoring, backup, retention, security, staffing, incident response, and upgrade cost. | adr-operational-cost |
rollback_path |
Reversible migration, dual-write cutoff, replay plan, and fallback read path. | adr-rollback-path |
core_contract_impact |
Explicit mapping back to data package, evidence, license, permission, audit, output policy, and quality contracts. | adr-core-contract-impact |
failure_mode |
Behavior when the new component is degraded, unavailable, or returns partial results. | adr-failure-mode |
release_gate |
Tests and review approvals required before production use. | adr-release-gate |
An ADR is rejected when it lacks measured bottleneck evidence, skips existing-component alternatives, weakens core contracts, has no rollback path, or requires users to access a human interface to recover core facts.
Candidate Technology Gates¶
| Technology | May be introduced only when | Must not replace | Required fallback |
|---|---|---|---|
| Kafka | Redis queues, Celery task routing, PostgreSQL change capture, and existing batch jobs cannot satisfy a verified event-stream, replay, or fan-out workload. | Audit log, data packages, evidence mapping, source connector registry, or task status records. | Core fact queries continue through PostgreSQL-backed services; delayed event processing is marked degraded with audit ids. |
| Elasticsearch | PostgreSQL tsvector, pg_trgm, GIN indexes, pgvector, and query tuning cannot satisfy a verified deep-search or high-cardinality retrieval workload. |
Filing fragment records, evidence snippets, field catalog, permissions, licenses, or output policy checks. | Search degrades to PostgreSQL-backed retrieval; exact fact and filing lookup remains available. |
| Airflow | Celery Beat, Celery workers, batch manifests, retry policies, and explicit job dependencies cannot satisfy a verified complex DAG, replay, or backfill workload. | Core batch services, audit records, license checks, task status, or deterministic replay metadata. | Existing Celery/batch workflow remains the recovery path for core fact refresh and exports. |
Stage Four Capability Gates¶
Stage four platform capabilities must remain auditable, reproducible, and contract-preserving before any runtime component is selected.
| Capability | Required gate | Replay or audit requirement | Gate id |
|---|---|---|---|
| Multi-source merge | Preserve every contributing evidence record and source priority. | Conflict decisions create quality issues and audit records. | cap-multi-source-merge |
| Deep historical data | Preserve as-of, known-at, revision, and source version boundaries. | Historical reads cannot use facts unknown at the requested time. | cap-deep-history |
| Custom field system | Map all custom fields to the Field Catalog with unit, currency, period, aliases, quality rules, and license tier. | Unmapped fields remain review-only or unavailable. | cap-custom-field-system |
| Internal research library | Use Source Connector Registry and internal data ingestion contracts. | Internal facts keep source system, extraction time, known-at time, parser version, license, and audit id. | cap-internal-research-library |
| Cross-agent tool registry | Publish machine-readable tool metadata only. | Tool versions, permissions, allowed uses, safe alternatives, and audit fields remain explicit. | cap-cross-agent-registry |
| Large-scale batch tasks | Preserve task status, retry reason, output manifest, license restrictions, and audit trace. | Batch failures cannot corrupt existing fact snapshots. | cap-large-scale-batch |
| Advanced data quality monitoring | Emit structured quality issues instead of silent corrections. | Alerts and anomalies reference evidence ids and policy versions. | cap-advanced-quality |
| Multi-agent collaboration audit | Append-only workflow trace with machine identities. | Agent actions remain attributable and replayable. | cap-multi-agent-audit |
| Deterministic Replay | Bind data version, schema version, permission version, license version, output-policy version, model configuration version, and source connector version. | Replay output must be comparable to the original package and audit id chain. | cap-deterministic-replay |
| Entity Resolution graph extension | Preserve entity ids, aliases, confidence, source evidence, known-at time, and review status. | No mapping may create future-function access or wrong-security substitution. | cap-entity-resolution |
| Corporate Actions Normalization extension | Preserve split, dividend, symbol, identifier, effective date, known-at time, source, and adjustment version. | Adjusted outputs cite the normalization version and evidence. | cap-corporate-actions-normalization |
| Bitemporal Data Store | Store valid time and transaction or known-at time separately. | Queries default to point-in-time correctness and exclude future-known records. | cap-bitemporal-data-store |
| Market Calendar & Session Service | Version exchange calendars, holidays, early closes, time zones, and session definitions. | Market time calculations cite calendar version and cannot infer real-time tradability from delayed data. | cap-market-calendar-session |
| Filing / Fact Delta Package deepening | Compare only structured facts, filings, snapshots, and known-at times. | Delta packages cite old evidence, new evidence, known-at time, revision time, and audit ids. | cap-filing-fact-delta |
| Source Connector Registry deepening | Version source metadata, license, update frequency, field coverage, failure status, parser version, and quality level. | Registry changes are audit-addressable and do not grant access by themselves. | cap-source-connector-registry |
Failure Matrix¶
Core fact queries must keep working when optional stage four technology is unavailable.
| Failure case | Required behavior | Test matrix id |
|---|---|---|
| Kafka unavailable | Stop or delay stream processing, mark source or task degraded, and preserve PostgreSQL-backed fact reads. | failure-kafka-core-query-available |
| Elasticsearch unavailable | Fall back to PostgreSQL search or exact lookup and return degraded retrieval metadata. | failure-elasticsearch-core-query-available |
| Airflow unavailable | Continue existing Celery and batch recovery path for required jobs. | failure-airflow-core-query-available |
| Optional component returns partial data | Do not publish partial facts without quality issues, evidence links, and audit status. | failure-partial-data-quality-audit |
| Optional component diverges from core state | Prefer core contract state and open a reconciliation quality issue. | failure-core-state-authoritative |
Contract Mapping¶
New technology outputs remain intermediate until converted to standard packages.
| Output type | Required mapping before exposure | Test matrix id |
|---|---|---|
| Event stream record | DataPackage, SourceEvidence, source connector id, parser version, license decision, quality state, audit id. |
contract-event-stream-to-package |
| Search index hit | Filing or fragment id, evidence pointer, permission check, license check, output policy version, audit id. | contract-search-hit-to-evidence |
| DAG task result | Task status, input manifest, output manifest, license restrictions, quality issues, audit id. | contract-dag-result-to-package |
| Replay result | Original package id, replay package id, version vector, comparison status, audit id. | contract-replay-result-to-package |
| Entity graph edge | Canonical entity id, alternate id, confidence, source evidence, known-at time, review status, audit id. | contract-entity-edge-to-evidence |
| Calendar result | Calendar id, exchange, session date, session state, time zone, calendar version, audit id. | contract-calendar-result-to-package |
Replay Matrix¶
Deterministic replay is approved only when the replay request includes a complete version vector.
| Version dimension | Required value | Test matrix id |
|---|---|---|
| Data version | Source connector version, source object hash, package id, and fact snapshot version. | replay-data-version |
| Schema version | Data package schema, field catalog version, parser version, and migration version. | replay-schema-version |
| Permission version | Machine identity version, role or ABAC policy version, tenant isolation version, and tool whitelist version. | replay-permission-version |
| License version | Source license, field license, redistribution policy, and restricted-field policy. | replay-license-version |
| Output-policy version | Output-origin rules, source-material labeling, neutral taxonomy, and compatibility metadata. | replay-output-policy-version |
| Model configuration version | Embedding model, reranker, LLM configuration, prompt version, and deterministic fallback setting when used. | replay-model-config-version |
Entity And Time Matrix¶
Entity resolution, corporate action normalization, bitemporal history, and market calendar logic must avoid future functions and wrong-security mapping.
| Scenario | Required behavior | Test matrix id |
|---|---|---|
| Entity alias learned after an as-of date | Historical query cannot use the alias unless known-at is earlier than or equal to the as-of time. | entity-no-future-alias |
| Security identifier reused or changed | Preserve identifier effective dates and source evidence; do not substitute another security silently. | entity-no-wrong-security |
| Corporate action announced after query date | Do not apply the adjustment to an earlier as-of package unless it was known at that time. | corporate-action-no-future-adjustment |
| Split or dividend has conflicting sources | Mark quality issue and preserve both evidence records. | corporate-action-conflict-quality |
| Market holiday calendar is revised | Store calendar version and recompute only under explicit replay or revision policy. | calendar-versioned-revision |
| Delayed market data is requested as real time | Return delayed or end-of-day metadata and block real-time trading implication. | calendar-no-real-time-implication |
Product Boundary Review¶
Stage four remains a data infrastructure platform. It may provide machine-readable facts, labeled source material, evidence, metadata, replay, source governance, quality findings, and neutral deltas. It must not generate its own buy, sell, hold, target-price, sizing, allocation, or return judgment, and it does not add automated order workflows or a human interaction surface. Quoted third-party material is preserved with provenance labels for independent client-Agent use.
| Boundary check | Required outcome | Test matrix id |
|---|---|---|
| Request includes trading direction or investment intent | Return otherwise permitted facts and labeled source material; the platform component itself does not generate a judgment. | boundary-client-analysis-independent |
| Unlabeled Argus-origin output presents a target price, sizing, allocation, or return forecast as its own judgment | Output-origin denial; labeled quoted source material is preserved. | boundary-output-origin-denied |
| Proposal requires a human dashboard, console, chat surface, or manual form | Reject from this stage-four gate. | boundary-human-interface-rejected |
| Proposal keeps machine-readable facts and evidence only | May proceed to ADR review when every other gate passes. | boundary-machine-readable-allowed |
Release Gate For Step 51¶
Step 51 is complete only when design-level tests confirm:
- Kafka, Elasticsearch, and Airflow have explicit measured-need gates and fallbacks.
- Every new technology proposal must include bottleneck evidence, alternatives, operational cost, rollback path, contract impact, and failure mode.
- Optional technology cannot replace data packages, evidence, license, permission, audit, provenance, tenant-isolation, or quality contracts, and cannot constrain independent client-Agent reasoning.
- Deterministic replay binds data, schema, permission, license, output-policy, and model-configuration versions.
- Entity resolution, corporate action normalization, bitemporal data, and market calendar rules prevent future functions and wrong-security mapping.
- Source Connector Registry deepening remains auditable and does not grant access by itself.
- Product boundaries remain unchanged and no human UI or prohibited financial capability is added.