Skip to content

Version 1.1.8 · Last updated

Connectors

Connectors fetch external source records and preserve enough metadata for normalization, licensing, evidence, and audit. They are operator capabilities, not an ungoverned data-import shortcut.

Supported source kinds and providers

The registry recognizes security master, market data, fundamentals, news events, regulatory filings, IR materials, corporate actions, market calendar, macro reference, ownership/insider, and sentiment/alternative-event source kinds. Implemented provider families include local files, SEC EDGAR, FRED/ALFRED, OpenFIGI, GDELT, and Polygon aliases.

The supplied config/connectors.yaml enables governed SEC filing/XBRL, FRED, and OpenFIGI sources. Removed providers have no runtime adapter, registry entry, credential requirement, or compatibility route.

Vendor news preserves the supplied title and original summary/body without removing analyst language. Normalized event records label source material with content_type, is_fact, publisher, published_at, original_text, and trust_level. Analyst opinions are explicitly marked is_fact=false and trust_level=third_party_opinion; client Agents decide whether and how to use them.

Registry fields

Field Meaning
source_id Stable operator-facing source identifier
kind Neutral connector domain
provider Supported adapter family
enabled Whether validation and synchronization may use the source
license_id License policy evaluated before use
allowed_uses Purposes declared for this source
endpoint / file_path Exactly one external endpoint or local file location
auth_env_var Name of the environment variable containing the provider credential
rate_limit Requests per minute and burst
parser_version Version of source parsing semantics
field_mapping_version Version of source-to-neutral field mapping

Local-file paths must remain inside the connector registry directory. The registry rejects path traversal, missing files, unsupported providers, missing licenses, empty allowed-use lists, absent secrets, and known placeholder secrets.

Validate before synchronization

argus connector validate \
  --api-key "$ARGUS_MACHINE_API_KEY" \
  --institution-id institution:customer \
  --caller-id service:connector-operator

The current runtime keeps the validation implementation but configures data_license.enforcement_enabled: false and connectors.fail_closed_on_license_error: false. Registry shape, credentials, schemas, bounded requests, normalization, persistence, and audit checks remain active; license and redistribution decisions are recorded but do not block Agent calls. Restore both switches to true to recover fail-closed license behavior.

Synchronize a bounded window

EODHD real-time quote and trade sources use a 240-second capture window by default. A dropped or idle WebSocket is reconnected with bounded exponential backoff; every new connection is re-authorized and re-subscribed before records are accepted. Replayed observations are deduplicated by stable observation ID, and an exhausted reconnect budget after receiving data is reported as a partial window. This is bounded capture, not a permanent consumer or an uninterrupted Quote/Trade tape. Operators can lower capture_seconds for smoke tests. EODHD's trade dp dark-pool flag and ms market-status field are not stored because MarketObservation v1 has no semantically matching fields; adding them requires a versioned contract change rather than overloading venue or condition fields. Reaching max_events before the capture deadline is reported as a partial window, so a resource-capped sample is never described as a complete window.

argus connector sync \
  --source-id sec_edgar_filings \
  --kind regulatory_filings \
  --api-key "$ARGUS_MACHINE_API_KEY" \
  --institution-id institution:customer \
  --caller-id service:connector-operator \
  --from 2026-06-01T00:00:00Z \
  --to 2026-06-30T23:59:59Z \
  --symbols AAPL \
  --company-ids 0000320193 \
  --dry-run

SEC EDGAR filing and XBRL sources require a CIK through --company-ids; a ticker alone is not sufficient. For a single-company request, keep the ticker and CIK lists aligned as shown above.

For SEC XBRL, from and to select the filing's conservative public-known time, including both boundaries. A filing date without a timestamp becomes the end of that day in New York, with daylight-saving time applied. A record without a filing date uses its actual observation time. This window does not filter the measurement period: a new filing can report older periods.

The prepared sec-edgar-xbrl-v2 parser and sec-edgar-xbrl-fields-v2 mapping preserve decimal lexical values and expose numeric .val facts with their actual taxonomy, concept, unit, currency when supplied, and instant or duration period. Quarterly and annual durations, currencies, and companies remain distinct. A later visible filing replaces the same measurement aspect; a late capture does not become visible before ingestion. Cross-source conflicts require matching taxonomy, concept, period, and unit; unknown legacy aspects are not treated as comparable measurements.

Versioned normalization retains existing records. The bounded operator reconciliation tool can verify the published SEC v1 parser/mapping pair against its original capture; it does not enrich old rows with v2 aspects, insert missing facts, or guess unknown historical versions. Migration 0062 stores the actual configured filing-source license snapshot for new ingestion. Historical documents with no snapshot remain explicitly unverified; the current policy is not evidence of their historical terms.

These source changes require deployment and data backfill before they describe production coverage. A sample's successful numeric and evidence checks do not prove the promised company, field, and historical universe. Missing provider concepts remain gaps; they are not replaced with similarly named measures.

Use dry_run to prove selection, provider access, parsing, and policy without committing normalized records. Local CLI execution, and REST execution in local or test, waits for the task to reach a terminal state; the returned task result contains the connector result and its run_id. In staging and production, the REST adapter dispatches the task through Celery and initially returns a task_id. That task_id is not a connector run_id and must not be passed to connector run-status or GET /v1/connector-runs/{run_id}.

Poll the returned Argus task id with connector task-status --task-id <task_id> or GET /v1/connector-tasks/{task_id}. The endpoint requires connectors:read, enforces institution isolation, and only exposes connector synchronization tasks. Do not report a synchronization as complete from the initial queued response. Wait for a terminal task status; on success, read output_payload.run_id and then inspect the persisted connector result with connector run-status --run-id <run_id> or GET /v1/connector-runs/{run_id}. If a run fails or only partially succeeds, its task is terminal failed and retains output_payload.run_id and the connector counts. Inspect that run and its source gaps before a bounded operator retry. A successful worker execution alone does not make an incomplete connector result successful. Celery's separate worker task id is diagnostic metadata and is not accepted by either endpoint.

An operator may provide an optional idempotency_key (1–128 characters, not blank) in the REST sync envelope or --idempotency-key in the CLI. Keep that key stable when repeating a submission: the same identity, key and request return the existing task. Reusing a key for a different request is rejected (REST 409). After a terminal failure has been investigated and corrected, use a new key to submit the same bounded source selection again. The failed task and its audit remain available; a new key does not reset it. Without an explicit key, the existing identity-and-payload deduplication remains unchanged.

The captured SEC ticker/exchange snapshot also registers its source-confirmed CIK and issuer name as a filing parent in the same transaction as the normalized facts. It verifies the stored raw checksum and issuer identity, preserves an existing company, and retains the raw source through a foreign key. Country and industry remain null when the source does not provide them; no security, MIC, industry hierarchy or historical listing interval is inferred. The public source-backed company reader continues to report these coverage gaps. The legacy complete-master reader excludes incomplete parents, which do not qualify as complete entity/security mappings. Direct external-fact repository writes alone do not register company parents.

Migration 0065_partial_company_identities preserves existing complete company rows and permits these explicit missing fields. Downgrading to the old required fields fails while partial parents remain; restore the verified pre-migration backup for rollback rather than inserting invented country or industry values. The key is task metadata, not a provider parameter or an authorization grant.

The REST synchronization payload also accepts a bounded parameters object for source-specific selections such as a macro series, table, year or record limit. For example, the configured no-key bea_nipa_flat_file source accepts {"TableName":"T10105","Year":"2025","Frequency":"A"}. Provider adapters still enforce their allowed parameters and source policies. Credentials, source policy files, URLs and task identity or authorization fields cannot be supplied through this object. Synchronization is an operator function requiring connectors:sync; ordinary public OAuth users do not receive that permission.

Raw and normalized data

In the prepared production connector, sec_edgar_filings also publishes an idempotent copy of a verified official SEC document into the configured OAuth data institution. Source distribution checks follow the server's default-off license switch. Explicitly enabling it requires the captured grant to authorize that institution and factual lookup. The archive URL, issuer CIK and original content SHA256 are always verified. Disabled enforcement leaves unknown rights metadata intact; failed financial-report extraction still fails the record. The ordinary data copy retains the same source bytes, times and license snapshot, and uses the existing governed public projections.

Machine credentials, tasks, connector runs and publication audits retain their original operator owner. Ordinary users receive neither connector permissions nor access to those protected records, and all filing repository tenant filters remain active. The publication target comes from server OAuth configuration, not synchronization parameters. Other sources and private uploads are not published this way. See ADR 0003. This prepared path does not establish full issuer/history coverage or claim that the signed production release and ordinary-user acceptance have completed.

The captured license terms participate in filing ingestion idempotency. A new verified grant for the same source document creates a separate immutable metadata capture, retaining the original bytes, document version and publication time. Earlier captured terms remain unchanged. Re-delivery under identical terms reuses that capture; list ordering in equivalent terms does not create another copy. The new record's creation time still prevents visibility before capture.

Migration 0066_filing_capture_versions preserves existing records, replaces the obsolete byte-only unique constraint with a scoped capture index, and retains the unique idempotency key. Broad fragment searches select the latest visible capture per source document in the caller's data institution. Explicit document IDs retain the original capture and its grant; a later grant never changes historical evidence. Source/recorded times apply to both capture selection and fragments. An unparsed latest capture cannot silently fall back to an old one. Downgrade refuses duplicate byte identities; restore the paired backup rather than delete captured terms.

Public filing_search and filing_evidence_extract return at most ten matching fragments. The prepared readers probe one additional match and return partial with filing_fragment_result_limit and structured_output_flags.filing_fragment_limit_reached=true when more matches exist. Refine the query or use a disclosed document ID with filing_evidence_extract to narrow its scope. There is no public cursor parameter for these tools, and a bounded fragment response does not establish complete filing-history coverage. Captured source-license gaps remain separately visible.

The prepared WDI macro_series reader preserves a finite captured value as Decimal and retains its annual period and evidence. A unit is taken only from the same source capture; an empty unit remains absent, with macro_source_unit_unavailable. Neither the indicator name nor another capture supplies a substitute currency. Current WDI captures contain decoded JSON values rather than the original HTTP number token. Results therefore remain partial with macro_numeric_http_lexical_precision_unverified, and macro_numeric_http_lexical_precision_verified is false. These prepared changes are not a claim of deployed coverage, complete country/history coverage, or verified revision/vintage semantics.

Connectors first produce ConnectorRecord values with source, kind, record number, payload, and observed time. Normalizers then create neutral market, event, filing, corporate-action, or evidence records. Provider-specific restricted fields are removed or masked before a public package is assembled.

Required permissions and secrets

Listing, validation, task status, and run status require connectors:read; synchronization requires connectors:sync. Provider credentials use the configured environment names such as SEC_USER_AGENT, FRED_API_KEY, and OPENFIGI_API_KEY. Never store their values in connectors.yaml.

Live testing

Live tests are off by default and require explicit CONNECTOR_LIVE_TESTS_ENABLED or production/staging test configuration plus real credentials. They can consume provider quota and should run with bounded symbols/time windows. Unit and integration tests otherwise use controlled adapters and do not require the network.