Multimodal perception

Why a perception layer

Text RAG remembers what was written. But an operational system produces and receives far more than text: process videos, photos of drawings, screenshots, sensor time series, measurement tables. A semantic OS that only perceives text is half blind.

Nika OS treats perception as a primitive, with the same discipline as the rest of the system: versioned embeddings, quality gates, and a unified query space.

Predictive embeddings (the JEPA family)

For non-textual perception, Nika OS builds on the JEPA family (Joint-Embedding Predictive Architecture, public research from Meta AI): encoders that learn by predicting representations in latent space rather than reconstructing pixels.

• Video — a V-JEPA encoder turns a clip into latents that capture both content and temporal dynamics. The checkpoint’s generative predictor additionally allows predicting a future latent and comparing prediction against reality.
• Time series and tabular — the same principle applies to sensor data and measurement tables: predictive embedding, similarity retrieval, regime detection.

These encoders run on serverless GPU on demand (dispatched through the compute primitive), billed by the second of inference — zero cost at idle.

Projectors: one query space

The classic multimodal problem: each modality lives in its own vector space, so each modality demands its own index and its own queries. Nika OS solves this with projectors: trained adapters that map perception latents into the same 384d space as textual memory.

The direct consequence: a text question can retrieve a video moment, a drawing image, or a time-series episode — through the same WATERFALL RETRIEVAL cascade as the rest of memory. Perception becomes queryable via the search_perception tool, just like any other knowledge.

flowchart LR
    V["🎥 Video"] --> E1["V-JEPA encoder<br/>latents"]
    T["📈 Time series"] --> E2["JEPA encoder<br/>latents"]
    E1 --> P["Trained projectors<br/>→ 384d text space"]
    E2 --> P
    Q["❓ Text question"] --> S["Unified space<br/>Qdrant"]
    P --> S
    S --> R["Cross-modal retrieval"]

    classDef modal fill:#F3EADE,color:#2C3E42,stroke:#7DB5A5,stroke-width:1.5px;
    classDef enc fill:#FAF5EC,color:#2C3E42,stroke:#A86640,stroke-width:1.5px;
    classDef proj fill:#E99971,color:#FDFBF8,stroke:#C97A55,stroke-width:2px;
    classDef store fill:#7DB5A5,color:#FDFBF8,stroke:#5E9384,stroke-width:2px;
    class V,T,Q modal;
    class E1,E2 enc;
    class P proj;
    class S,R store;

Release discipline: manifest, golden check, canary

A perception model is a component that can regress silently. Nika OS therefore applies to perception the same rigor as to a software deployment:

Mechanism	Role
Versioned manifest	Every projector/encoder version is declared in a manifest (perception_versions): model, dimensions, training dataset, metrics
Fail-closed golden check	A frozen reference set must reproduce its scores at every release. Failure or missing baseline → the release stops; it never “skips”
Canary + rollback	A new version first serves a fraction of traffic; observed regression → automatic rollback to the previous version

The fail-closed principle is central: a quality gate that, lacking a baseline, lets things through by default is a worse bug than the regression it was meant to catch — because it turns a visible regression into a silent one.

Honest benchmarking as doctrine

Every perception building block must earn its place against a baseline, on real data, and defeats are recorded alongside victories. Two examples of verdicts actually on file:

• On a public predictive-maintenance benchmark, the tabular JEPA embedding wins in the label-scarce regime (≤ 50 annotated examples), but classic gradient boosting dominates from ~100 labels on. Verdict kept: the primitive is routed to few-label cases only.
• On video, the embedding captures temporal order where a “bag of frames” cannot — but the content signal remains dominant. Verdict: dynamics live in the predicted latents, not in an average.

These negative verdicts are not failures: they are routing rules. The algorithms as tools doctrine applies to perception as everywhere else — the best tool for the observed data regime, not the newest one.

See also

• Memory system — the 384d text space the projectors land in, and multimodal RAG.
• Meta-curator and evolution — who decides to retrain perception models, and behind which gate.
• Doctrines — algorithms as tools and antifragility apply to perception choices.