GEOLIP-Bertenstein — multimodal Procrustes alignment prototype

An early test of a single question: can whitened Procrustes alignment build a usable shared space across independently-trained encoders of different modalities and different dimensionalities?

Four frozen experts — DINOv2-large, Whisper-large-v3, ESM-2-650M, CodeBERT-base — each pre-aligned into BERT-large's text space by a whitened Procrustes map, then fused by a single bidirectional transformer layer. 41.5M trainable parameters over ~11 minutes.

Status: archived precursor — superseded, retained for lineage

This is one of the preliminary Procrustes-whitening exercises that opened the alignment chain in this account. It predates geolip-captionbert-8192 (Mar 18), geolip-constellation-core (Mar 21), the geolip-vit-*-x3 line and geolip-vit-captionbank-coco (Jul 28), and clip-vitb-mini-distilled (Jul 30). It follows directly from procrustes-analysis (Mar 6), which profiled Procrustes alignability across 17 pretrained models in weight space; Bertenstein was the first attempt to apply the same idea in activation space across modalities.

A 2026-07-31 audit found the original evaluation protocol invalid, and the headline retrieval numbers on this card have been withdrawn and replaced below. The architecture is best understood as an alignment determinant — an instrument for deciding whether the approach carried — rather than as a deployable model. The answer it returned was largely negative, and it was not pursued because it did not yield the invariant elements needed for distillation, which is the direction this line went instead.

The negative result was local, not general. Later alignment work found Procrustes to be substantially more capable than this prototype indicated, and a corrected form of exactly this operation is now used as a standing instrument (see Lineage).

---

What it was for

The design goal was a shared multimodal space obtained without training the encoders: take independently-trained models with incompatible dimensionalities (768 / 1024 / 1280), Procrustes-align each into a common text-anchored frame, and let a small fusion layer do the rest. If that worked, it would be a cheap route to cross-modal association — and, by extension, to distillation targets that don't require running the experts at inference.

The intended pay-off was indirect association: train only text↔X pairs and get X↔Y alignment transitively through the shared text hub.

Corrected results

The original card reported R@1 ≈ 1.0000 across all expert pairs. Those figures came from an evaluation that encoded both modalities in a single forward pass and compared the two readouts taken from that same pass — so the two vectors had already attended to one another. Re-measured with independent encoding (the standard retrieval protocol):

measurement	value	note
text↔image, independent encoding	R@1 0.0007	N=4962 COCO val, chance 0.0002
text↔image, original joint-pass protocol	R@1 1.0000	withdrawn — see below
Procrustes map alone, out-of-sample	R@1 0.0088	N=4962, 43.5× chance; cos 0.112 vs 0.011 random
text↔{audio, code, protein}	withdrawn	not yet re-measured under the corrected protocol

Three findings from the audit, kept here so the record is complete:

1. Joint-pass evaluation. <TEXT> and <IMAGE> are read from one unmasked sequence, and the training objective compares the same two readouts. That objective is satisfiable without any cross-modal alignment. Diagnostics: R@1 stayed at 1.0000 from gallery size 64 to 4096 (real retrieval decays with gallery size), deliberately mismatched pairs scored identically to matched ones, and the two readouts agreed at cos 0.94 on pure-noise inputs.
2. The image_coco_test cache has no usable captions. It was built from COCO-Caption split="test", whose annotations are withheld; upstream encodes that as answer: ['None'], and the ingestion embedded the literal string. Every row is the same 3-token sequence. The "40K test" row was additionally computed on n=5000, not 40K. image_coco, audio_librispeech, code_csn and protein_p2t are unaffected and carry real text.
3. The alignment figures were in-sample. The Procrustes map is fit on the first 5,000 rows and was reported on those same rows. The rotation maximizes the very statistic being reported, and it has ~524k free parameters fit from 5,000 pairs; at that ratio a no-relationship null reproduces cos ≈ 0.39. The previously reported 0.377–0.440 figures should be read as in-sample, in the whitened frame.

Procrustes pre-alignment — corrected

Expert	dimension	reported (in-sample, whitened frame)	out-of-sample
audio	1280 → 1024 (PCA)	0.4404	not re-measured
code	768 → 1024 (pad)	0.4036	not re-measured
image	1024 → 1024	0.4107	cos 0.112, R@1 0.0088
protein	1280 → 1024 (PCA)	0.3771	not re-measured

The out-of-sample image figure is small but not null — it sits ~43× above chance and ~17× above a matched no-relationship null. Whitened Procrustes does recover real cross-modal structure here; it was the reported magnitude that was an artifact of the fit protocol.

Pentachoron CV

The original claim of a universal 0.20 ± 0.01 band across modalities does not hold as stated: CV is dimension-dependent. Measured on untrained normalized Gaussians — d=16 → 0.211, d=64 → 0.090, d=256 → 0.040, d=1024 → 0.021. This model reads CV ≈ 0.196 at a nominal d=1024, with an effective rank of ~20. That is consistent with the 0.20 band being the natural value on S¹⁵ and the embedding having collapsed to roughly 16–20 effective dimensions, rather than with convergence to a universal constant. Report effective rank alongside CV; CV alone cannot distinguish the two.

Architecture

                        ┌─────────────┐
                        │  Shared     │
    ┌──────┐           │  Fusion     │           ┌──────┐
    │ BERT │──text──→  │  Transformer│  ←──img──│DINOv2│
    │large │           │  (1 layer)  │           │large │
    └──────┘           │  1024-d     │           └──────┘
                        │  16 heads   │
    ┌──────┐           │             │           ┌──────┐
    │Whisp.│──audio─→  │  Procrustes │  ←─prot──│ESM-2 │
    │large │           │  pre-aligned│           │650M  │
    └──────┘           │             │           └──────┘
                        │             │
    ┌──────┐           │             │
    │Code- │──code──→  │             │
    │BERT  │           └─────────────┘
    └──────┘

Frozen encoders (not trained, not included here): BERT-large (336M) text hub, DINOv2-large (302M), Whisper-large-v3 encoder (1.5B), ESM-2-650M, CodeBERT-base (125M).

Trainable fusion (this repo): 41.5M — one shared transformer layer (8.9M) plus five expert modules (~7M each), including the Procrustes pre-alignment buffers.

How it runs. Each expert is centered, dimension-matched (PCA down or zero-pad up), whitened, rotated into text space, and un-whitened. Each modality gets a projection, a learned cross-attention pooling (257 image patches → 16 tokens; 1500 audio frames → 16), a <|MODALITY|> token and an output head. The fusion sequence is [<|TEXT|>] [text] [<|IMAGE|>] [img] … with bidirectional attention, and each modality's embedding is read from its special-token position. Loss: InfoNCE + a pentachoron volume term + a Procrustes alignment term.

Known defect in that description: because attention is unmasked across modalities and both embeddings are read from the same sequence, the <|TEXT|> readout is not a function of the text alone. Measured: it retains ~0.8% of itself when the attached image is swapped, and ~99.5% when the text is swapped. An untrained model shows the same asymmetry more mildly (34% / 59%), so this is a property of the unmasked joint sequence that training amplified. It also means the intended indirect association had no mechanism to travel on — alignments cannot compose through an anchor that isn't a function of its own modality.

Training

• Data: COCO-Caption (40K), LibriSpeech clean-100 (10K), CodeSearchNet Python (50K), Protein2Text-QA (15K)
• Schedule: 3 epochs, round-robin across experts, cosine LR 3e-4 → 1e-6, batch 256
• Hardware: NVIDIA RTX PRO 6000 Blackwell (102 GB) · Time: ~11 min (3 × ~220s)

Precomputed embedding caches (Arrow): AbstractPhil/bertenstein-v1 — configs image_coco, image_coco_test, audio_librispeech, code_csn, protein_p2t.

Lineage — what carried forward

• Whitened Procrustes cross-space alignment. The technique held; the fit discipline didn't. Its corrected descendant — an fp64 Procrustes frame fit with a held-out split — is now a standing instrument on the distillation line, where it recovers alignment that absolute gauges otherwise read as a floor. Procrustes turned out to be considerably more capable than this prototype suggested.
• Alignment to a reference member, with per-expert projectors. Bertenstein's soup-style construction is the direct ancestor of geolip-captionbert-8192's cached consensus, which aligns heterogeneous frozen encoders to a reference member rather than to a running mean — the property that lets it fuse experts of differing dimension.
• The move toward distillation. What this prototype did not produce was an invariant object stable enough to distill against. That gap is what redirected the line toward consensus targets and student distillation (geolip-vit-captionbank-coco, clip-vitb-mini-distilled), where the target is cached and model-free at training time.
• Evaluation discipline. Three checks now run by default upstream of any retrieval number: verify the query set actually varies; never score an alignment map on the rows it was fit to; report effective rank beside CV.

Possible return

The idea is not closed. What it would need:

1. Independent encoding — separate passes, or an attention mask forbidding cross-modal attention during the contrastive computation, so the readouts are functions of their own modality.
2. A held-out fit for the alignment maps, and a smaller/regularized map: ~524k free parameters from 5,000 samples is far past where in-sample cosine means anything.
3. Data that can actually test transitivity — items present in two or more non-text modalities (video gives audio + frames of the same event; a protein with both sequence and structure encoders). The four corpora here are disjoint, so the central claim was never evaluable on its own data.
4. A caption-validity check at ingestion — the image_coco_test failure was a validity flag derived from the image alone, letting empty and placeholder captions through.

Files

geolip-bertenstein/
├── checkpoints/
│   ├── epoch_001/ · epoch_002/ · epoch_003/
│   └── final/          model.safetensors · loss.safetensors · training_state.pt
│                       config.json · aligner_{audio,code,image,protein}.safetensors
├── tensorboard/
├── bertenstein_results.json      # original metrics — see Corrected results above
├── cell1_prepare_data.py         # Stage 1: encode + cache
├── cell2_prototype_model_trainer_v1.py   # Stage 2: align, train, evaluate
└── README.md

checkpoints/final/model.safetensors contains 102 tensors; the Procrustes aligner buffers ship inside it as experts.<name>.aligner.* as well as separately. Class definitions (BertensteinFusion, ExpertModule, ProcrustesAligner, FusionConfig) live in cell2_prototype_model_trainer_v1.py — earlier revisions of this card referenced a stage2_bertenstein.py, which was never part of this repo.

Related

• AbstractPhil/procrustes-analysis — the weight-space profiling this grew out of
• AbstractPhil/geolip-captionbert-8192 — reference-member consensus, the successor construction
• AbstractPhil/clip-vitb-mini-distilled — where the distillation direction landed

Citation

@misc{abstractphil2026bertenstein,
  title={GEOLIP-Bertenstein: A Multimodal Procrustes Alignment Prototype},
  author={AbstractPhil},
  year={2026},
  note={Archived research precursor; evaluation protocol corrected 2026-07-31},
  url={https://huggingface.co/AbstractPhil/geolip-bertenstein}
}

License

MIT