BiliSakura/SkySense-transformers

SkySense Transformers

HuggingFace-compatible checkpoints for the SkySense (CVPR 2024) foundation model backbones.

Checkpoints

Directory	Modality	Architecture	Source
skysense-swinv2-huge-rgb	High-res optical (RGB)	SwinV2 Huge	skysense_model_backbone_hr.pth
skysense-vit-large-s2	Sentinel-2	ViT-Large	skysense_model_backbone_s2.pth
skysense-vit-large-s1	Sentinel-1 SAR	ViT-Large	skysense_model_backbone_s1.pth

Each subdirectory is a self-contained HuggingFace model repo with remote code (trust_remote_code=True).

Usage

from transformers import pipeline
import torch

# HR RGB backbone — input 224×224
hr_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-swinv2-huge-rgb",
    trust_remote_code=True,
    device="cpu",
)
hr_img = torch.randn(1, 3, 224, 224)
features = hr_pipe(hr_img)
print(features["last_hidden_state"].shape)  # (1, 2816, 7, 7)

# Sentinel-2 — 10 bands, 64×64
s2_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-vit-large-s2",
    trust_remote_code=True,
    device="cpu",
)
s2_img = torch.randn(1, 10, 64, 64)
features = s2_pipe(s2_img)
print(features["last_hidden_state"].shape)  # (1, 1024, 16, 16)

# Sentinel-1 — VV/VH, 64×64
s1_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-vit-large-s1",
    trust_remote_code=True,
    device="cpu",
)
s1_img = torch.randn(1, 2, 64, 64)
features = s1_pipe(s1_img)
print(features["last_hidden_state"].shape)

Conversion

Source project: /home/czy/local/projects/SkySense-transformers

conda activate rsgen

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_hr.pth \
  --modality hr \
  --output-dir /path/to/skysense-swinv2-huge-rgb \
  --clean-output

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_s2.pth \
  --modality s2 \
  --output-dir /path/to/skysense-vit-large-s2 \
  --clean-output

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_s1.pth \
  --modality s1 \
  --output-dir /path/to/skysense-vit-large-s1 \
  --clean-output

The converter also accepts unified pretraining checkpoints with backbone_gep.* / backbone_s2.* / backbone_s1.* prefixes.

Notes

• HR conversion skips Swin relative-position buffers (relative_position_index, relative_coords_table) and mask_token. Buffers are deterministically recomputed at init; learned CPB weights are loaded.
• ViT checkpoints use per-layer ln1/ln2 keys remapped to norm1/norm2.
• Swin FFN keys ffn.layers.0.0 → ffn.layers.0, ffn.layers.1 → ffn.layers.3.
• HR Swin uses pad_small_map=True so 224×224 inputs work with window size 8 at deep stages.