Using precomputed image features with VLMs

[ysulsky]

Hi all! I was wondering if anyone had thoughts about how I might precompute my own image features and pass them to SGLang for use in inference.

One idea would be to add a precomputed_features: Optional[torch.Tensor] field to MultimodalDataItem and a per-model:

def stack_precomputed_features(self, items: List[MultimodalDataItem]) -> torch.Tensor

That's passed into general_mm_embed_routine. Does that sound like a reasonable way to do this?

Thanks!
Yury

[ysulsky]

OK I have a proof of concept here: https://github.com/sgl-project/sglang/pull/6103/files!