# V-JEPA-LM Aims to Help Video AI Read Motion

> A new open video-language model uses transformed clips to teach AI about changes such as speed and direction. Its authors report higher scores than other open models in its size class.

Oossa · 2026-11-16 · https://oossa.com/en/v-jepa-lm-aims-to-help-video-ai-read-motion

Researchers have introduced V-JEPA-LM, a family of models designed to answer questions about how a video changes over time—not just what objects or scenes it contains. The models pair a frozen V-JEPA 2.1 video encoder with a language model from Qwen or Llama, then train them using ordinary video-and-text examples plus transformed clips whose answers change in a controlled way.

The team says those transformations can teach motion without requiring new human-written labels. For example, a clip can be reversed or played at a different speed, creating a version where the answer to a question about direction or speed must change. The models were trained on 4.53 million public samples in one main training stage, using 32 video frames per clip.

## Why focus on motion?

The researchers argue that captions and question-answer examples often mention only a few details in a clip. A model can learn to find those details without making use of the video encoder’s information about movement. In one test, a simple probe could read speed from frozen V-JEPA features with 87% accuracy, while a language model trained on conventional video-and-text examples answered speed questions at roughly chance level.

Their proposed fix is counterfactual supervision: present two related clips with the same question but different answers because the video was changed. The paper says this encourages the language model to attend to the change itself. In a matched-budget comparison, the team reports that this training added 4.5 points on average and 38 points on speed questions in natural-video benchmarks.

## How did it perform?

The researchers report that their Qwen3-4B version scored 72.14 across 12 scores from 11 video benchmarks. That was higher than the 61.08 reported for the strongest open 4B-class model, Molmo2-4B, and the team says its models led every size class they tested. The evaluation used the same clips, prompts and 32-frame setting for the models compared. These are results reported by the model’s creators; independent validation is not described.

The Qwen3-4B weights and code are publicly available, and the team also provides a browser demo of its Llama 3.1 8B version. The Qwen3-4B download is about 10.1 GB, and its model card says local use needs a GPU with at least 16 GB of memory. Although that model’s language-model base is Apache-2.0 licensed, some training data carry a Llama 3.2 license, so the repository lists additional license terms.

## The facts

- V-JEPA-LM uses a frozen V-JEPA 2.1 video encoder with Qwen or Llama language models ranging from 1B to 8B.
- The team reports training on 4.53 million public samples, with 32 video frames per clip.
- The Qwen3-4B version scored 72.14 across 12 scores from 11 benchmarks, compared with 61.08 for Molmo2-4B in the reported evaluation.
- The Qwen3-4B weights are about 10.1 GB; the model card says local use needs at least 16 GB of GPU memory.

## Why it matters

For developers with a suitable GPU, public weights offer a way to try a model built to answer questions about motion, timing and changes in a clip. The reported benchmark lead is promising, but it does not establish how well the model will handle everyday videos or questions outside those tests.

## Sources & references

1. [V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations](https://arxiv.org/abs/2611.17831) – arXiv cs.CV, 2026-11-16
2. [Llama-V-JEPA-LM GitHub repository](https://github.com/kkt94/Llama-V-JEPA-LM) – Llama-V-JEPA-LM GitHub repository
3. [Keuntae Kim, Hanyang University, Artificial Intelligence Lab](https://huggingface.co/kkt94/Llama-V-JEPA-LM-Qwen3-4B) – Keuntae Kim, Hanyang University, Artificial Intelligence Lab

Last updated: 2026-11-16
