JEPA · explained simply

“You will never get
to real intelligence
by predicting words.”

That is Yann LeCun’s argument in one line. He is Meta’s chief AI scientist, he won the Turing Award for deep learning, and he thinks the thing everyone is currently building is a detour. JEPA is what he proposes instead.

This page explains both — the argument, and the architecture — without the maths.

Start here

A teenager learns to drive in about twenty hours.

Nobody shows them a billion miles of footage. They sit in a car, and after roughly twenty hours of practice they can drive — in rain, at night, past a cyclist they have never seen before.

Self-driving has taken decades, tens of billions of dollars, and enormous fleets collecting data, and it still isn’t there. LeCun’s point isn’t that engineers are bad at it. It’s that the teenager is doing something categorically different, and we are not building that thing.

~20 h
for a human to learn the basics of driving
decades
and billions of dollars, still not at human level
~960 h
of waking visual experience a 4-month-old baby has had

That last number is the interesting one. In four months a baby has seen roughly 960 hours of video through its own eyes — about the size of a standard AI video dataset. And by then it already understands that objects still exist when hidden, that solid things don’t pass through each other, that some things move on their own and some don’t.

The teenager isn’t learning to drive from twenty hours of data. They’re learning to drive from twenty hours plus sixteen years of understanding how the world works.

That accumulated understanding is what LeCun calls a world model. It is the thing that lets you know, without trying it, that driving into the barrier would be bad. Today’s systems have to be shown. That, in his view, is the whole problem.

The argument

Six things LeCun is actually claiming.

These get flattened into “LeCun says LLMs are useless,” which is not what he says. He says they are useful and they are not the road to human-level intelligence. Here is the actual chain.

1

Hallucination isn’t a bug to be patched out

An LLM is trained to produce likely text, not true text. When it doesn’t know, the most likely-sounding continuation is still a confident sentence. Truth was never the target.

2

Errors compound, one word at a time

Each word carries some chance of being wrong, and the model is committed to it once it’s out. Over a long answer that risk multiplies. There’s no step where it steps back and checks the whole plan.

3

They “speak without thinking”

Humans work out roughly what they mean, then say it. An LLM must emit a token at every step — it has no private, variable-length place to think first. Chain-of-thought is thinking out loud, which helps, but it is still thinking in words, in public, one word at a time.

4

The compute you spend is fixed, not matched to the problem

A hard question and a trivial one go through the same number of layers. Humans spend more thought on harder things. That flexibility — deciding how long to think — is missing.

5

Text alone is too thin a signal

Language is a compressed summary of experience, written by people who already share a reality with the reader. Learning only from the summary means never learning the thing being summarised. Babies get there without reading anything.

6

What’s missing is a world model you can plan inside

Something that can answer “if I do this, what happens?” fast enough to try a hundred options in your head before moving. That, he argues, is what common sense is — and it is the part nobody has built.

His position paper is called “A Path Towards Autonomous Machine Intelligence” (2022). JEPA is the piece of that path he has actually specified in detail.

The core of it

Predict the gist, not the pixels.

If there is one thing to take away, it is this. Everything else in JEPA is machinery around this single decision.

Suppose you show a model the first half of a video and ask it to predict the rest. A generative model — the Sora or GPT approach — tries to produce the actual next frames, pixel by pixel. That is a hopeless target, for two separate reasons.

THE GENERATIVE WAY predict every pixel of what comes next what you’ve seen generator draws it all a blurry average Problem one: it’s unknowable. A hundred different futures are all plausible. Forced to pick one, the model hedges — and the average of many futures is mud. Problem two: most of it doesn’t matter. Predicting a park means predicting every blade of grass. Almost all the effort goes into detail nobody needed. THE JEPA WAY squash both halves down to their gist — then predict the gist what you’ve seen encoder the gist predictor guessed gist Both problems disappear. “A person walks left” has the same gist whichever exact pixels happen. And the grass never enters it at all — the encoder drops whatever it can’t use.
“Gist” is the everyday word for what the papers call a representation, embedding or latent: a short list of numbers standing for what’s in the scene, with the unimportant detail dropped.

Generative

Judged on reproducing the input exactly. Must model everything, including what doesn’t matter. Can’t represent “many things could happen” except by blurring them together.

Joint-Embedding Predictive

Judged on predicting a compressed description. Free to discard detail. “Many things could happen” collapses naturally, because many futures share one gist.

That’s the whole name, by the way. Joint-Embedding: both halves get embedded into the same space. Predictive: one half predicts the other. Architecture: it’s a shape you can build lots of models in, not a single model.

The machinery

What one training step actually does.

Concretely, for images (this is I-JEPA, the first real one). Take a photo, hide most of it, and make the model guess the gist of the hidden parts from the bit it can see.

1 · ONE IMAGE CONTEXT you may look TARGETS blue = visible · orange = hidden, must be guessed 2 · TWO PATHS context encoder predictor “+ where it is” guessed gist target encoder the encoder’s gist of the hidden part 3 · THE LESSON how far apart are they? that distance is the error to shrink Only the blue path learns from the error. The orange encoder is a slow copy of the blue one, trailing behind on purpose. That lag is what stops the cheat in the next section. The context block is kept deliberately small, and anything it overlaps with a target is blanked out. If the task were easy, the model would learn something shallow — like matching textures. Making it hard is what forces it to learn what’s in the picture.
The predictor is told where each hidden patch is but not what’s there — so it must answer “given this corner, what sort of thing is over there?” Video (V-JEPA) is the same picture with time as a third axis: the mask is a shape punched through a stack of frames.

Note carefully what the orange box is not: it is not a ground truth. There is no true gist of a photo sitting anywhere. It is simply what the model’s own encoder says about the hidden part — which is exactly why the cheat in the next section is possible at all.

Note what never appears: no labels. Nobody tagged these photos. The image supervises itself by having a piece of it hidden. That’s what self-supervised means, and it’s why this can eat raw video off the internet.

The catch

There is an obvious way to cheat, and it nearly killed the idea.

If you score a model on predicting its own summaries, it can win by making every summary identical. Predict a constant, score a perfect zero, learn nothing. This is called collapse, and avoiding it is most of the practical difficulty.

WHAT YOU WANT a dog running a cup falling a door opening a wave Different scenes land in different places, so the position means something. WHAT IT DOES IF YOU LET IT everything, same point Prediction error: zero. Perfect score. Information retained: also zero. THE TWO DEFENCES Show it wrong answers Push mismatched pairs apart. Simple, but you need absurdly many of them as detail grows. Rig it so cheating can’t pay Make the target a lagging copy of the student, so the goalposts move whenever it tries to stand still.
JEPA takes the second route. The target encoder is a slowly-updated average of the context encoder, so the “right answer” drifts behind the student and a constant answer never settles. The same trick powers BYOL and data2vec.  → This one trips everybody; there’s a whole page on it.

And one reframe worth knowing

LeCun prefers not to ask “what happens next?” at all. He asks “is this a plausible next?” — scoring a pair as low energy if they fit together and high if they don’t.

The difference matters. There are a hundred plausible next seconds, so producing one is guesswork. But judging whether a proposed one is sane is a well-posed question — and judging is exactly what you need when you’re trying out possible actions in your head. Collapse, in this language, is a model that calls everything plausible.

The bigger picture

JEPA is one organ. This is the animal.

The world model is one box in a larger design LeCun sketched for a complete agent. Worth seeing, if only to know how much of it is still a drawing.

CONFIGURATOR sets the task — tells every other box what to care about right now. How it would learn to do this is unsolved. PERCEPTION what’s happening right now WORLD MODEL “if I do this, then what?” ← this is where JEPA goes COST how bad is that outcome? hardwired part + learned part the hardwired part can’t be edited ACTOR proposes what to do, then does the best one try another action — all of this happens in its head, before moving MEMORY keeps past states and how bad they turned out, so the learned half of “cost” has something to learn from. THE SAFETY ARGUMENT Because the agent picks actions by scoring them against a cost you wrote, you can make some of those costs unmodifiable. Safety by construction, not by asking nicely in a prompt.
Almost none of this is built. It’s a specification, published in 2022, of what LeCun thinks the parts list looks like. JEPA is the only box with real working code in it.

Fast mode and slow mode

The actor runs in two modes, borrowed openly from Kahneman’s System 1 and System 2.

Mode 1 · reflex

See it, do it. One pass, no deliberation. Fast and cheap, and roughly what every AI system today does — including LLMs.

Mode 2 · imagine first

Propose a sequence of actions, run them forward in the world model, add up the cost, adjust, repeat. Only then move. This is planning, and it’s what’s missing.

The nice detail is how they connect: once Mode 2 has solved something enough times, a Mode 1 reflex can be trained to shortcut it. That’s learning a skill — a new chess player calculates every line; a strong one just sees the move.

Reality check

What has actually been built.

Between the 2022 paper and now, four real models. The pattern is clear: each one makes the representations better, and until the most recent one, none of them did any planning at all.

modelyearwhat it added
I-JEPA2023Images. Proved the idea works: guess the gist of hidden patches from a visible one. Learned good features faster than the methods it was compared against.
MC-JEPA2023Motion. Learns how things move alongside what they are, and is arguably the first thing with a layered structure like the proposed hierarchy.
V-JEPA2024Video — about 3 seconds of it. The same masking trick punched through a short stack of frames.
V-JEPA 22025Scale, and finally a loop that closes: 1B parameters, 22M videos, ~16 seconds of context — plus a robot arm that plans with it.
2M → 22M
training videos, V-JEPA 1 to 2
3 s → 16 s
how much video it can hold at once
62 h
of robot footage to learn to plan arm movements

V-JEPA 2 is the one that matters

Everything before it learned to see. V-JEPA 2 is the first that uses what it sees to decide, and it does it in the shape the 2022 paper described.

The recipe: freeze the video encoder. Train a second network that takes “here’s the scene, here’s the move I’m considering” and predicts the gist of the scene afterwards. Now give it a photo of the goal. At run time, it tries out candidate action sequences internally, scores each by how close the imagined end state lands to the goal, and executes the winner — on a real robot arm, with no training data for that specific task.

That is Mode 2 planning, running on a JEPA world model. Small, clumsy, on a tabletop arm — but it is the loop the whole argument was about, closed for the first time.

And what’s still missing

It fills in gaps, it doesn’t see ahead

V-JEPA hides a patch in the middle of a clip and reconstructs it. That’s not the same as watching ten minutes and predicting the next one. Real anticipation across time is the open problem.

The hierarchy is still a drawing

The plan calls for layers — seconds at the bottom, plans at the top. What exists has two levels at most, and nobody knows how many you actually need, or whether it should vary.

Most of the blueprint has no code

The configurator, the memory, the learned critic, breaking a goal into subgoals — all still prose.

It’s eyes-only

No sound, no touch, no language in the loop. A world model for a creature that only sees.

Decoder ring

The words, in English.

Representation · embedding · latent
Three words for the same thing: a short list of numbers standing in for something complicated. The “gist”.Near-interchangeable in practice; “latent” hints that it’s internal and hidden.
Self-supervised learning
Training with no human labels, by hiding part of the data and predicting it from the rest. The data supervises itself.
Energy-based model
A model that scores how well two things go together rather than generating one from the other. Low energy means compatible.“Energy” is borrowed from physics; it just means a score you want to be small.
Collapse
The failure where a model gives everything the same answer and scores perfectly while learning nothing.
World model
Whatever lets you answer “if I do this, what happens?” without doing it. The thing that makes you flinch before the glass hits the floor.
Latent variable
A dial of randomness you add so one input can produce several plausible outputs, instead of one blurry average of them.
Vision Transformer (ViT)
The standard image network here. It chops a picture into patches and treats them like words in a sentence — which is why masking works the same way it does in language.
Probing
How you test what a model learned: freeze it, bolt a tiny classifier on top, and see how much useful information was sitting in those numbers.