Draft:JEPA

Draft article not currently submitted for review. This is a draft Articles for creation (AfC) submission. It is not currently pending review. While there are no deadlines, abandoned drafts may be deleted after six months. To edit or make changes to this draft, simply click on the "Edit" tab at the top of the window. To be accepted, a draft should: Show the subject qualifies for a Wikipedia article by using multiple sources that meet four criteria. The sources should be (1) reliable (2) secondary (3) independent of the subject (4) talk about the subject in some depth. For some topics, there are alternative criteria. Be written from a neutral point of view Respect copyright and do not plagiarize. Do not copy-paste. It is strongly discouraged to write about either yourself or your business or employer. If you do so, you must declare it. Where to get help If you need help editing or submitting your draft, please ask us a question at the AfC Help Desk or get live help from experienced editors. These venues are only for help with editing and the submission process, not to get reviews. If you need feedback on your draft, or if the review is taking a lot of time, you can try asking for help on the talk page of a relevant WikiProject. Some WikiProjects are more active than others so a speedy reply is not guaranteed. How to improve a draft Wikipedia:Contributing to Wikipedia – a basic overview on how to edit Wikipedia. Help:Wikitext – how to use the markup Help:Referencing for beginners – how to include references Wikipedia:Article development – how to develop your article Wikipedia:Writing better articles – how to improve your article Wikipedia:Verifiability – make sure your article includes reliable third-party sources You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Add tags to your draft Editor resources Easy tools: Citation bot (help) | Advanced: Fix bare URLs Last edited by Colorado001 (talk | contribs) 13 days ago. (Update) Submit the draft for review!

JEPA (Joint-Embedding Predictive Architecture) is a type of self-supervised, non-generative, non-contrastive AI architecture which predicts embeddings. The concept was popularized by Yann LeCun.

The term "Joint-Embedding Predictive Architecture" was coined by Yann LeCun in a 2022 "position paper".^[1][2] LeCun's paper is built on past work involving world models, or models with internal representations of the physical world;^{[1]: 2 [3][4]} LeCun claimed that in addition to representing the dynamics of the world, an autonomous intelligence must also have the capacity to understand and predict future actions.^[5] He also argued systems like large language models are lacking in common-sense understanding of the real world due to their lack of physical interaction.^{[1]: 45 [6]: 37 [7]}

JEPA is also defined by how it differs from other forms of self-supervised learning (SSL). While supervised learning takes in labeled inputs ${\displaystyle (x,y)}$ and attempts to predict the label ${\displaystyle y}$ from the input ${\displaystyle x}$, self-supervised systems generate their own "labels" from their inputs, which are then predicted. SSL models can be categorized as generative/reconstruction-based or contrastive/invariance-based, with other categories sometimes included.^[8][9][10]

Generative models attempt to reconstruct or synthesize a signal ${\displaystyle y}$ based on input data ${\displaystyle x}$.^{[9]: 3 [10]: 2} For example, masked autoencoders (MAEs) for images remove patches of pixels from an image and attempt to reconstruct the missing patches.^{[11][12]: 9} LeCun suggests that generative models, in attempting to predict missing information as closely as possible (e.g. the next frame of a video), waste computing power on unhelpful noise which cannot be predicted (e.g. the movement of individual leaves on a tree in the wind).^{[1]: 27 [13]: 1 [14]}

Contrastive models – or models using contrastive learning – encode ${\displaystyle x}$ and ${\displaystyle y}$ with the goal of producing similar outputs for related inputs and dissimilar (contrasting) outputs for unrelated inputs.^{[10]: 2 [15]: 3 [16]} Typically, the paired inputs are considered as different "views" of the same underlying concept; for example, photos of the same object from different angles.^[10] The goal is to make the model learn representation invariance: regardless of the specific view of an object, the model should produce the same output.^[17] However, contrastive methods require exponentially more contrasting samples as the dimensionality of the data increases (i.e. as the inputs contain more information).^[1]: 23 Non-contrastive invariance-based methods, such as Bootstrap Your Own Latent (BYOL),^[18] instead maximize the agreement between views^{[clarification needed]} while preventing collapse through statistical constraints, i.e. regularization.^{[19]: 4 [20]: 5}

Joint Embedding Architectures (JEAs), as described by LeCun, are a form of SSL with two encoders which are given different representations of the same input, whose output encodings (called "embeddings") are then compared.^[21] They are commonly used with contrastive learning,^[22]: 2 for example in Siamese networks.^{[23][1]: 42}

[1]

JEPA is a form of self-supervised learning for representations of objects,^{[13]: 1 [15]: 2} meaning that instead of being provided labeled data to predict, the model uses information from co-occurring data (e.g. inputs which commonly appear together) to create its own labeled pairs.^{[24]: 2–3 [25]} The goal is to generate semantic embeddings, which encode abstract or high-level information about concepts (e.g. the idea of a cow).^[15]: 1 This is done by creating neural networks which, given two semantically similar inputs (e.g. a video of a cow and the audio of a cow), produce outputs that are "predictive" of each other.^{[26]: 1 [clarification needed]}

Unlike autoencoders, JEPAs operate entirely in latent space, avoiding pixel-level noise to focus on semantic structure. Rather than (just) learning invariance, JEPAs learn by predicting masked latent representations from visible context.^{[27][clarification needed]} This is also in contrast to LLMs, which operate in token space.^[28]: 18

JEPA can also be described as an energy-based model.^[8]: 5–6 Specifically, it defines an energy function ${\displaystyle E(h_{c},h_{t})}$ which quantifies the "compatibility" between context ${\displaystyle h_{c}}$ and target ${\displaystyle h_{t}}$; a lower energy means a higher compatibility. The goal is then to construct the function so the energy is minimized with correct pairs and maximized with incorrect pairs.^{[13]: 3 [29]: 16}

Given two inputs ${\displaystyle x}$ and ${\displaystyle x'}$, along with optional conditioning or auxiliary information ${\displaystyle a}$, the general form of JEPA's energy function is

${\displaystyle E(x,x';a)={\mathcal {L}}_{\text{pred}}(g_{\phi }(z;u),z')={\mathcal {L}}(g_{\phi }(f_{\theta }(x);q_{\omega }(a)),f_{\theta }(x'))}$

where ${\displaystyle {\mathcal {L}}_{\text{pred}}(\centerdot ,\centerdot )}$ is the prediction loss function, ${\displaystyle g_{\phi }(\centerdot )}$ is the predictor, ${\displaystyle u=q_{\omega }(a)}$ is the output of the conditioning encoder, and ${\displaystyle z,z'=f_{\theta }(x),f_{\theta }(x')}$ are the representations of the inputs.^{[13]: 3 [15]: 7} The general training objective or overall loss function ${\displaystyle {\mathcal {L}}}$ can be written as

${\displaystyle {\mathcal {L}}(x,x';a)={\mathcal {L}}_{\text{pred}}(g_{\phi }(z;u),z')+\lambda {\mathcal {R}}(z)}$

where ${\displaystyle {\mathcal {R}}}$ is a regularization function and ${\displaystyle \lambda }$ is a parameter which controls the importance of the regularization term.^{[13]: 3 [contradictory]}

The first implementation of JEPA, named I-JEPA (Image-JEPA), was published in 2023.^[30][31] I-JEPA is trained by taking an image and cutting out ("masking") multiple rectangles, called "target blocks". The rest of the image, called the "context block", is fed into the "context encoder" (a vision transformer) which generates a representation of the context block in latent space. The "context decoder" (another vision transformer) takes the context block's latent representation, along with the pixel location of a target block, and outputs a prediction of the latent-space embedding of the target block.^[32]: 38 This process allows learning of invariant features without hand-crafting data augmentation processes.^{[33]: 9 [clarification needed]}

JEPA was extended to video data with V-JEPA, published 2024.^[34][35] This architecture enabled hierarchical modelling of temporal dependencies across multiple time scales.^{[36]: 2 [clarification needed]}

JEPA has also been applied to image analysis,^[37] audio processing,^[38] and motion in images and video,^[39] among other domains.^[40]

Jürgen Schmidhuber says that LeCun's work uses but fails to properly credit his past research,^[41] specifically noting the similarities with his work on predictability maximization.^[42]

• Vector symbolic architectures
• Word embedding

Category:Artificial intelligence