Representation collapse

Representation collapse is a phenomenon in machine learning and representation learning where a model maps different inputs to the same or very similar embeddin

Representation collapse

Representation collapse is a phenomenon in machine learning and representation learning where a model maps different inputs to the same or very similar embeddings, which means it loses important information about how the data is spread out.[1][2] It is frequently encountered in self-supervised learning, especially within contrastive and non-contrastive frameworks, when training objectives or model architectures do not maintain variance across representations. Collapse results in degenerate solutions characterized by uninformative learned features, significantly impairing downstream task performance. Various techniques have been proposed to mitigate representation collapse, including the use of negative samples, architectural asymmetry, stop-gradient operations, variance regularization, and redundancy reduction objectives, as seen in methods such as SimCLR, BYOL, and VICReg. Comprehending and averting representation collapse is regarded as a fundamental challenge in the advancement of stable and efficient self-supervised learning systems.[3][4][5][6][7][8]

See also

References

  1. ^ Zhao, Wenhao; Zou, Qiran; Shah, Rushi; Liu, Dianbo (2024-11-25). "Representation Collapsing Problems in Vector Quantization". arXiv:2411.16550 [cs.LG].
  2. ^ Jing, Li; Vincent, Pascal; LeCun, Yann; Tian, Yuandong (2022-04-23). "Understanding Dimensional Collapse in Contrastive Self-supervised Learning". arXiv:2110.09348 [cs.CV].
  3. ^ Chen, Ting; Kornblith, Simon; Norouzi, Mohammad; Hinton, Geoffrey (2020). "A Simple Framework for Contrastive Learning of Visual Representations". Proceedings of the 37th International Conference on Machine Learning. arXiv:2002.05709.
  4. ^ Grill, Jean-Bastien; Strub, Florian; Altché, Florent; Tallec, Corentin (2020). "Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning". Advances in Neural Information Processing Systems. arXiv:2006.07733.
  5. ^ Bardes, Adrien; Ponce, Jean; LeCun, Yann (2022). "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning". International Conference on Learning Representations.
  6. ^ Zbontar, Jure; Jing, Li; Misra, Ishan; LeCun, Yann; Denis, Stéphane (2021). "Barlow Twins: Self-Supervised Learning via Redundancy Reduction". International Conference on Machine Learning. arXiv:2103.03230.
  7. ^ Jing, Li; Tian, Yingli (2021). "Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence. 43 (11): 4037–4058. Bibcode:2021ITPAM..43.4037J. doi:10.1109/TPAMI.2020.2992393. PMID 32386141.
  8. ^ Wang, Xiaolong; Isola, Phillip (2020). "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere". International Conference on Machine Learning. arXiv:2005.10242.
Stub icon

This artificial intelligence–related article is a stub. You can help Wikipedia by adding missing information.

Content Disclaimer

Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.

  1. The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
  2. There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
  3. It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
  4. Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
  5. Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.