Research Papers

13 papers

• inbox important

  TransMLA: Multi-Head Latent Attention Is All You Need

  2025 arXiv ArXivIsual PDF

  Fanxu Meng, Pingzhi Tang, Xiaojuan Tang, Zengwei Yao, Xing Sun, Muhan Zhang

  cs.lg cs.ai machine-learning artificial-intelligence attention language-models pretraining optimization
• inbox important

  Vision Transformers Need Registers

  2023 arXiv ArXivIsual PDF

  Timothée Darcet, Maxime Oquab, Julien Mairal, Piotr Bojanowski

  What causes artifacts in ViT feature maps, and how can they be fixed?

  vision-transformers attention-maps registers self-supervised
  cs.CV computer-vision vision-transformers self-supervised attention-maps
• inbox useful

  Visualizing the Loss Landscape of Neural Nets

  2017 arXiv ArXivIsual PDF

  Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein

  How does network architecture affect the loss landscape and generalization?

  loss-landscape visualization residual-networks optimization
  cs.LG cs.CV stat.ML machine-learning loss-landscape visualization optimization
• inbox useful

  The Shattered Gradients Problem: If resnets are the answer, then what is the question?

  2017 arXiv ArXivIsual PDF

  David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, Brian McWilliams

  Why do skip-connections enable training of deep networks even with well-chosen initializations?

  gradients residual-networks orthogonal-matrices deep-learning
  cs.NE cs.LG stat.ML deep-learning gradients residual-networks
• inbox important

  Residual Networks Behave Like Ensembles of Relatively Shallow Networks

  2016 arXiv ArXivIsual PDF

  Andreas Veit, Michael Wilber, Serge Belongie

  Why do residual networks work so well despite being extremely deep?

  residual-networks ensemble-methods network-analysis shallow-networks
  cs.CV cs.AI cs.LG cs.NE computer-vision residual-networks ensemble-methods
• inbox important

  Identity Mappings in Deep Residual Networks

  2016 arXiv ArXivIsual PDF

  Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

  What propagation formulations behind residual blocks enable training of extremely deep networks?

  residual-networks skip-connections identity-mappings deep-learning
  cs.CV cs.LG computer-vision deep-learning residual-networks skip-connections
• inbox core

  Deep Residual Learning for Image Recognition

  2015 arXiv ArXivIsual PDF

  Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

  How can we train neural networks that are substantially deeper than previously possible without degradation?

  residual-networks skip-connections deep-learning image-classification
  cs.CV computer-vision deep-learning residual-networks image-classification
• inbox important

  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

  2015 arXiv ArXivIsual PDF

  Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

  How can we train extremely deep rectified networks from scratch, and can they surpass human-level performance?

  activation-functions deep-learning initialization image-classification
  cs.CV cs.AI cs.LG computer-vision deep-learning activation-functions image-classification
• reading useful

  TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate

  2025 arXiv ArXivIsual PDF

  Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni

  Can we achieve near-optimal distortion rates for vector quantization in an online, data-oblivious setting?

  vector-quantization information-theory online-algorithms approximation
  cs.lg cs.ai cs.db cs.ds machine-learning artificial-intelligence databases algorithms fairness
• read core

  Generative Adversarial Networks

  2014 arXiv ArXivIsual PDF

  Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

  Can we train generative models by pitting two networks against each other in a minimax game?

  generative-models adversarial-training minimax-game deep-learning
  stat.ml cs.lg machine-learning generative-models
• inbox important

  VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

  2025 arXiv ArXivIsual PDF

  Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Yejin Bang, Allen Bolourchi, Yann LeCun, Pascale Fung

  Can we build a vision-language model that learns in embedding space rather than token space?

  joint-embedding vision-language self-supervised multimodal
  cs.cv computer-vision language-models contrastive-learning multimodal video embeddings classification
• reading important

  Less is More: Recursive Reasoning with Tiny Networks

  2025 arXiv ArXivIsual PDF

  Alexia Jolicoeur-Martineau

  Can tiny neural networks with recursive reasoning outperform large language models on hard puzzle tasks?

  recursive-reasoning small-models ARC-AGI generalization
  cs.lg cs.ai machine-learning artificial-intelligence language-models deep-learning
• read core

  Attention Is All You Need

  2017 arXiv ArXivIsual PDF

  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

  Can we build a sequence transduction model using only attention, without recurrence or convolutions?

  transformers attention sequence-to-sequence parallel-computing
  cs.cl cs.lg nlp machine-learning transformers attention cnn deep-learning