Research Papers
The papers behind modern AI.
Ten papers, in order, tell the story of the last decade of AI - from residual networks to RLHF. Each one changed how the field builds. Read the summary, download the PDF, go to the source.
- Papers
- 21
- Span
- 2010–2026
01Spec v0.1 · 2026Google Cloud, Data CloudOpen Knowledge Format (OKF): An Open Specification for the LLM-Wiki Pattern
Sam McVeety, Amir Hormati · 2026
Google's answer to the question every agent team hit at once: how should curated knowledge be packaged so any model can consume it? OKF formalises Karpathy's LLM-wiki pattern into a vendor-neutral spec - a directory of markdown files with YAML frontmatter, no SDK, no registry, no lock-in. Our hosted copy is rendered from the official spec (Apache 2.0); knowledge bases you can git-clone are becoming how context ships.
Standardised the LLM wiki - knowledge as plain files
02arXiv:2601.23265arXiv 2026Google Cloud AI Research · Peking UniversityPaperBanana: Automating Academic Illustration for AI Scientists
Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, Jinsung Yoon · 2026
Figures are one of the last hand-made artifacts in a research workflow, and they are slow to produce. PaperBanana is an agentic pipeline that plans a diagram's content, renders it with image models, then critiques and refines its own output. On PaperBananaBench, 292 cases drawn from NeurIPS 2025 papers, it improves overall diagram quality 17% over the vanilla baseline and wins 72.7% of blind human comparisons - a concrete look at agents doing production creative work, not just chat.
Agentic pipelines aimed at the research workflow itself
03arXiv:2501.12948Nature 2025DeepSeekDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI (Guo, Yang, Zhang, et al.) · 2025
Showed that long-form reasoning can be *incentivised* rather than taught: pure reinforcement learning on verifiable answers made the model spontaneously develop reflection, self-checking, and ever-longer chains of thought. Released open-weight at a fraction of frontier cost, R1 cracked the reasoning-model recipe open for everyone - and put a reasoning LLM on the cover of Nature.
Open-sourced the reasoning-model recipe
04arXiv:2302.13971arXiv 2023Meta AILLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. · 2023
Trained Chinchilla-style on trillions of public tokens, LLaMA-13B outperformed GPT-3 at a thirteenth of the size - and Meta released the weights to researchers. The models escaped, the community fine-tuned, quantised, and ran them on laptops, and the open-weights ecosystem (Alpaca to Llama.cpp to today's local AI) exists because of this paper.
Ignited the open-weights model ecosystem
05arXiv:2203.02155NeurIPS 2022OpenAITraining language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, et al. · 2022
The RLHF paper: fine-tune on human demonstrations, train a reward model from human preference rankings, then optimise the policy against it. A 1.3B InstructGPT was preferred by humans over 175B GPT-3, proving alignment tuning beats raw scale for usefulness. This is the technique that turned raw language models into usable assistants like ChatGPT and Claude.
RLHF - the step that turned LLMs into assistants
06arXiv:2203.15556NeurIPS 2022DeepMindTraining Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. · 2022
Re-derived the scaling laws and found the field had been training models that were too big on too little data: parameters and tokens should scale together. Chinchilla (70B, 1.4T tokens) beat the much larger Gopher and GPT-3 at the same compute, and its 'Chinchilla-optimal' ratio reshaped how every subsequent frontier model was trained.
Rewrote the scaling laws every frontier lab trains by
07arXiv:2201.11903NeurIPS 2022Google Research, Brain TeamChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. · 2022
Showed that simply prompting a large model to write out intermediate reasoning steps - 'let's think step by step' style exemplars - unlocks large gains on math, logic, and symbolic tasks, an ability that emerges only at scale. Chain-of-thought became a foundational prompting technique and the seed of today's reasoning models.
The prompt pattern that became today's reasoning models
08arXiv:2106.09685ICLR 2022MicrosoftLoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, et al. · 2021
Freezes the pre-trained weights and injects small trainable low-rank matrices into each layer, cutting trainable parameters by up to 10,000x with no inference latency and little quality loss. LoRA made fine-tuning frontier-scale models affordable on commodity GPUs and is the backbone of today's open-model fine-tuning ecosystem.
Made fine-tuning large models affordable for everyone
09arXiv:2103.00020ICML 2021OpenAILearning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, et al. · 2021
Trained an image encoder and a text encoder on 400 million web image-caption pairs to simply agree with each other. The result, CLIP, classifies images it has never been trained on by comparing them to text prompts - zero-shot. CLIP embeddings became the bridge between language and vision: they steer Stable Diffusion, power visual search, and seed most multimodal models.
The language-vision bridge behind multimodal AI
10arXiv:2010.11929ICLR 2021Google Research, Brain TeamAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. · 2020
Cut images into 16x16 patches, treated the patches as tokens, and fed them to a plain Transformer - no convolutions at all. Given enough pre-training data, ViT matched and then beat the best CNNs. It collapsed vision and language onto one architecture, which is why today's multimodal models can treat pixels and words the same way.
One architecture for vision and language - CNNs dethroned
11arXiv:2005.11401NeurIPS 2020Facebook AI ResearchRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. · 2020
Bolted a dense retriever onto a generator so the model consults an external corpus before answering, instead of relying on what its weights memorised. Answers became more factual, more specific, and updatable by swapping the index rather than retraining. RAG is now the default architecture for grounding LLMs in private enterprise data.
Named and defined the default enterprise LLM architecture
12arXiv:2006.11239NeurIPS 2020UC BerkeleyDenoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, Pieter Abbeel · 2020
Made diffusion practical: corrupt an image with noise step by step, train a network to reverse each step, and sample by denoising from pure noise. DDPM matched the best GANs with a far more stable training recipe, and its framework is what Stable Diffusion, DALL-E, Midjourney, and modern video models are built on.
The recipe behind Stable Diffusion, DALL-E and modern image AI
13arXiv:2005.14165NeurIPS 2020OpenAILanguage Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, et al. · 2020
Scaled a decoder-only Transformer to 175B parameters and found that new abilities emerge from scale alone: GPT-3 could perform tasks from a few examples placed in the prompt, with no gradient updates. This paper made prompting a programming model and set off the scaling era that produced today's LLM industry.
Proved scale unlocks in-context learning - prompting is born
14arXiv:1810.04805NAACL 2019Google AI LanguageBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · 2018
Showed that pre-training a bidirectional Transformer on masked-word prediction, then fine-tuning it per task, beats task-specific architectures across the board - eleven NLP benchmarks fell at once. BERT made 'pre-train then fine-tune' the default recipe of applied NLP and still powers a large share of production search and classification systems.
Made pre-train-then-fine-tune the industry default
15arXiv:1706.03762NeurIPS 2017Google Brain · Google ResearchAttention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. · 2017
Dropped recurrence and convolution entirely and built a sequence model from self-attention alone. The Transformer trained in a fraction of the time, parallelised across the whole sequence, and set new translation records. Every frontier model today - GPT, Claude, Gemini, Llama - is a descendant of this eight-page architecture.
The Transformer - the architecture behind every modern LLM
16arXiv:1512.03385CVPR 2016Microsoft ResearchDeep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · 2015
Deep networks were hitting a wall: adding layers made training accuracy worse, not better. ResNet's skip connections let each block learn a residual on top of the identity, and suddenly 152-layer networks trained cleanly, winning ImageNet 2015. The residual connection is now in virtually every deep architecture, Transformers included.
The skip connection every modern network is built on
17arXiv:1409.0473ICLR 2015Jacobs University · Université de MontréalNeural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014
Sequence models were choking a whole sentence through one fixed-size vector. This paper let the decoder look back and softly *attend* over every source word as it translates - the first attention mechanism. Three years later, 'Attention Is All You Need' took this one component and made it the entire architecture.
Invented attention - the Transformer's direct ancestor
18arXiv:1406.2661NeurIPS 2014Université de MontréalGenerative Adversarial Networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. · 2014
Framed generation as a game: a generator forges samples, a discriminator calls fakes, and both improve until the forgeries pass. GANs made neural networks *produce* rather than just classify, and carried generative imagery from blurry curiosities to photorealism. The generative-media era - deepfakes included - traces back to this two-player idea.
Taught neural networks to generate, not just classify
19arXiv:1301.3781ICLR 2013GoogleEfficient Estimation of Word Representations in Vector Space
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean · 2013
Showed that a deliberately shallow model trained on billions of words learns vectors where meaning is geometry: king minus man plus woman lands near queen. Word2vec made embeddings cheap, and turned 'represent it as a vector, compare by distance' into the default move of applied ML - the ancestor of every embedding model and vector search system in production today.
Made embeddings the working currency of applied ML
20NeurIPS 2012University of TorontoImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · 2012
AlexNet won ImageNet 2012 by a margin nobody thought possible - a deep convolutional network trained on two consumer GPUs, using ReLU activations and dropout. The result ended the hand-crafted-features era overnight and convinced the field that depth plus data plus compute wins. Modern AI's timeline starts here.
The result that started the deep learning era
21AISTATS 2010Université de MontréalUnderstanding the Difficulty of Training Deep Feedforward Neural Networks
Xavier Glorot, Yoshua Bengio · 2010
Before this paper, deep networks mostly failed to train and nobody could say precisely why. Glorot and Bengio measured the actual signal: saturating activations and gradients that vanish layer by layer, caused by how weights were initialised. Their fix - now called Xavier initialisation, scaled to the fan-in and fan-out of each layer - plus the case against sigmoid, made deep training reliable enough for everything that followed.
Diagnosed why deep nets would not train, and fixed it
PDFs are mirrored here for convenient reading and remain the work of their authors; each paper's linked record (arXiv or the publishing venue) is the canonical source for its license and citation details.