Research Papers
The papers behind modern AI.
In chronological order, these papers tell the story of modern AI - from Xavier initialisation to Stable Diffusion, RLHF, and the reasoning models. Each one changed how the field builds. Read the summary, open the PDF, go to the source.
- Papers
- 57
- Span
- 2010–2026
01Spec v0.1 · 2026Google Cloud, Data CloudOpen Knowledge Format (OKF): An Open Specification for the LLM-Wiki Pattern
Sam McVeety, Amir Hormati · Jun 2026
Google's answer to the question every agent team hit at once: how should curated knowledge be packaged so any model can consume it? OKF formalises Karpathy's LLM-wiki pattern into a vendor-neutral spec - a directory of markdown files with YAML frontmatter, no SDK, no registry, no lock-in. Our hosted copy is rendered from the official spec (Apache 2.0); knowledge bases you can git-clone are becoming how context ships.
Standardised the LLM wiki - knowledge as plain files
02arXiv:2601.23265arXiv 2026Google Cloud AI Research · Peking UniversityPaperBanana: Automating Academic Illustration for AI Scientists
Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, Jinsung Yoon · Jan 2026
Figures are one of the last hand-made artifacts in a research workflow, and they are slow to produce. PaperBanana is an agentic pipeline that plans a diagram's content, renders it with image models, then critiques and refines its own output. On PaperBananaBench, 292 cases drawn from NeurIPS 2025 papers, it improves overall diagram quality 17% over the vanilla baseline and wins 72.7% of blind human comparisons - a concrete look at agents doing production creative work, not just chat.
Agentic pipelines aimed at the research workflow itself
03arXiv:2501.12948Nature 2025DeepSeekDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI (Guo, Yang, Zhang, et al.) · Jan 2025
Showed that long-form reasoning can be *incentivised* rather than taught: pure reinforcement learning on verifiable answers made the model spontaneously develop reflection, self-checking, and ever-longer chains of thought. Released open-weight at a fraction of frontier cost, R1 cracked the reasoning-model recipe open for everyone - and put a reasoning LLM on the cover of Nature.
Open-sourced the reasoning-model recipe
04arXiv:2406.17557NeurIPS 2024Hugging FaceThe FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Guilherme Penedo, Hynek Kydlíček, et al. · Jun 2024
Showed that data curation is a science: 15 trillion tokens of filtered Common Crawl, every deduplication and filtering decision ablated and published, plus FineWeb-Edu - an LLM-classifier-selected educational subset that lifts reasoning benchmarks sharply. The open corpus quality frontier moved here.
Open pre-training data, curated like it matters
05arXiv:2405.09673arXiv 2024Columbia University · DatabricksLoRA Learns Less and Forgets Less
Dan Biderman, Jacob Portes, et al. · May 2024
Measured LoRA's actual trade-off across code and math: it absorbs less new capability than full fine-tuning, but forgets less of the base model - a regulariser, not a free lunch. Its target-all-modules guidance became the community's default settings. The paper that made adapter choices evidence-based.
Quantified the fine-tuning trade-off everyone was guessing at
06arXiv:2404.10719ICML 2024Tsinghua UniversityIs DPO Superior to PPO for LLM Alignment? A Comprehensive Study
Shusheng Xu, Wei Fu, et al. · Apr 2024
Stress-tested the field's rush to DPO: theory and experiments showing DPO can drift off-distribution, plus the tuning that makes PPO work - advantage normalisation, large batches, exponential-moving-average references. Properly tuned PPO swept DPO on dialogue and code, keeping RL in the alignment toolkit on evidence, not fashion.
The evidence check on the DPO gold rush
07arXiv:2403.08763arXiv 2024Mila · Université de Montréal · EleutherAISimple and Scalable Strategies to Continually Pre-train Large Language Models
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, et al. · Mar 2024
Showed you can keep pre-training a model on new data without starting over: re-warm then re-decay the learning rate, and mix in a small fraction of the original corpus to hold off catastrophic forgetting. Simple, validated at scale, and directly practical for every team whose model needs to stay current.
The practical recipe for keeping models current
08arXiv:2402.09353ICML 2024NVIDIADoRA: Weight-Decomposed Low-Rank Adaptation
Shih-Yang Liu, Chien-Yi Wang, et al. · Feb 2024
Decomposed each weight update into magnitude and direction, adapting them separately - closing much of LoRA's quality gap with full fine-tuning at the same parameter budget. Analysis of the decomposition also explained *why* LoRA falls short, making DoRA both a better method and a better lens.
Sharpened LoRA by splitting magnitude from direction
09arXiv:2401.04088arXiv 2024Mistral AIMixtral of Experts
Albert Q. Jiang, Alexandre Sablayrolles, et al. · Jan 2024
Released the first open-weights sparse mixture-of-experts that beat much larger dense models: eight experts per layer, two active per token - 47B parameters stored, 13B of compute per token. Mixtral outperformed Llama 2 70B with ~5x fewer active parameters and mainstreamed MoE as the efficiency play frontier labs now default to.
Made sparse mixture-of-experts open-weights mainstream
10arXiv:2312.00752COLM 2024CMU · PrincetonMamba: Linear-Time Sequence Modeling with Selective State Spaces
Albert Gu, Tri Dao · Dec 2023
Made state-space models select what to remember: input-dependent SSM parameters plus a hardware-aware parallel scan give linear-time sequence modelling that matches transformers at multi-billion scale. Mamba is the strongest challenger yet to attention's monopoly - and its ideas now hybridise into production architectures.
The first credible linear-time challenger to the Transformer
11arXiv:2308.04079SIGGRAPH 2023Inria · MPI Informatik3D Gaussian Splatting for Real-Time Radiance Field Rendering
Bernhard Kerbl, Georgios Kopanas, et al. · Aug 2023
Replaced NeRF's slow ray-marched MLP with millions of explicit 3D Gaussians, optimised directly and rendered by a differentiable rasteriser - real-time radiance fields at state-of-the-art quality. 3DGS reset the speed/quality frontier for novel-view synthesis and is now the default for 3D capture, VR, and simulation.
Radiance fields at real-time speed - NeRF's successor
12arXiv:2308.07037arXiv 2023NNAISENSEBayesian Flow Networks
Alex Graves, Rupesh Kumar Srivastava, et al. · Aug 2023
Graves's rethink of generative modelling: instead of corrupting data, run Bayesian belief updates over noisy observations and train a network to steer the beliefs. BFNs put continuous, discretised, and discrete data - including language - under one differentiable framework, a rare genuinely new generative family.
A new generative family beyond diffusion
13arXiv:2305.14314NeurIPS 2023University of WashingtonQLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer · May 2023
Fine-tuned 65B models on one 48GB GPU: freeze the base model in 4-bit NF4, backprop through it into LoRA adapters, page optimiser states to CPU. Guanaco came within a few points of ChatGPT-grade eval scores at a fraction of the cost - and hobbyist fine-tuning of frontier-class open models became normal.
Frontier-scale fine-tuning on a single GPU
14arXiv:2305.18290NeurIPS 2023Stanford UniversityDirect Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. · May 2023
Collapsed RLHF's reward-model-plus-PPO pipeline into a single supervised loss on preference pairs, derived from the same underlying objective. No reward network, no rollouts, no RL instability - and quality matching PPO in its regime. DPO made preference alignment something any lab can run, and spawned the *PO family that followed.
Alignment without the RL - preference tuning for everyone
15arXiv:2304.07193TMLR 2024Meta AIDINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, et al. · Apr 2023
Scaled DINO's label-free recipe into a visual foundation model: 142M curated images, stabilised self-distillation, and frozen features that match or beat supervised and CLIP-style backbones across classification, depth, and segmentation. DINOv2 made 'frozen visual features, no labels' a production default.
The label-free visual backbone that became a default
16arXiv:2304.02643ICCV 2023Meta AISegment Anything
Alexander Kirillov, Eric Mintun, et al. · Apr 2023
Built segmentation's foundation model: a promptable architecture - click, box, or rough mask in; segmentation out - trained on 1.1 billion masks collected with the model in the loop. SAM segments objects it has never seen, and its zero-shot masks became infrastructure for labelling, editing, robotics, and medical imaging almost overnight.
Segmentation's foundation-model moment
17arXiv:2302.13971arXiv 2023Meta AILLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. · Feb 2023
Trained Chinchilla-style on trillions of public tokens, LLaMA-13B outperformed GPT-3 at a thirteenth of the size - and Meta released the weights to researchers. The models escaped, the community fine-tuned, quantised, and ran them on laptops, and the open-weights ecosystem (Alpaca to Llama.cpp to today's local AI) exists because of this paper.
Ignited the open-weights model ecosystem
18arXiv:2302.04761NeurIPS 2023Meta AIToolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, et al. · Feb 2023
Let a language model teach itself to call APIs: sample candidate tool calls, keep the ones that reduce perplexity on the following text, fine-tune on the result. With a handful of demonstrations per tool, the model learned when to reach for a calculator, search, or translation - the self-supervised seed of today's tool-using agents.
Models that taught themselves to use tools
19arXiv:2301.13688ICML 2023Google ResearchThe Flan Collection: Designing Data and Methods for Effective Instruction Tuning
Shayne Longpre, Le Hou, et al. · Jan 2023
Systematised instruction tuning: 1,800+ tasks, zero-shot, few-shot and chain-of-thought templates mixed deliberately, and ablations showing which choices actually move generalisation. Flan-T5 beat far larger untuned models, and the collection's recipes became the public reference for turning base models into instruction followers.
The public recipe book for instruction tuning
20arXiv:2212.13345arXiv 2022Google BrainThe Forward-Forward Algorithm: Some Preliminary Investigations
Geoffrey Hinton · Dec 2022
Hinton's proposed escape from backpropagation: replace the backward pass with two forward passes - one on real data, one on negatives - and train each layer locally to score them apart. Accuracy trails backprop, but as a biologically plausible, pipeline-friendly learning rule it keeps open a research lane that future hardware may need.
Hinton's challenge to backprop's monopoly
- Science 2022Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic ReasoningRead at the source21Science 2022Meta AI (FAIR)
Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning
Meta FAIR Diplomacy Team (Bakhtin, Brown, et al.) · Nov 2022
Reached human-level play in Diplomacy - a seven-player game won through negotiation, alliance, and betrayal - by coupling a dialogue model to an explicit strategic planner that keeps the talk consistent with the plan. Cicero ranked in the top 10% against humans who never suspected a bot, and previewed the agent pattern: language model plus planner.
LLM plus planner - the agent blueprint, demonstrated at human level
22arXiv:2210.05189arXiv 2022AAC TechnologiesNeural Networks are Decision Trees
Caglar Aytekin · Oct 2022
A short proof that any ReLU network is exactly equivalent to a decision tree - every activation pattern defines a leaf with its own linear model. The tree is exponentially large, so it is no free interpretability lunch, but the equivalence sharpened the debate about what 'black box' actually means.
The equivalence proof that sharpened the interpretability debate
23arXiv:2208.09392NeurIPS 2023University of MarylandCold Diffusion: Inverting Arbitrary Image Transforms Without Noise
Arpit Bansal, Eitan Borgnia, et al. · Aug 2022
Asked whether diffusion even needs noise - then trained models to invert blurs, masks, pixelation, and snow with the same iterative recipe. Generation quality trailed Gaussian diffusion, but the point stood: the framework is about learned inversion of degradations, not Gaussian noise specifically. It widened how the field thinks about generative processes.
Showed diffusion is inversion, not noise
24arXiv:2207.12598arXiv 2022Google Research, Brain TeamClassifier-Free Diffusion Guidance
Jonathan Ho, Tim Salimans · Jul 2022
Dropped the conditioning signal at random during training so one network learns both conditional and unconditional generation - then, at sampling, extrapolates between the two to sharpen prompt adherence. Two lines of math, no extra classifier, and the quality dial every text-to-image system has shipped with since.
The guidance dial inside every text-to-image model
25arXiv:2206.00364NeurIPS 2022NVIDIAElucidating the Design Space of Diffusion-Based Generative Models
Tero Karras, Miika Aittala, Timo Aila, Samuli Laine · Jun 2022
Took diffusion's tangle of noise schedules, parameterisations, and samplers and separated the design space into independent, testable choices. Karras's cleaned-up formulation cut sampling to ~35 steps at state-of-the-art quality, and 'EDM' became the reference frame in which diffusion research is now written.
Turned diffusion from folklore into an engineering discipline
26arXiv:2205.14135NeurIPS 2022Stanford UniversityFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, et al. · May 2022
Observed that attention is bottlenecked by GPU memory traffic, not FLOPs - then computed exact attention in tiles that never materialise the N×N matrix. 2-4x faster, memory linear in sequence length, and the direct enabler of today's long-context models. Now in every serious training and inference stack.
The kernel that made long context economically possible
27arXiv:2203.15556NeurIPS 2022DeepMindTraining Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. · Mar 2022
Re-derived the scaling laws and found the field had been training models that were too big on too little data: parameters and tokens should scale together. Chinchilla (70B, 1.4T tokens) beat the much larger Gopher and GPT-3 at the same compute, and its 'Chinchilla-optimal' ratio reshaped how every subsequent frontier model was trained.
Rewrote the scaling laws every frontier lab trains by
28arXiv:2203.02155NeurIPS 2022OpenAITraining language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, et al. · Mar 2022
The RLHF paper: fine-tune on human demonstrations, train a reward model from human preference rankings, then optimise the policy against it. A 1.3B InstructGPT was preferred by humans over 175B GPT-3, proving alignment tuning beats raw scale for usefulness. This is the technique that turned raw language models into usable assistants like ChatGPT and Claude.
RLHF - the step that turned LLMs into assistants
29arXiv:2201.11903NeurIPS 2022Google Research, Brain TeamChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. · Jan 2022
Showed that simply prompting a large model to write out intermediate reasoning steps - 'let's think step by step' style exemplars - unlocks large gains on math, logic, and symbolic tasks, an ability that emerges only at scale. Chain-of-thought became a foundational prompting technique and the seed of today's reasoning models.
The prompt pattern that became today's reasoning models
30arXiv:2201.03545CVPR 2022Meta AI · UC BerkeleyA ConvNet for the 2020s
Zhuang Liu, Hanzi Mao, et al. · Jan 2022
Asked whether transformers really beat CNNs, or whether CNNs were just stuck with 2015 training recipes - then modernised ResNet step by measured step until it matched Swin at every scale. ConvNeXt kept convolutions competitive and forced honesty about how much of the transformer era was architecture versus recipe.
Proved the ConvNet had recipe problems, not architecture problems
31arXiv:2201.02177ICLR 2022 WorkshopOpenAIGrokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
Alethea Power, Yuri Burda, Harri Edwards, et al. · Jan 2022
Documented networks that memorise a small algorithmic dataset, sit at chance-level generalisation for thousands of steps - then abruptly 'grok' the underlying rule and jump to perfect accuracy. A short curiosity that opened a research programme into training dynamics, phase transitions, and what optimisation is actually doing.
The delayed-generalisation puzzle that fed mechanistic interpretability
32arXiv:2112.10752CVPR 2022LMU Munich · RunwayHigh-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, et al. · Dec 2021
Ran diffusion in the compressed latent space of a VQGAN-style autoencoder instead of raw pixels, cutting compute by an order of magnitude while adding cross-attention conditioning for text. This is the Stable Diffusion paper - the architecture that took image generation from lab demo to something running on consumer GPUs everywhere.
Stable Diffusion - image generation for everyone
33Nature 2021DeepMindHighly Accurate Protein Structure Prediction with AlphaFold
John Jumper, Richard Evans, et al. · Jul 2021
Solved structural biology's 50-year grand challenge: predicting a protein's 3D structure from its amino-acid sequence, at experimental accuracy, for most of the proteome. AlphaFold's Evoformer coupled evolutionary alignments with geometric attention - and its open database of 200M+ structures changed how biology is done. The strongest proof yet that deep learning can crack real science.
AI's first solved grand-challenge science problem
34arXiv:2106.09685ICLR 2022MicrosoftLoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, et al. · Jun 2021
Freezes the pre-trained weights and injects small trainable low-rank matrices into each layer, cutting trainable parameters by up to 10,000x with no inference latency and little quality loss. LoRA made fine-tuning frontier-scale models affordable on commodity GPUs and is the backbone of today's open-model fine-tuning ecosystem.
Made fine-tuning large models affordable for everyone
35arXiv:2104.13478arXiv 2021Imperial College · NYU · DeepMindGeometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
Michael M. Bronstein, Joan Bruna, Taco Cohen, Petar Veličković · Apr 2021
The 'proto-book': derived CNNs, GNNs, transformers, and equivariant nets as one family - architectures that respect the symmetries of their data domain. It gave practitioners a principled answer to 'which architecture fits this data?' and became the standard reference for structure-aware ML.
One symmetry framework explaining CNNs, GNNs and transformers
36arXiv:2104.14294ICCV 2021Meta AI · InriaEmerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, et al. · Apr 2021
Trained ViTs with self-distillation - no labels at all - and found something nobody expected: the attention maps segment objects on their own, and the features work for retrieval out of the box. DINO showed self-supervised vision could produce semantics, not just invariances, and set up the DINOv2 foundation-model line.
Label-free ViTs that learned to segment by themselves
37arXiv:2103.14030ICCV 2021Microsoft Research AsiaSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Ze Liu, Yutong Lin, Yue Cao, et al. · Mar 2021
Made transformers work like vision backbones: compute self-attention inside local windows, shift the windows between layers so information flows globally, and build a hierarchical pyramid the way CNNs do. Swin brought linear complexity in image size and became the default transformer backbone for detection and segmentation.
The transformer backbone dense vision tasks standardised on
38arXiv:2103.00020ICML 2021OpenAILearning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, et al. · Mar 2021
Trained an image encoder and a text encoder on 400 million web image-caption pairs to simply agree with each other. The result, CLIP, classifies images it has never been trained on by comparing them to text prompts - zero-shot. CLIP embeddings became the bridge between language and vision: they steer Stable Diffusion, power visual search, and seed most multimodal models.
The language-vision bridge behind multimodal AI
39arXiv:2012.12877ICML 2021Meta AI · Sorbonne UniversitéTraining Data-Efficient Image Transformers & Distillation Through Attention
Hugo Touvron, Matthieu Cord, et al. · Dec 2020
ViT needed 300 million pre-training images; DeiT matched CNNs using ImageNet alone. The trick was heavy augmentation plus a distillation token that lets the transformer learn from a CNN teacher through attention. It made vision transformers practical for anyone without Google-scale data.
Vision transformers without Google-scale data
40arXiv:2012.09841CVPR 2021Heidelberg UniversityTaming Transformers for High-Resolution Image Synthesis
Patrick Esser, Robin Rombach, Björn Ommer · Dec 2020
Married a convolutional VQ autoencoder to a transformer: compress images into a grid of discrete learned tokens, then model token sequences autoregressively. VQGAN made high-resolution synthesis tractable for transformers and supplied the codebook trick behind DALL-E-style generation - and its autoencoder became the backbone Stable Diffusion compresses through.
Discrete image tokens - the bridge from pixels to transformers
41arXiv:2010.11929ICLR 2021Google Research, Brain TeamAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. · Oct 2020
Cut images into 16x16 patches, treated the patches as tokens, and fed them to a plain Transformer - no convolutions at all. Given enough pre-training data, ViT matched and then beat the best CNNs. It collapsed vision and language onto one architecture, which is why today's multimodal models can treat pixels and words the same way.
One architecture for vision and language - CNNs dethroned
42arXiv:2006.11239NeurIPS 2020UC BerkeleyDenoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, Pieter Abbeel · Jun 2020
Made diffusion practical: corrupt an image with noise step by step, train a network to reverse each step, and sample by denoising from pure noise. DDPM matched the best GANs with a far more stable training recipe, and its framework is what Stable Diffusion, DALL-E, Midjourney, and modern video models are built on.
The recipe behind Stable Diffusion, DALL-E and modern image AI
43arXiv:2005.04118ACL 2020Microsoft Research · University of WashingtonBeyond Accuracy: Behavioral Testing of NLP Models with CheckList
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh · May 2020
Brought software engineering's unit-testing discipline to NLP: behavioural test suites - minimum functionality, invariance, directional expectations - that expose failures aggregate accuracy hides. Commercial systems billed as production-grade failed basic negation and typo tests. Best Paper at ACL 2020, and the ancestor of today's LLM eval-suite culture.
Unit tests for models - the seed of modern LLM evals
44arXiv:2005.11401NeurIPS 2020Facebook AI ResearchRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. · May 2020
Bolted a dense retriever onto a generator so the model consults an external corpus before answering, instead of relying on what its weights memorised. Answers became more factual, more specific, and updatable by swapping the index rather than retraining. RAG is now the default architecture for grounding LLMs in private enterprise data.
Named and defined the default enterprise LLM architecture
45arXiv:2005.14165NeurIPS 2020OpenAILanguage Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, et al. · May 2020
Scaled a decoder-only Transformer to 175B parameters and found that new abilities emerge from scale alone: GPT-3 could perform tasks from a few examples placed in the prompt, with no gradient updates. This paper made prompting a programming model and set off the scaling era that produced today's LLM industry.
Proved scale unlocks in-context learning - prompting is born
46arXiv:2003.08934ECCV 2020UC Berkeley · Google Research · UC San DiegoNeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, et al. · Mar 2020
Represented a scene as a neural field: query a small MLP with a 3D position and viewing direction, get back colour and density, render by ray marching. From a handful of photos, NeRF synthesised photorealistic new viewpoints and launched neural rendering as a subfield. The idea that a network *is* the scene representation reshaped 3D vision.
The network became the scene - neural rendering is born
47arXiv:1910.10683JMLR 2020Google ResearchExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, et al. · Oct 2019
Reframed every NLP task - translation, summarisation, classification - as text in, text out, then ran the field's most systematic ablation of pre-training choices to find what actually matters. The result, T5, plus the C4 corpus it introduced, became a workhorse of the pre-LLM era. Its everything-is-text framing is how every modern LLM now operates.
Made every NLP task text-to-text - the framing LLMs run on
48arXiv:1810.04805NAACL 2019Google AI LanguageBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · Oct 2018
Showed that pre-training a bidirectional Transformer on masked-word prediction, then fine-tuning it per task, beats task-specific architectures across the board - eleven NLP benchmarks fell at once. BERT made 'pre-train then fine-tune' the default recipe of applied NLP and still powers a large share of production search and classification systems.
Made pre-train-then-fine-tune the industry default
49arXiv:1806.01261arXiv 2018DeepMind · Google Brain · MITRelational Inductive Biases, Deep Learning, and Graph Networks
Peter W. Battaglia, Jessica B. Hamrick, et al. · Jun 2018
Argued that the next step for deep learning was relational: encode entities and relations explicitly, and let networks reason over structure instead of flat vectors. Its graph-network formalism unified the message-passing architectures under one framework and gave graph ML its vocabulary. GNNs in production - recommendations, molecules, maps - trace their design language here.
Gave graph neural networks their unifying framework
50arXiv:1706.03762NeurIPS 2017Google Brain · Google ResearchAttention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. · Jun 2017
Dropped recurrence and convolution entirely and built a sequence model from self-attention alone. The Transformer trained in a fraction of the time, parallelised across the whole sequence, and set new translation records. Every frontier model today - GPT, Claude, Gemini, Llama - is a descendant of this eight-page architecture.
The Transformer - the architecture behind every modern LLM
51arXiv:1512.03385CVPR 2016Microsoft ResearchDeep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · Dec 2015
Deep networks were hitting a wall: adding layers made training accuracy worse, not better. ResNet's skip connections let each block learn a residual on top of the identity, and suddenly 152-layer networks trained cleanly, winning ImageNet 2015. The residual connection is now in virtually every deep architecture, Transformers included.
The skip connection every modern network is built on
- Nature 2015Deep LearningRead at the source52Nature 2015NYU · Université de Montréal · University of Toronto
Deep Learning
Yann LeCun, Yoshua Bengio, Geoffrey Hinton · May 2015
The three future Turing laureates wrote the field's manifesto: why representation learning beats hand-engineered features, how convolutional and recurrent nets actually work, and where the field was headed. As a review it produced no new result - but it consolidated deep learning as a discipline at the exact moment the world started paying attention, and it remains the most-cited overview of the era.
The manifesto that declared the deep learning era open
53arXiv:1409.0473ICLR 2015Jacobs University · Université de MontréalNeural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · Sep 2014
Sequence models were choking a whole sentence through one fixed-size vector. This paper let the decoder look back and softly *attend* over every source word as it translates - the first attention mechanism. Three years later, 'Attention Is All You Need' took this one component and made it the entire architecture.
Invented attention - the Transformer's direct ancestor
54arXiv:1406.2661NeurIPS 2014Université de MontréalGenerative Adversarial Networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. · Jun 2014
Framed generation as a game: a generator forges samples, a discriminator calls fakes, and both improve until the forgeries pass. GANs made neural networks *produce* rather than just classify, and carried generative imagery from blurry curiosities to photorealism. The generative-media era - deepfakes included - traces back to this two-player idea.
Taught neural networks to generate, not just classify
55arXiv:1301.3781ICLR 2013GoogleEfficient Estimation of Word Representations in Vector Space
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean · Jan 2013
Showed that a deliberately shallow model trained on billions of words learns vectors where meaning is geometry: king minus man plus woman lands near queen. Word2vec made embeddings cheap, and turned 'represent it as a vector, compare by distance' into the default move of applied ML - the ancestor of every embedding model and vector search system in production today.
Made embeddings the working currency of applied ML
56NeurIPS 2012University of TorontoImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Dec 2012
AlexNet won ImageNet 2012 by a margin nobody thought possible - a deep convolutional network trained on two consumer GPUs, using ReLU activations and dropout. The result ended the hand-crafted-features era overnight and convinced the field that depth plus data plus compute wins. Modern AI's timeline starts here.
The result that started the deep learning era
57AISTATS 2010Université de MontréalUnderstanding the Difficulty of Training Deep Feedforward Neural Networks
Xavier Glorot, Yoshua Bengio · May 2010
Before this paper, deep networks mostly failed to train and nobody could say precisely why. Glorot and Bengio measured the actual signal: saturating activations and gradients that vanish layer by layer, caused by how weights were initialised. Their fix - now called Xavier initialisation, scaled to the fan-in and fan-out of each layer - plus the case against sigmoid, made deep training reliable enough for everything that followed.
Diagnosed why deep nets would not train, and fixed it
PDFs are mirrored here for convenient reading and remain the work of their authors; each paper's linked record (arXiv or the publishing venue) is the canonical source for its license and citation details.
About this reading list
What is this list?
The papers behind the systems we build, each with a short summary of what it introduced and why it still matters in production work.
How are papers chosen?
By whether they changed how practitioners build. The list favours work that shows up in real architectures over work that is merely recent.
Who owns the PDFs mirrored here?
Their authors. The PDFs are mirrored for convenient reading, and each paper's linked record at arXiv or its publishing venue remains the canonical source for license and citation.
Turning one of these ideas into a production system?
That gap - paper to running system, monitored against a number - is the whole job here. Tell us what you are trying to build.
Talk to a senior engineer