Open Source Projects · Sub-chapter 6 · 4 min read

MLflow

Sub-chapter 6 of Open Source Projects · The ML and GenAI lifecycle, open

MLflow is the project that made "reproducible ML" a default expectation. Before MLflow, comparing two model runs meant comparing two notebooks by eye. After MLflow, you log params, metrics, and artifacts per run, and the framework gives you the UI to compare. The same pattern now extends to LLM tracing and agent evaluation in MLflow 3.x.

For a fresh grad, MLflow is the rare project that's both huge and approachable. The codebase is well-organised Python, the maintainers are active, and the GenAI surface area means there are new issues every week.

TL;DR

What - Open-source platform for end-to-end ML and GenAI lifecycle: experiment tracking, model packaging, deployment, LLM evaluation.
License - Apache 2.0.
Governance - Linux Foundation (LF AI & Data Foundation) since June 2020.
Origin - Created at Databricks in 2018 by a team led by Matei Zaharia (creator of Apache Spark). Donated to the LF in June 2020.
Current line - MLflow 3.x (May 2026) treats LLMs and agents as first-class.

Why it exists

By 2018, the ML world had two reproducibility crises stacked on top of each other:

Within a single team, no two ML runs were truly comparable - different notebooks, different data slices, different hyperparameters, different model files, no consistent way to log any of it.
Across teams, even when a model was working, the path from "trained checkpoint" to "served endpoint" was custom every time.

Databricks open-sourced MLflow in June 2018 to address both. Two years later, in June 2020 at the Spark + AI Summit, Databricks transferred the project to the Linux Foundation so it could become a true vendor-neutral standard. By that point, it had 200+ contributors and 2M+ monthly downloads.

Since 2023, MLflow has expanded aggressively into GenAI:

LLM tracing (OpenTelemetry-compatible).
Prompt management.
LLM-as-a-Judge evaluators.
Multi-turn agent evaluation.

MLflow 3.x is the line you'll work with in 2026 and beyond.

Architecture

text

       Your code
           │   mlflow.log_param/metric/artifact
           ▼
   ┌───────────────┐
   │  Tracking API │
   └───────┬───────┘
           │
   ┌───────▼───────┐    ┌────────────────────┐
   │ Backend store │    │ Artifact store     │
   │  (SQLite/PG)  │    │ (local FS, S3, GCS)│
   └───────────────┘    └────────────────────┘
           │
   ┌───────▼───────┐
   │ MLflow Server │   ←──── UI at :5000
   └───────────────┘

Components, classical:

Tracking - log params, metrics, artifacts per "run." A run lives inside an "experiment."
Projects - MLproject YAML format describing how to reproducibly run code.
Models - a standard MLmodel packaging format with "flavors" (sklearn, pytorch, transformers, openai, …).
Model Registry - versioned models with stages and approvals.
Deployments - unified gateway for serving models and LLMs.

Components, newer (3.x):

GenAI Tracing - capture spans for LLM/agent calls (OTel-compatible).
Evaluation - built-in scorers (faithfulness, relevance) and custom LLM-as-judge prompts.
Prompt Registry - version-controlled prompts you can A/B between runs.

The server itself is a Python (Flask/FastAPI) app talking to a backend DB (SQLite by default) and an artifact store.

License and governance

License: Apache 2.0.
Foundation: Linux Foundation (often grouped with LF AI & Data).
Maintainers: A mix of Databricks and external contributors.

Install and hello world

bash

pip install mlflow
mlflow server --host 127.0.0.1 --port 5000
# UI at http://localhost:5000

In a Python script:

python

import mlflow

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("hello-mlflow")

with mlflow.start_run():
    mlflow.log_param("lr", 0.01)
    mlflow.log_param("optimizer", "adam")
    mlflow.log_metric("accuracy", 0.92)
    mlflow.log_metric("loss", 0.18)

Reload the UI - your run appears with params, metrics, and a comparison view. Add a second run with different params and you can compare them side-by-side.

For LLM tracing (3.x):

python

import mlflow
mlflow.openai.autolog()  # or .anthropic, .langchain, .litellm, etc.

# Now every LLM call from the OpenAI SDK is logged as an MLflow trace.

How to contribute

Repo: github.com/mlflow/mlflow
Contributing guide: CONTRIBUTING.md
Good first issues: filter link
DCO - you'll need to sign-off commits (git commit -s).
Community: MLflow Slack (invite at mlflow.org) + LF AI & Data mailing lists.

Where to start as a contributor:

Docs / examples - friendlier entry-point. The examples/ directory has dozens of standalone scripts that drift out of date.
GenAI integrations - newer, smaller surface area, lots of issues open.
Core tracking - high bar but a great career signal.

Hands-on Checkpoints

Run MLflow locally. Log 5 runs with different hyperparameters. Compare them in the UI.
Use mlflow.<provider>.autolog() to capture an LLM trace. Inspect the resulting spans in the UI.
Wrap a sklearn model with mlflow.sklearn.log_model. Load it back. Predict.
Register a model in the Model Registry. Promote it through Stages.
Browse mlflow/mlflow issues. Pick one good first issue. Read the linked code.