Read OSS
FFmpeg/FFmpeg

FFmpeg Source Code: Architecture Overview and How to Navigate 2 Million Lines of C

•Intermediate

Prerequisites

  • ›Basic C programming (structs, pointers, function pointers)
  • ›Familiarity with Makefiles and build systems
  • ›General understanding of what FFmpeg does (video/audio processing)

FFmpeg Source Code: Architecture Overview and How to Navigate 2 Million Lines of C

FFmpeg is one of the most consequential open-source projects ever written. Nearly every piece of software that touches multimedia—VLC, Chrome, OBS, Handbrake, YouTube's backend, your smart TV—depends on FFmpeg either directly or through its libraries. The project contains roughly 2.4 million lines of C code, supports over 400 codecs, nearly 400 container formats, and dozens of I/O protocols. Despite its ubiquity, surprisingly few developers have read beyond the public API headers.

This article is the first in a six-part series that will take you deep inside FFmpeg's source code. We'll start here with the big picture: what the project contains, how its pieces fit together, and the design patterns you'll encounter everywhere once you start reading the code.

What FFmpeg Is: 7 Libraries and 3 Tools

FFmpeg is not a single program—it's a platform. The repository ships seven shared libraries and three command-line tools, each with a distinct responsibility:

Library Purpose
libavutil Common utilities: math, logging, pixel formats, AVBuffer, AVFrame, AVClass
libswscale Image scaling and pixel format conversion
libswresample Audio resampling and sample format conversion
libavcodec Encoding and decoding (400+ codecs)
libavformat Container muxing/demuxing and I/O protocols
libavfilter Audio and video filter graph engine
libavdevice Platform-specific capture and playback devices
Tool Purpose
ffmpeg The transcoding workhorse—reads, decodes, filters, encodes, and writes media
ffplay SDL2-based media player for previewing
ffprobe Stream/container analysis and metadata extraction

The libraries are designed to be used independently. You can link just libavcodec and libavutil if all you need is decoding—you don't have to drag in the entire project.

Directory Structure Map

Navigating 2.4 million lines requires knowing where things live. Here's the top-level directory map:

Directory Contents
libavutil/ ~300 files. Shared primitives: buffers, frames, pixel/sample formats, math, logging
libavcodec/ ~2,700 files. Every codec implementation plus the codec framework
libavformat/ ~800 files. Container format demuxers/muxers, I/O protocols
libavfilter/ ~600 files. Audio/video filter implementations and the graph engine
libswscale/ ~100 files. Pixel format conversion and scaling
libswresample/ ~50 files. Audio sample rate/format conversion
libavdevice/ ~50 files. OS-specific device I/O (ALSA, PulseAudio, V4L2, etc.)
fftools/ ~20 files. The three command-line tools and the scheduler/pipeline engine
ffbuild/ Build tool support files (common.mak, library.mak, version.sh)
doc/ Documentation and 24 standalone example programs in doc/examples/
tests/ FATE test framework and test data
compat/ Compatibility shims for non-POSIX platforms

Tip: The libavcodec/ directory alone accounts for more than half the codebase. Each codec typically lives in one or two .c files named after the codec (e.g., libavcodec/h264dec.c, libavcodec/libx264.c).

Library Dependency Hierarchy

The seven libraries form a strict dependency hierarchy. This isn't a suggestion—it's enforced at link time by the Makefile. The linking order is defined at Makefile#L33-L41:

# $(FFLIBS-yes) needs to be in linking order
FFLIBS-$(CONFIG_AVDEVICE)   += avdevice
FFLIBS-$(CONFIG_AVFILTER)   += avfilter
FFLIBS-$(CONFIG_AVFORMAT)   += avformat
FFLIBS-$(CONFIG_AVCODEC)    += avcodec
FFLIBS-$(CONFIG_SWRESAMPLE) += swresample
FFLIBS-$(CONFIG_SWSCALE)    += swscale

FFLIBS := avutil

The dependency flows downward—each library may only use symbols from libraries listed below it:

flowchart TD
    avdevice["libavdevice"] --> avfilter["libavfilter"]
    avdevice --> avformat["libavformat"]
    avfilter --> avformat
    avfilter --> avcodec["libavcodec"]
    avformat --> avcodec
    avcodec --> swresample["libswresample"]
    avcodec --> swscale["libswscale"]
    swresample --> avutil["libavutil"]
    swscale --> avutil
    avcodec --> avutil

This layering is a deliberate architectural decision. It means libavutil has zero dependencies on any other FFmpeg library, making it a safe foundation. It also means libavcodec knows nothing about container formats—the separation between codec and container is physical, not just conceptual.

The Build Pipeline: configure → config.h → make

FFmpeg uses a custom configure script—not autotools, not CMake—written as a single POSIX shell script. The build pipeline has three stages:

flowchart LR
    A["./configure"] --> B["config.h\nconfig_components.h\nffbuild/config.mak"]
    B --> C["make"]
    C --> D["7 libraries + 3 tools"]

The configure script starts with shell compatibility detection at configure#L1-L55—it literally tries alternative shells if the current one is broken. This pragmatism is characteristic of the entire project.

The most interesting part of configure is find_things_extern, the function that discovers available components at configure#L4507-L4513:

find_things_extern(){
    thing=$1
    pattern=$2
    file=$source_path/$3
    out=${4:-$thing}
    sed -n "s/^[^#]*extern.*$pattern *ff_\([^ ]*\)_$thing;/\1_$out/p" "$file"
}

This function uses sed to scan source files for extern declarations matching a pattern. For example, it scans libavcodec/allcodecs.c for lines like extern const FFCodec ff_h264_decoder; and extracts h264_decoder. The result is a list of every available component, which configure then uses to generate config_components.h with #define CONFIG_H264_DECODER 1 entries.

On the Make side, the DOSUBDIR macro at Makefile#L123-L132 is the engine that builds all seven libraries from a single template:

define DOSUBDIR
$(foreach V,$(SUBDIR_VARS),$(eval $(call RESET,$(V))))
SUBDIR := $(1)/
include $(SRC_PATH)/$(1)/Makefile
-include $(SRC_PATH)/$(1)/$(ARCH)/Makefile
-include $(SRC_PATH)/$(1)/$(INTRINSICS)/Makefile
include $(SRC_PATH)/ffbuild/library.mak
endef

$(foreach D,$(FFLIBS),$(eval $(call DOSUBDIR,lib$(D))))

For each library, DOSUBDIR resets all variables, includes the library's own Makefile (which sets OBJS, HEADERS, etc.), optionally includes arch-specific Makefiles for SIMD code, and then includes library.mak which contains the actual build rules. It's a clean example of Make metaprogramming.

Component Registration Pattern

FFmpeg achieves compile-time component selection through a registration pattern that connects configure-time discovery with runtime iteration. The pattern appears identically in three files:

Each file declares extern symbols for every possible component:

extern const FFCodec ff_a64multi_encoder;
extern const FFCodec ff_aasc_decoder;
extern const FFCodec ff_h264_decoder;
// ... hundreds more
flowchart TD
    subgraph Build Time
        A["allcodecs.c declares\nextern FFCodec ff_h264_decoder"] --> B["configure scans with\nfind_things_extern"]
        B --> C["Generates config_components.h\n#define CONFIG_H264_DECODER 1"]
    end
    subgraph Link Time
        C --> D["Linker includes h264dec.o\nonly if CONFIG_H264_DECODER=1"]
    end
    subgraph Runtime
        D --> E["av_codec_iterate() walks\nthe codec_list[] array"]
    end

The configure script uses find_things_extern to discover which components exist by scanning these extern declarations. Components that are enabled get #define CONFIG_XXX 1 in config_components.h. The library-specific Makefile conditionally compiles the implementation file only when the config flag is set. The linker resolves the extern symbol only for enabled components; disabled ones simply aren't compiled.

At runtime, functions like av_codec_iterate() walk a static array of pointers to the enabled component structs, which is also generated based on the config flags. This design means adding a new codec requires exactly three changes: writing the implementation, adding an extern line to allcodecs.c, and adding a build rule to the Makefile.

Naming Conventions and How to Find Things

FFmpeg's codebase is vast, but its naming conventions are remarkably consistent. Once you learn the patterns, you can predict file and symbol names:

Symbol prefixes:

  • av_ — Public API (safe for external use, ABI-stable)
  • ff_ — Internal to FFmpeg (shared between files within a library, but not public)
  • No prefix — Static to a single file

Codec naming:

  • Decoder: ff_{name}_decoder (e.g., ff_h264_decoder)
  • Encoder: ff_{name}_encoder (e.g., ff_libx264_encoder)
  • File: usually {name}dec.c or {name}enc.c or lib{name}.c for external wrappers

Format naming:

  • Demuxer: ff_{name}_demuxer (e.g., ff_mov_demuxer)
  • Muxer: ff_{name}_muxer (e.g., ff_mp4_muxer)

Filter naming:

  • Video filter: ff_vf_{name} (e.g., ff_vf_scale)
  • Audio filter: ff_af_{name} (e.g., ff_af_aresample)
  • Video source: ff_vsrc_{name}, Audio source: ff_asrc_{name}
  • Video sink: ff_vsink_{name}, Audio sink: ff_asink_{name}

Tip: When looking for a specific codec implementation, grep allcodecs.c for the codec name. The extern declaration tells you the symbol name, and the codec name maps predictably to the source file. For instance, ff_h264_decoder lives in libavcodec/h264dec.c.

The Public/Private Struct Split Pattern

This is the single most important design pattern in FFmpeg. If you understand it, you can navigate every major subsystem. If you don't, the code will seem bewildering.

Every major type in FFmpeg has two versions: a public struct that's part of the ABI, and a private struct that embeds the public one as its first member. The private struct is only used within the library; external consumers only see the public struct.

Here's the pattern as seen in libavcodec/codec_internal.h#L127-L131:

typedef struct FFCodec {
    /**
     * The public AVCodec. See codec.h for it.
     */
    AVCodec p;
    // ... private fields follow
} FFCodec;

The C language guarantees that a pointer to a struct can be cast to a pointer to its first member (and vice versa). This means an FFCodec* can be safely cast to an AVCodec* and passed to external code. Internally, FFmpeg casts back with a simple inline function:

static av_always_inline const FFCodec *ffcodec(const AVCodec *codec)
{
    return (const FFCodec*)codec;
}
classDiagram
    class AVCodec {
        +name: const char*
        +long_name: const char*
        +type: AVMediaType
        +id: AVCodecID
        +capabilities: int
    }
    class FFCodec {
        +p: AVCodec
        +caps_internal: unsigned
        +cb_type: unsigned
        +cb: union
        +init(): int
        +close(): int
    }
    class AVInputFormat {
        +name: const char*
        +long_name: const char*
        +flags: int
    }
    class FFInputFormat {
        +p: AVInputFormat
        +read_probe(): int
        +read_header(): int
        +read_packet(): int
    }
    class AVFilter {
        +name: const char*
        +description: const char*
        +inputs: AVFilterPad*
        +outputs: AVFilterPad*
    }
    class FFFilter {
        +p: AVFilter
        +preinit(): int
        +init(): int
        +activate(): int
        +uninit(): void
    }
    FFCodec *-- AVCodec : embeds as first member
    FFInputFormat *-- AVInputFormat : embeds as first member
    FFFilter *-- AVFilter : embeds as first member

This pattern appears across every subsystem:

Public (ABI-stable) Private (internal) Cast helper
AVCodec FFCodec ffcodec()
AVInputFormat FFInputFormat ffinputformat()
AVOutputFormat FFOutputFormat ffoutputformat()
AVFilter FFFilter fffilter()
AVFilterContext FFFilterContext fffilterctx()

The benefit is ABI stability: the public struct's layout never changes (fields are only appended), while the private struct can be freely rearranged or extended. This is why FFmpeg can maintain binary compatibility across versions despite rapid internal development.

What's Next

With this mental model—seven layered libraries, a configure-based build system, extern-based component registration, and the ubiquitous public/private struct split—you're equipped to explore any part of FFmpeg's source code.

In Part 2, we'll dive deep into the data structures that serve as the "currency" of the entire ecosystem: AVBuffer for reference counting, AVPacket for encoded data, AVFrame for decoded data, and the AVClass/AVOption reflection system that enables structured logging and runtime configuration across every subsystem.