[bug] plg_video_transcoder (libav): ffmpeg 8 rejects AAC encode — 'open encoder: Invalid argument' for sources with unspecified channel layout

[SteamedFish]

Description of the bug

With a CGO build (plg_video_transcoder/libav) running against ffmpeg 8.x (libavcodec 62, e.g. Arch Linux ffmpeg 2:8.1.2), HLS playlist generation succeeds but every segment request fails:

SYST ERROR plg_video_transcoder::segment::run open encoder: Invalid argument

The browser gets the playlist (/hls/index.m3u8 → 200) but no playable segment ever arrives, so video playback is completely broken on distros shipping ffmpeg 8.

Root cause (traced in server/plugin/plg_video_transcoder/libav/transcode.c):

In add_stream()'s audio branch (~lines 163–171), the code queries the aac encoder's supported channel layouts:

if (avcodec_get_supported_config(s->enc_ctx, s->enc_codec,
        AV_CODEC_CONFIG_CHANNEL_LAYOUT, 0, (const void **)&layouts, &nl) == 0 && nl > 0) {
    /* pick a supported layout */
} else {
    s->enc_ctx->ch_layout = s->dec_ctx->ch_layout;   /* fallback: copy decoder layout */
}

On ffmpeg 8, avcodec_get_supported_config() for the native aac encoder succeeds but returns nl == 0 (the encoder exports no layout list), so the fallback copies the decoder's ch_layout. For sources whose demuxer doesn't specify a channel layout — e.g. wmav2 in ASF (AV_CHANNEL_ORDER_UNSPEC, nb_channels=2) — the encoder context ends up with an UNSPEC layout.

ffmpeg ≤7's aac_encode_init() silently defaulted UNSPEC → stereo. ffmpeg 8 validates the layout and returns EINVAL (-22), so avcodec_open2() at transcode.c:302 fails with exactly the logged open encoder: Invalid argument.

I confirmed this with a standalone C reproducer mirroring the exact add_stream()/open_output() sequence against Arch's ffmpeg 8.1.2 libraries:

[libx264 ...] using cpu capabilities: MMX2 SSE2Fast ...   ← video encoder opens fine
[aac @ ...] Unsupported channel layout "2 channels"       ← audio encoder fails
avcodec_open2(aac) = -22 (Invalid argument)

Note the same input transcodes fine via the ffmpeg CLI with equivalent parameters (exit 0) — the CLI inserts its own layout negotiation, so this bug is specific to the manual libav pipeline in transcode.c.

Suggested fix (verified: after patching, the same reproducer reports avcodec_open2(aac) = 0, layout=stereo):

--- a/server/plugin/plg_video_transcoder/libav/transcode.c
+++ b/server/plugin/plg_video_transcoder/libav/transcode.c
@@ -169,6 +169,9 @@
 		} else {
 			s->enc->ch_layout = s->dec->ch_layout;
 		}
+		if (s->enc->ch_layout.order == AV_CHANNEL_ORDER_UNSPEC) {
+			av_channel_layout_default(&s->enc->ch_layout, s->enc->ch_layout.nb_channels);
+		}
 		s->enc->sample_rate = s->dec->sample_rate;

av_channel_layout_default() is public libavutil API since ffmpeg 5.1, so this stays compatible with ffmpeg 6/7 while fixing ffmpeg 8. It is also codec-agnostic: it guards any input whose demuxer leaves the layout unspecified (ASF/WMV confirmed; likely also some raw/MPEG-PS streams).

Step by step instructions to reproduce the bug

1. Build filestash with CGO enabled (the default) on a system with ffmpeg 8.x (e.g. Arch Linux: ffmpeg 2:8.1.2-10, libavcodec.so.62).
2. Upload a video whose audio stream has no specified channel layout — any .wmv/.asf with wmav2 audio works (verifiable via ffprobe: wmav2, 48000 Hz, 2 channels with no layout name).
3. Open the video in the web UI (transcoding enabled, default libx264 encoder).
4. Watch the server log: /api/files/cat?...&transcode=hls → 200, /hls/index.m3u8 → 200, then for each /hls/segment_N.ts:
   SYST ERROR plg_video_transcoder::segment::run open encoder: Invalid argument.

Can you replicate that error from the demo?

No — and that's expected: the official Docker image is Debian-based and ships ffmpeg 7.x, which silently defaults the unspecified layout to stereo. The bug only manifests when the host's shared libav is ffmpeg ≥8 (CGO build links the system libav, not a bundled one). Distros on ffmpeg 8 today: Arch Linux; others will follow as they upgrade.

Observed behavior

• /hls/index.m3u8 returns 200 with a valid playlist.
• Every /hls/segment_N.ts request fails server-side; log shows open encoder: Invalid argument.
• Video player sits loading forever / playback never starts.

Expected behavior

Segments transcode and stream as MPEG-TS (h264 + aac), playback starts — as it does with the same build on ffmpeg ≤7.

Environment

• filestash: master @ cdcb9566 (2026-07-27)
• Build: CGO enabled (default go build cmd/main.go), so plg_video_transcoder/libav is active
• OS: Arch Linux (kernel 6.x, x86_64)
• ffmpeg: 2:8.1.2-10 — libavformat 62, libavcodec 62, libavfilter 11, libavutil 60, libswscale 9, libswresample 6
• x264: 0.165
• Source file triggering it: ASF container, wmv3 1920x1080 yuv420p video + wmav2 48000 Hz stereo audio