SEPTEMBER 2022 EDIT: I updated this guide a little bit because a couple of things changed. Basically, you don't need the `MOZ_X11_EGL=1` environment variable anymore, and you also want to install v21.08 of org.freedesktop.Platform.ffmpeg-full now instead of 19.08. As Firefox updates the ffmpeg runtime version they use, you will need to update the Flatpak accordingly.
This is how I was able to get hardware acceleration working on Flatpak Firefox (version 83.0 at the time of writing this). Note that I am using an AMD GPU for this, though I believe it should work with other brands as well. This guide is for the Xorg display server.
flatpak install org.freedesktop.Platform.ffmpeg-full. Choose branch 21.08 when prompted
flatpak override --env="MOZ_LOG=PlatformDecoderModule:5" org.mozilla.firefox
gfx.webrender.all=true media.ffmpeg.vaapi.enabled=true media.ffvpx.enabled=false (maybe not required anymore?) media.av1.enabled=false (maybe not required anymore?)
flatpak run --verbose org.mozilla.firefox. Then, open a YouTube video. Right as the video starts playing, look for logs indicating that it's working. As an example, I will show the relevant lines of output from the logs that I see, indicating that it's working:
[Child 389: MediaPDecoder #2]: D/PlatformDecoderModule Initialising VA-API FFmpeg decoder [RDD 437: MediaPDecoder #1]: D/PlatformDecoderModule FFMPEG: Initialising VA-API FFmpeg decoder [RDD 437: MediaPDecoder #1]: D/PlatformDecoderModule FFMPEG: codec h264 : H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 libva info: VA-API version 1.12.0 ... RDD 437: MediaPDecoder #1]: D/PlatformDecoderModule FFMPEG: VA-API FFmpeg init successful ... [RDD 437: MediaPDecoder #1]: D/PlatformDecoderModule FFMPEG: VA-API Got one frame output with pts=1933333 dts=1933334 duration=33333 opaque=-9223372036854775808