Fix "Failed to open NVENC codec" in OBS
Reason 1: Outdated Drivers
OBS updates frequently to use the latest NVIDIA features. If your driver is 6 months old, OBS might try to call a function that doesn't exist yet on your PC.
Reason 2: Unsupported Resolution (Older Cards)
If you are using an older GPU (like a GTX 660 or 750 Ti), it might not support encoding at resolutions like 1920x1080 or specific odd resolutions.
- Try lowering your Output Resolution to 1280x720.
- Try switching from Rescale Output to changing the Base Canvas.
Reason 3: Too Many NVENC Sessions
Consumer NVIDIA cards (GTX/RTX) used to have a limit of 3 simultaneous encoding sessions. If you are recording gameplay with ShadowPlay AND streaming with OBS AND using another tool, you might hit the limit.
Fix: Close other programs that use the GPU (Plex, Video Editors, ShadowPlay).
Can't fix it? Switch encoders.
See if your CPU is strong enough to handle the stream instead.