The Ghost in the Build: Fixing HDRP Custom Pass Stencil Failures

In high-fidelity game development, the High Definition Render Pipeline (HDRP) Custom Pass system is a powerhouse for creating portals, outlines, and x-ray vision effects. However, developers in 2026 frequently encounter a frustrating "Build-Only" bug: a stencil-based effect that looks perfect in the Unity Editor completely disappears or renders black in the standalone build. This discrepancy usually stems from how Unity handles Depth/Stencil buffer allocation during runtime or the aggressive Shader Stripping that occurs during the build process. To fix this, you must ensure your render targets are explicitly allocated with depth bits and that your shader passes aren't being optimized away. This tutorial will walk you through the architectural fixes to ensure your stencil masks remain rock-solid across all platforms.

Table of Content

• Purpose: Visual Parity Across Environments
• The Logic: Why Stencils Fail in Standalone Builds
• Step-by-Step: Validating Your Custom Pass Build Settings
• Use Case: Character Outline Post-Processing
• Best Results: Explicit Allocation and Pass Naming
• FAQ
• Disclaimer

Purpose

Ensuring stencil buffer stability in builds is critical for:

• Gameplay Mechanics: Maintaining the visibility of "Special Vision" or "Detective Mode" mechanics that rely on stencil masks.
• UI/UX Fidelity: Keeping stylized UI elements (like round minimaps or cutout portraits) consistent between dev and play.
• QA Efficiency: Reducing the time spent debugging "phantom" issues that only appear after a long build process.

The Logic: Why Stencils Fail in Standalone Builds

The Editor is more "forgiving" with memory than a standalone build. In the Editor, the SceneView often shares a common depth/stencil buffer that is already initialized. In a Build:

1. Missing Depth Bits: If you are using a Custom Color Buffer in your Custom Pass, Unity might allocate it without a stencil/depth channel unless explicitly told otherwise.

2. Shader Pass Stripping: Unity’s build pipeline removes unused shader variants. If your Custom Pass uses a specific LightMode tag that isn't referenced by any standard renderer, the build might strip that pass out entirely.

3. MSAA Conflict: If MSAA is enabled in your Quality settings but not handled in your Custom Pass script, the stencil buffer resolve can fail, leading to empty masks.

Step-by-Step

1. Explicitly Allocate Depth in C#

If you are scripting your custom pass using RTHandles.Alloc, ensure you include DepthBits.Depth8 (which includes the stencil channel). If you leave this out, it may default to 0 in build mode.

this.maskBuffer = RTHandles.Alloc(
    Vector2.one, 
    TextureXR.slices, 
    dimension: TextureDimension.Tex2D, 
    colorFormat: GraphicsFormat.R16G16B16A16_SFloat, 
    useDynamicScale: true, 
    name: "StencilMask", 
    depthBufferBits: DepthBits.Depth8 // CRITICAL FOR STENCILS
);

2. Use Pass Names instead of Indices

Never use DrawRenderers with an integer index for the shader pass. Build-time stripping changes these indices. Use ShaderTagId to find the pass by name.

// Bad: ctx.DrawRenderers(renderers, 0);
// Good:
var shaderTag = new ShaderTagId("MyCustomStencilPass");
ctx.DrawRenderers(renderers, shaderTag);

3. Check Injection Points

If your stencil is being used for a post-process effect, ensure the Injection Point is set to AfterOpaqueDepthAndNormal. If it’s set to AfterPostProcess, the stencil buffer might have already been cleared or jittered by TAA.

4. Force Shader Inclusion

If your stencil shader is only called via a Custom Pass script, add it to the Always Included Shaders in Project Settings > Graphics to prevent the build stripper from deleting it.

Use Case

A developer is creating a Portal Effect where the player looks through a doorway into another world.

• The Problem: The portal works in the Editor, but in the Build, the doorway is just a black square.
• The Fix: The developer realizes the "Portal Mask" shader pass was named Forward. Because no standard objects used that specific variant, it was stripped. By renaming the pass and adding a DepthBits.Depth8 allocation to the custom buffer, the portal renders perfectly in the build.

Best Results

FAQ

Why is my stencil always black in the Frame Debugger?

If your target buffer is None or Camera, the Frame Debugger might not show the stencil channel clearly. Try outputting the stencil value as a color in your shader to verify it is being written.

Does 'Clear Flags' on the Custom Pass Volume matter?

Yes. If you have multiple custom passes, ensure the first one Clears Stencil, and subsequent passes Don't Clear. In Build mode, uninitialized buffers can contain "garbage" data that breaks your logic.

Can I use Stencils with Transparent objects?

Yes, but you must set the Injection Point to BeforeTransparent. If you do it after, the stencil is often ignored by the transparent pass's depth-testing logic.

Disclaimer

Custom Pass behavior is highly dependent on your HDRP version (e.g., 17.x vs 12.x). Some older versions of HDRP had a bug where the "Custom Buffer" was not properly resolved on consoles. Always test on your target hardware early in the development cycle. March 2026.

Tags: HDRP, CustomPass, StencilBuffer, UnityBuildBug