Stencil

From GMod Wiki

Go to: Useful Information

Lua: Stencil Buffer

Description:	A tutorial which teaches the use of stencil buffers in Gmod.
Original Author:	Overv (Loosely based on the tutorial by ralle105)
Created:	13th January 2010

Contents 1 What is the stencil buffer? 2 Stencil Comparison Functions 3 Stencil Operations 4 The reference value 5 Examples

What is the stencil buffer?

The stencil buffer allows you to decide which pixels should be rendered by testing them against certain conditions. Each pixel has an integer value which can be used and/or changed with the various stencil operation and comparison types. The following functions are available to trigger a stencil operation:

• render.SetStencilPassOperation( STENCILOPERATION_* ) - Called when the comparison is successful and the pixel is visible from the player view.
• render.SetStencilZFailOperation( STENCILOPERATION_* ) - Called when the pixel is not visible from the player view.
• render.SetStencilFailOperation( STENCILOPERATION_* ) - Called when the stencil comparison fails.

Note that you need to enable the stencil buffer with render.SetStencilEnable and you should clear it with render.ClearStencil before using it.

Stencil Comparison Functions

The following stencil comparison functions can be passed to render.SetStencilCompareFunction.

• STENCILCOMPARISONFUNCTION_NEVER - Never passes.
• STENCILCOMPARISONFUNCTION_LESS - Passes when the pixel value is lower than the reference value.
• STENCILCOMPARISONFUNCTION_EQUAL - Passes when the pixel value is equal to the reference value.
• STENCILCOMPARISONFUNCTION_LESSEQUAL - Passes when the pixel value is lower or equal than the reference value.
• STENCILCOMPARISONFUNCTION_GREATER - Passes when the pixel value is higher than the reference value.
• STENCILCOMPARISONFUNCTION_NOTEQUAL - Passes when the pixel value is not equal to the reference value.
• STENCILCOMPARISONFUNCTION_GREATEREQUAL - Passes when the pixel value is higher or equal to the reference value.
• STENCILCOMPARISONFUNCTION_ALWAYS - Always passes.
• STENCILCOMPARISONFUNCTION_FORCE_DWORD

The above comparisons will always fail when the pixel is not visible from the player view.

Stencil Operations

The following stencil operations can be passed to render.SetStencilPassOperation, render.SetStencilFailOperation and render.SetStencilZFailOperation.

• STENCILOPERATION_KEEP - Preserves the existing pixel value.
• STENCILOPERATION_ZERO - Sets the pixel value to 0.
• STENCILOPERATION_REPLACE - Replaces the pixel value with the reference value.
• STENCILOPERATION_INCRSAT
• STENCILOPERATION_DECRSAT
• STENCILOPERATION_INVERT - Inverts the pixel value.
• STENCILOPERATION_INCR - Increases the pixel value.
• STENCILOPERATION_DECR - Decreases the pixel value.
• STENCILOPERATION_FORCE_DWORD

The reference value

The reference value can be set with render.SetStencilReferenceValue. It will be compared against the pixel value when using comparison functions like _GREATER and _EQUAL. It will also be used to replace pixel values when a STENCILOPERATION_REPLACE is called.

Examples

• Draw an outline around physics props. First we set the value of every pixel with a prop and its outline to 1 and then we set the pixels with only a prop to 2. This makes us end up with a stencil buffer with 1 values for outline pixels and 2 values for prop pixels. Finally, we draw the material over every pixel with the value 1, thus only the outlines. Here is a preview.

local matOutline = CreateMaterial( "BlackOutline", "UnlitGeneric", { [ "$basetexture" ] = "vgui/black" } )
 
hook.Add( "PostDrawOpaqueRenderables", "PostDrawing", function()
	cam.Start3D( EyePos(), EyeAngles() )
 
	// First we clear the stencil and enable the stencil buffer
	render.ClearStencil()
	render.SetStencilEnable( true )
 
	// First we set every pixel with the prop + outline to 1
	render.SetStencilFailOperation( STENCILOPERATION_KEEP )
	render.SetStencilZFailOperation( STENCILOPERATION_REPLACE )
	render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
	render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_ALWAYS )
	render.SetStencilReferenceValue( 1 )
 
	for _, ent in pairs( ents.FindByClass( "prop_physics*" ) ) do
		ent:SetModelScale( Vector( 1.1, 1.1, 1.1 ) )
		print( ent:GetModelScale() )
		ent:DrawModel()
		ent:SetModelScale( Vector( 1, 1, 1 ) )
	end
 
	// Now we set every pixel with only the prop to 2
	render.SetStencilReferenceValue( 2 )
 
	for _, ent in pairs( ents.FindByClass( "prop_physics*" ) ) do
		ent:DrawModel()
	end
 
	// Now we only draw the pixels with a value of 1 black, which are the pixels with only the outline
	render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )
	render.SetStencilPassOperation( STENCILOPERATION_REPLACE )
	render.SetStencilReferenceValue( 1 )
 
	render.SetMaterial( matOutline )
	render.DrawScreenQuad()
 
	// Disable the stencil buffer again
	render.SetStencilEnable( false )
 
	cam.End3D()
end )