Need help with a pixel shader

For all coding issues - MODers and programmers, HTML and more.

Moderators: Jeff250, fliptw

Post Reply
User avatar
Diedel
D2X Master
D2X Master
Posts: 5278
Joined: Thu Nov 05, 1998 12:01 pm
Contact:

Need help with a pixel shader

Post by Diedel » Thu Aug 24, 2006 4:58 pm

Background:

I am trying to work around color key transparency used in D2(X(-XL)). Background is that D2 has two kinds of transparency: A regular one you can handle with an alpha channel, and 'supertransparency', where transparency in an overlay texture goes through the base texture.

So what I am trying to do is to pass a mask texture to the pixel shader rendering the base and overlay textures, telling it where the super transparency has to be applied. The mask has a 0 for each super transparent and a 1 for each opaque pixel.

I am using a GL_ALPHA texture as I only need the transparency info (actually a single bit would be sufficient).

I am binding the mask to TMU 2.

Now my problem is that apparently I can only get vec4 values from the TMU (RGBA, all float), although I specified the mask texture as single byte per pixel alpha. So the shader doesn't even compile. Here it is:

Code: Select all

uniform sampler2D btmTex, topTex, maskTex;
uniform float grAlpha;
vec4 btmColor, topColor;
bool bMask;
void main (void)
{
bMask = texture2D (maskTex, vec2 (gl_TexCoord [1]));
if (!bMask)
	discard;
else 
	{
	topColor = texture2D (topTex, vec2 (gl_TexCoord [1]));
	btmColor = texture2D (btmTex, vec2 (gl_TexCoord [0]));
	if(topColor.a == 0.0)
		gl_FragColor = vec4 (vec3(btmColor), btmColor.a * grAlpha) * gl_Color;
	else 
		gl_FragColor = vec4 (vec3 (mix (btmColor, topColor, topColor.a)), (btmColor.a + topColor.a) * grAlpha) * gl_Color;
	}
}
I cannot find anything on the inet telling me how to achieve the desired functionality in my shader program. Can someone here help me with this?
User avatar
DCrazy
DBB Alumni
DBB Alumni
Posts: 8826
Joined: Wed Mar 15, 2000 3:01 am
Location: Seattle

Post by DCrazy » Thu Aug 24, 2006 5:49 pm

Are shaders even necessary for this? Why can't you combine the textures into a new texture and apply that to the desired face?
User avatar
Diedel
D2X Master
D2X Master
Posts: 5278
Joined: Thu Nov 05, 1998 12:01 pm
Contact:

Post by Diedel » Thu Aug 24, 2006 6:46 pm

Because that takes a lot more textures, and that hurts particularly if using hires textures.

Edit (8/30/06): Case closed, I got a solution.
Post Reply