How to properly use visibility masks?

Hi,
We’re trying to use the new visibility mask API to selectively rtTrace specific scene groups but it does not seem to have an effect.

To reproduce the issue, we modified the optixSphere project from the Optix 6 samples:

optixSphere.cpp line 243:

+ RT_CHECK_ERROR( rtGeometryGroupSetVisibilityMask(geometrygroup, 2) );

pinhole_camera.cu line 72:

~ rtTrace(top_object, ray, prd, 4);

So the group that contains the sphere instance has visibility mask 2 and we trace visibility mask 4.
With this change, I expected the sphere not to be intersected since there’s no overlap between the two bitmasks, but it’s still rendered.

Is there something wrong with the way we use visibility masks?

Thanks!

Works for me on a Quadro RTX 6000 with 419.17 drivers. This might be GPU specific.

Please always list the following system configuration information when asking about OptiX isuses:
OS version, installed GPU(s), VRAM amount, display driver version, OptiX major.minor.micro version, CUDA toolkit version used to generate the input PTX, host compiler version.

Hi Detlef,

We found that we have the issue with 2 different configurations, and we have 2 other configurations that work.

Those do not work:

GTX 1080 Ti (x2) with 11 Gb RAM each
Windows 7
Display driver 419.67
Optix 6.0.0
Cuda 10.0

Quadro GV100 (x2) with 32 Gb RAM each
CentOS Linux 7.6.1810
Display driver 418.43
Optix 6.0.0 ?
Cuda 10.0 ?

However, those work:

RTX 2080 Ti with 11 Gb RAM
Windows 10
Display driver 419.35
Optix 6.0.0
Cuda 10.0

RTX 2080 Ti with 11 Gb RAM
CentOS Linux 7.6.1810
Display driver 418.43
Optix 6.0.0
Cuda 10.0

So basically it works on RTX GPUs and it doesn’t work with other GPUs, which are paired.

Thanks, that’s what I was looking for. I’ll file a bug report against OptiX. (=> That already existed.)

Hi,
My system: 2x RTX 2080 Ti 11GB RAM, optix 6.8.0, Ubuntu 18.04.5, display driver 465.19.01, cuda 10.2, gcc 7.5.0
has this issue been solved?
With the code above the sphere is still rendered in my system.

I have found this thread because I have an issue with visibility masks: I put a number of meshes under a geometry group with a mask, I rtTrace with that mask and I get some hits as I expect. Then I rtTrace with RT_VISIBILITY_ALL and some of the previous hits are now misses. I would expect to hit the same meshes, since RT_VISIBILITY_ALL includes also my mask.
Any advice?
Thank you very much.

The original problem in OptiX 6.0.0 on non-RTX boards in this thread should have been solved long before OptiX 6.5.0 got released. This was never an issue on RTX boards.

There is no OptiX 6.8.0 version.
If that is with OptiX 6.0.0, then please try with OptiX 6.5.0.
If this is with OptiX 6.5.0, then we’d need a minimal and complete reproducer of your issue.

Is the GeometryGroup node the root of your render graph? (That wouldn’t work. See the API reference guide for rtGroupSetVisibilityMask and rtGeometryGroupSetVisibilityMask.)

Does this also happen with single GPU?

Hi Detlef,
Thanks a lot.
I am using OptiX 6.5. I print the version information with the code that is in the optix 6.5 samples, optixDeviceQuery.cpp, lines 56-59, and I see Optix 6.8.0. If I run that sample I get the same value. It used to print Optix 6.5.0, then 6.6.0 and recently it prints 6.8.0.

In the sample case if I insert a group node in the optixSphere it works correctly and the sphere is not rendered.

In my case, I have the following graph:

                         "root" (Group, Trbvh)
                             /           \  ...\ 
"static meshes" (Geometry Group, Trbvh)    transform
         / ..../                                \
      GI      GI                                GI (Trbvh)
                                                    GT ....GT

so, I have a root group, in the next level: a Geometry Group with a number of static meshes under it, and an (optional) number of moving meshes under their transforms.
In the case I mention I do not have moving meshes. (I know that I do not really need the mask in this case). I set for the “static meshes” GeometryGroup a mask of 1 and rtTrace with that mask against the rootNode and get a number of hits which is correct. Then I rTrace with RT_VISIBILITY_ALL and get 9 fewer hits which is not correct. I can render the rays and see that they are passing through meshes they should hit. There may be something wrong with my graph but I do not know what. It is going to be difficult to provide a reproducer that is minimal.

If I force the use of a single GPU with context->setDevices() I get exactly the same problem.

Thanks a lot again.

I am using OptiX 6.5. I print the version information with the code that is in the optix 6.5 samples, optixDeviceQuery.cpp, lines 56-59, and I see Optix 6.8.0. If I run that sample I get the same value. It used to print Optix 6.5.0, then 6.6.0 and recently it prints 6.8.0.

Oh, I see. That’s an information from the display driver which incremented the minor number for different branches due to internal changes and bug fixes.
I didn’t know it’s reporting 6.8.0 in the meantime, because I’m not using OptiX 6 versions anymore. Quite confusing to be honest.
You can ignore that when reporting issues. The crucial version information is the OptiX SDK and the display driver for that part.

In the sample case if I insert a group node in the optixSphere it works correctly and the sphere is not rendered.

so, I have a root group, in the next level: a Geometry Group with a number of static meshes under it …
I set for the “static meshes” GeometryGroup a mask of 1 and rtTrace with that mask against the rootNode and get a number of hits which is correct. Then I rTrace with RT_VISIBILITY_ALL and get 9 fewer hits which is not correct.

Ok, so you’re saying that

  • you have a similar hierarchy in both the changed optixSphere example and your application with
    Group AS (no visibility mask) -> GeometryGroup AS (visibility mask = 1) -> GeometryInstance -> Geometry

  • the changed optixSphere example can correctly switch between a mask which is not matching any bit and doesn’t render the sphere, and a matching mask or RT_VISIBILITY_ALL which does render the sphere.

  • although you have a similar render graph hierarchy the same visibility setting using RT_VISIBILITY_ALL is not working for all rays but has 9 fewer hits (of how many?).

So the remaining differences are that

  • the optixSphere example is using a custom primitive and you’re most likely using built-in triangles?
  • your GeometryGroup contains multiple GeometryInstances. I assume these all contain Geometry nodes with identical primitive types.

I assume you’re using the RTX execution strategy (default), you’re calculating the correct OptiX stack size, you have an exception program running when debugging your application and there are no exceptions raised?

Could you try changing the optixSphere to have the same geometric primitives like you’re using?
If that still works, could you add multiple GeometryInstances like in your case?
If that fails, we’d have a standalone reproducer.
If not, something is probably incorrect in your application.

Hi Detlef,
thank you

Out of 198 hits

Yes to all. About the stack size, I am not recursively tracing and I am not using callable programs, so I do not set anything related to stack size.

Do you mean loading my meshes and using GeometryTriangles with them in that scene? I may do that but in a few days.
Anyway, I am not sure because I have modified the optixGeometryTriangles sample to put several tetrahedrons under its GeometryGroup and set a visibility mask and then trace with the mask and a different masks and RT_VISIBILITY_ALL and it seems to work OK. Since this is also quite similar to my setup and works I guess there may be some error in my application but I am no sure how to debug it.
Thanks

Yes, I’m just looking for any minimal and complete reproducer to be able to file a bug report for investigation.

If you cannot reproduce any of that by changing existing OptiX SDK examples, it’s most likely something inside your code, and then another approach would be to strip down your application to the least amount of code which still reproduces the issue. Sometimes that helps finding programming errors on the way.

Hi,
I have been testing this problem. It does not seem to trace the ray. I am not rendering so let me explain the program:
I just want to check the visibility between two positions (tx and rx) and an interest point (dp). So I trace a ray from tx to dp and another one from rx to dp and check for hits. The relevant code is below:

////trace visibility from transmitter to DP
		float3 originToDP=dp-origin;
		float dist_originToDp=length(originToDP);
		float3 txRayDirection = originToDP/dist_originToDp;
		optix::Ray visibilityRay(origin,txRayDirection , rayTypeIndex, 0.0f,dist_originToDp);
		
		rtTrace(root, visibilityRay, visibilityRayPayload,RT_VISIBILITY_ALL,RT_RAY_FLAG_DISABLE_ANYHIT);

if (visibilityRayPayload.result.x!=OPAL_DIFFRACTION_BLOCKED) {
			rtPrintf("%u\t%u\t%u e=%u dp=(%f,%f,%f) tx not blocked \n",launchIndex.x,launchIndex.y,launchIndex.z,e.id, dp.x,dp.y,dp.z);
			//trace visibility from receiver to DP
			float3 destinationToDP=dp-destination;
			float dist_destinationToDp=length(destinationToDP);		
			float3 rxRayDir=destinationToDP/dist_destinationToDp;
			optix::Ray visibilityRayRx(destination, rxRayDir , rayTypeIndex, 0.0f,dist_destinationToDp); //Visibility ray type = 1
			visibilityRayPayload.result.x=OPAL_DIFFRACTION_LOS;
			visibilityRayPayload.result.y=1;
	        rtTrace(root, visibilityRayRx, visibilityRayPayload,RT_VISIBILITY_ALL,RT_RAY_FLAG_DISABLE_ANYHIT);

In the closest hit and miss programs I set the corresponding payload flag and rtPrintf to see the results. Now the first rtTrace seems to work correctly, but for the second rtTrace I do not see anything printed for the rays in closest hit or miss. So no hit and no miss. Of course, I have checked that the rtTrace is reached past the if, but I do not know how to make sure that rtTrace is executed.
If I just change in the code above the rtTrace with

rtTrace(root, visibilityRay, visibilityRayPayload,OPAL_STATIC_MESH_MASK,RT_RAY_FLAG_DISABLE_ANYHIT);
...
rtTrace(root, visibilityRayRx, visibilityRayPayload,OPAL_STATIC_MESH_MASK,RT_RAY_FLAG_DISABLE_ANYHIT);

everything works fine, and I can see prints for either hits or miss in the second rtTrace
I am quite confused. Maybe you could point me to some error I am not aware of.
Thanks a lot.

I cannot say what is happening with these code excerpts. That would require a minimal complete reproducer with all host and device programs involved.

1.) The rtTrace default RTvisibilitymask is RT_VISIBILITY_ALL anyway. You only set it because you do not use the default RTrayflags RT_RAY_FLAG_NONE but RT_RAY_FLAG_DISABLE_ANYHIT.
Do you have an anyhit program in that visibility ray?
If not, leave both arguments away and try again.

2.) If your visibility origins or destinations are located on geometry surfaces, it’s incorrect to shoot with the [t_min, t_max] interval from 0.0f to full length because that will be affected by self-intersection on either side if you’re not prohibiting that with some other robust method, like checking for the primitive ID which would require an anyhit program you don’t invoke on your visibility ray.

Please read this thread and esp. the comments I made about visibility ray intervals:
https://forums.developer.nvidia.com/t/anyhit-program-as-shadow-ray-with-optix-7-2/181312/2

(Since the OptiX 7 example code in there is a port of my OptiX 5.1 based introduction examples, you’ll find the same code for the old OptiX API when following the links in the README.md of that github repository.)

3.) You initialized these fields only in the second ray:

visibilityRayPayload.result.x=OPAL_DIFFRACTION_LOS;
visibilityRayPayload.result.y=1;

This incomplete code excerpt doesn’t explain how that is set for the first ray, or what is happening with these fields in any other program.

4.) Again, do you have an exception program setup and is there any exception thrown?
For example, your ray direction normalization code is not guarded against invalid values and could result in invalid rays.

Yes, I know. I would like to get one but there is a lot of additional code involved.

No, there is no anyhit. With rtTrace(root, visibilityRayRx, visibilityRayPayload); I get the same wrong results, it only works when I set the mask.

No, they are not located on surfaces. Anyway, I have put a min_epsilon_t =1e-4 for t_min and t_max=maxDist-min_epsilon_t and there is no change

Ok, I did not want to add too much code, but I can copy here the code. The interest point (dp) are always located at edges between faces (faces are made of multiple mesh triangles). I allow the hit on the faces that make the edge, to allow numerical errors. So if any of the rx and tx rays hit on that faces, I consider that there is LoS, that is why I use closest hit, and because I read here it was more efficient. So the code is:
The payload is initialized before that code as:

VisibilityPayload visibilityRayPayload;
		visibilityRayPayload.result.x=OPAL_DIFFRACTION_LOS;
		visibilityRayPayload.faces=e.faces; //Faces of the edge, where the ray can hit to compute the diffraction. If it hits any other face, there is no LoS
		visibilityRayPayload.result.y=0;

The closest hit, miss and exception programs, and geometryTriangles programs:

RT_PROGRAM void closestHitTriangleDiffraction() {
	float3 hp=ray_hit.origin + ray_hit.direction*ch_triangle_data.geom_normal_t.w;
	if (ch_triangle_data.faceId!=rayPayload.faces.x && ch_triangle_data.faceId!=rayPayload.faces.y) { 
		rtPrintf("%u\t%u\t%u\t%d blocked on face=%u ray.faces=%u,%u hp=(%f,%f,%f) \n",launchIndex.x,launchIndex.y,launchIndex.z,rayPayload.result.y,ch_triangle_data.faceId,rayPayload.faces.x, rayPayload.faces.y,hp.x,hp.y,hp.z);
		//LoS is blocked, set flag 
		rayPayload.result.x=OPAL_DIFFRACTION_BLOCKED;
	} else {
		rtPrintf("%u\t%u\t%u\t%d LoS on face=%u ray.faces=%u,%u hp=(%f,%f,%f) \n",launchIndex.x,launchIndex.y,launchIndex.z,rayPayload.result.y,ch_triangle_data.faceId,rayPayload.faces.x, rayPayload.faces.y,hp.x,hp.y,hp.z);
	}

}
RT_PROGRAM void missDiffraction() {
		rtPrintf("%u\t%u\t%u\t%d miss \n", launchIndex.x,launchIndex.y,launchIndex.z,rayPayload.result.y); 
	rayPayload.result.x=OPAL_DIFFRACTION_MISS;
}
RT_PROGRAM void exception()
{
	const unsigned int code = rtGetExceptionCode();
	if (RT_EXCEPTION_USER <= code)
	{
		rtPrintf("User exception %d at (%d, %d)\n", code - RT_EXCEPTION_USER, launchIndex.x, launchIndex.y);
	}
	else
	{
		rtPrintf("Exception code 0x%X at (%d, %d)\n", code, launchIndex.x, launchIndex.y);
	}

}

RT_PROGRAM void triangle_attributes()
{
	const unsigned int primIdx = rtGetPrimitiveIndex();
	const int3   v_idx = index_buffer[primIdx];
	const float3 p0    = vertex_buffer[v_idx.x];
	const float3 p1    = vertex_buffer[v_idx.y];
	const float3 p2    = vertex_buffer[v_idx.z];
	const float3 e0 = p1 - p0;
	const float3 e1 = p0 - p2;
	const float3 n  = cross( e1, e0 );

	const float3 e2 = ( 1.0f / dot( n, ray.direction ) ) * ( p0 - ray.origin );
	

	TriangleHit h;
	
	h.triId = primIdx;

	h.geom_normal_t=make_float4(n.x,n.y,n.z,dot(n,e2));
	

	h.faceId = faceId_buffer[primIdx];

	//Get hitpoint from barycentrics
	const float2 bar=rtGetTriangleBarycentrics();
	const float3 hp=(1.0f -bar.x -bar.y)*p0 + bar.x*p1 + bar.y*p2;
	h.hp=hp;

	int_triangle_data = h;

}


There are other definitions and son on.

Code is above. If you mean the possibility that the distance is 0 and the direction has NaN, in this sceanario it is not the case for sure. Which are any other invalid values?

As I said, I understand it
is difficult to help with only that information. I hoped that you could point out something obvious (not for me).
Thanks a lot

Please explain the calculation of the ch_triangle_data.geom_normal_t.w value.
That is, why isn’t the hit point simply calculated with a float variable with rtIntersectionDistance semantic?

Last wild guess: Does is work with all rtPrintf() calls removed? I’ve seen the weirdest issues with that function.
You can also use CUDA’s native printf() function and limit it to the launch indices you want to debug.

Which are any other invalid values?

NaNs or INF in any ray component, t_min > t_max, t_min < 0.

Other than that, I can’t say if there is anything broken in the rest of your code or if there is actually an issue inside OptiX 6.5.0 without a minimal and complete reproducer I can attach to a bug report.

Yes, I do not remember why I did that in the moment. I will use the function.

Commenting them out I still get the same results.

Thanks for your help anyway. If I am able to get a reproducer I will come back to you.