I am raycasting against a mesh that may have tiny triangles (probably an error in our mesh generation code). In that case rtpModelUpdate hangs, then crashes and somehow disrupts the GPU so much that I sometimes have to reboot to regain control.
I can send the actual triangles and sample code (100 lines) to reproduce.
Even though I am now filtering out those tiny triangles, I think Optix could be improved to fail more gracefully.
Which OptiX version are you using?
Is this dependent on the number of triangles in the scene or is it happening with any tiny triangle?
What means “tiny” in absolute coordinates?
What happens when not building the acceleration structure on the GPU?
>>I can send the actual triangles and sample code (100 lines) to reproduce.<<
You could send a reproducer to the OptiX-Help e-mail address you’ll find inside the OptiX release note.
Note that *.zip attachments will be blocked by our e-mail servers. The extension needs to be renamed to something like *.zi_.
There is also an e-mail attachment size limit. The archive should be as small as possible and your minimal reproducer needs to be in failing state.
If your e-mail bounces because of size limitation let me know and I can setup a temporary FTP account to upload bigger data.