Optix Alpha, Merge with OpenGL

Greetings,

The idea for my project is to combine an Camera Background texture with the rendering created from OptiX,
Is it possible for Optix to output a color buffer which includes an alpha for the miss program?
(So any rays that miss objects have an alpha value of 0, complete transparency).

Yes, that is pretty easy to do.

All rendering OptiX SDK examples either use BGRA8 or RGBA32F output buffers so you have the alpha information available inside the OpenGL texture which is used to blit the resulting image.

You start off with an alpha of 1.0f for your standard output color and set that to 0.0f when hitting the miss shader with the primary(!) ray only.

If you do any antialiasing inside the ray tracer by jittering the ray direction you pick up the color from the miss shader which would result in incorrectly colored silhouettes around your objects when blending against a different background.

You could even generate an opacity value along rays which only pass through transparent objects before hitting the miss shader, which would allow to blend them over the background. Though that alone wouldn’t handle refractions correctly.