Custom CSI Camera image quality at high framerate

Hi, we are using an IMX219 camera to capture video at 640x480 @ 90FPS. We can capture the video just fine but the image is darker than we would like. When we capture video at 1280x720 @ 60FPS or 1920x1080 @ 30FPS the video looks great.

When I wrote the IMX219 driver I used a logic analyzer to capture the traffic between a Raspberry Pi and the Raspberry Pi Camera board V2 and I’ve noticed some difference between how the Raspberry Pi and the TX1 interacts with the sensor.

Specifically both the Raspberry Pi and the TX1 adjust the frame length and coarse time (exposure time) of the image to maximize image quality and high frame rate but the Raspberry Pi also uses the analog gain within the sensor as well to improve image quality.

I can tell when the driver has adjusted the gain because I’ve setup my driver to output a dmesg whenever anything attempts to modify the gain.

I can appreciate that using the frame length and coarse time would be optimal over the gain but it looks like the ISP on the TX1 should start adjusting the gain when the image is dark at a high frame rate.

I believe the problem might be related to a configuration within the DTS file for the camera.

Currently, within my DTSI file for the camera I have configured the mode of the IMX219 640x480 with this:

...
          mode2 { // IMX219_MODE_640X480

            //mclk_khz = "47000";
            mclk_khz = "24000";
            mclk_multiplier = "17.0";
            //pix_clk_hz = "182400000";
            pix_clk_hz = "170000000";

            num_lanes = "2";
            tegra_sinterface = "serial_e";
            discontinuous_clk = "yes";
            cil_settletime = "0";
            pixel_t = "bayer_rggb";
            readout_orientation = "270";
            inherent_gain = "0";

            active_w = "640";
            active_h = "480";

            //line_length = "752";
            //line_length = "1752";
            line_length = "3448";
            //line_length = "3559";
            dpcm_enable = "false";

            min_gain_val = "0";
            max_gain_val = "10";
            min_hdr_ratio = "1";
            max_hdr_ratio = "1";
            min_framerate = "1";
            max_framerate = "90";
            min_exp_time = "22";
            max_exp_time = "358733";
            //embedded_metadata_height = "2";
          };
...

Note: There is a function within the driver that converts that gain value to a more appropriate value for the camera. This is similar to how the E3326 modes are configured. They have a gain range of 1.0 - 16.

As an experiment I’ve used the nvgstcapture application to test out the camera, I’ve adjusted various settings including ‘auto exposure’ and ‘scene mode’ but the image still looks pretty dark.

I’ve even used the v4l2-ctl command the manually adjust the gain using the following command:

v4l2-ctl --set-ctrl=gain=5

while the nvgstcapture application is running and the ISP will automatically put it back down to 0!

Is the DTS the right place to be configuring the camera’s gain?
If not where?
Is there another place where I could configure how the ISP interacts with the sensor?

Thanks for any feedback,

Dave

To improve image quality at high frame rate (short integration time), you would need to use camera analogue binning and increase analogue gain though I2C interface of IMX219.

Digital gain after analogue to digital converters will only improve appearance, not signal to noise ratio of images.

I’m not sure TX1 ISP is capable of adjusting analogue gain of IMX219. You can use I2C analyser to monitor values written to IMX219 analogue gain register, or add driver debug information from reading IMX219 analogue gain register.

That makes sense. I think this is the purpose of the HDR mode of (High Dynamic Range) I didn’t think about HDR when I wrote the driver but now it seems I should.

I appreciate the help,

Thank you.