The matrix multiplication in Awb is swapped: the gains should be applied
after combinedMatrix, i.e. on the left side. The mistake happened when
`ccm' was replaced with combinedMatrix and gainMatrix multiplication was
moved to Awb. While CCM must be applied after gains, the gains must be
applied after the combined matrix, which no longer corresponds to CCM in
Awb.
Since there is currently no algorithm modifying combinedMatrix before
Awb, combinedMatrix is an identity matrix there and the wrong order
doesn't influence the output at the moment.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
The Lut algorithm is not really an algorithm. Moreover, algorithms may
be enabled or disabled but with Lut disabled, nothing will work.
Let's move the construction of lookup tables to CPU debayering, where it
is used. The implied and related changes are:
- DebayerParams is changed to contain the real params rather than lookup
tables.
- contrastExp parameter introduced by GPU ISP is used for CPU ISP too.
- The params must be initialised so that debayering gets meaningful
parameter values even when some algorithms are disabled.
- combinedMatrix must be put to params everywhere where it is modified.
- Matrix changes needn't be tracked in the algorithms any more.
- CPU debayering must watch for changes of the corresponding parameters
to update the lookup tables when and only when needed.
- Swapping red and blue is integrated into lookup table constructions.
- gpuIspEnabled flags are removed as they are not needed any more.
Reviewed-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
control_ids.h defines the contrast type as float, let's use the same in
simple IPA, instead of double. Saturation and gamma already use float,
except for the knobs initializers, let's use float for the knobs too.
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Saturation adjustments are implemented using matrix operations. They
are currently applied to the colour correction matrix. Let's move them
to a newly introduced separate "Adjust" algorithm.
This separation has the following advantages:
- It allows disabling general colour adjustments algorithms without
disabling the CCM algorithm.
- It keeps the CCM separated from other corrections.
- It's generally cleaner.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The combined matrix must be reset to the initial value before each frame
is prepared. This must be done outside algorithms because any of the
algorithms may be disabled while the matrix must be always initialised.
Let's initialise the combined matrix to the identity matrix (which keeps
the pixel values unchanged) in software ISP just before calling
`prepare' on the algorithms.
Matrix updates can no longer be skipped in ccm.cpp, otherwise the CCM
won't be applied if there is no temperature or saturation change. We
explicitly track whether the CCM has been set up completely rather than
relying on the frame number, to avoid missing the initialisation in case
the first frame is skipped due to some error.
Reviewed-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Let's introduce IPAActiveState::combinedMatrix that is separate from
IPAActiveState::ccm and represents the overall correction matrix, not
only the sensor colour correction matrix.
IPAActiveState::ccm still includes everything; this is changed in the
followup patch.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
IPAActiveState::ccm stores the colour correction matrix (CCM) and
whether it has been changed. The change flag is later used when
recomputing or not the lookup tables.
But the CCM may include other corrections than just the sensor colour
correction, for example white balance and saturation adjustments. These
things should be separated and IPAActiveState::ccm should represent just
the CCM itself.
As the first step towards that cleanup, let's separate the change flag
from the CCM. And wrap the only remaining member of
IPAActiveState::ccm.
Also, let's reset the separated change flag in the lookup tables; it'll
be no longer tied to just CCM handling.
This patch doesn't change actual behaviour and it still reports the
combined matrix as CCM in metadata. This is addressed in the followup
patches.
Reviewed-by: Bryan O'Donoghue <bod.linux@nxsw.ie>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The black level offset subtracted in AWB is wrong. It assumes that the
stats contain sums of the individual colour pixels. But they actually
contain sums of the colour channels of larger "superpixels" consisting
of the individual colour pixels. Each of the RGB colour values and the
computed luminosity (a histogram entry) are added once to the stats per
such a superpixel. This means the offset computed from the black level
and the number of pixels should be used as it is, not divided.
The patch fixes the subtracted offset. Since the evaluation is the same
for all the three colours now, the individual class variables are
replaced with a single RGB variable.
Fixes: 4e13c6f55b ("Honor black level in AWB")
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Robert Mader <robert.mader@collabora.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Pass contrastExp as calculated in lut to debayer params not the raw
contrast. This way we calculate contrastExp once per frame in lut and pass
the calculated value into the shaders, instead of passing contrast and
calculating contrastExp once per pixel in the shaders.
Signed-off-by: Bryan O'Donoghue <bryan.odonoghue@linaro.org>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Tested-by: Hans de Goede <johannes.goede@oss.qualcomm.com> # ThinkPad T14s gen 6 (arm64) ov02c10 + X1c gen 12 ov08x40
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com> # Lenovo X13s
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
If the first frame of a stream is bad, the IPA will not get called with
frame == 0, leaving activeState.agc expo/again uninitialized. This causes
the agc algorithm to set a very low gain and exposure on the next run
(where it will hit the if (!stats->valid) {} path) resulting in starting
with a black image.
Fix this by using a valid flag instead of checking for frame == 0.
The entire activeState gets cleared to 0 on configure() resetting the new
valid flag.
Signed-off-by: Hans de Goede <johannes.goede@oss.qualcomm.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The AWB gains reported in the metadata are currently scaled against the
maximum gain, but in fact the gain itself is already stored as a
floating point gain value.
Remove the incorrect scaling and report the gains directly.
Fixes: a0b97475b1 ("ipa: simple: Report the ColourGains in metadata")
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Software CPU ISP computes a gamma lookup table. The table incorporates
black level and contrast. All entries in the table below the black
level are set to 0. This is not necessarily correct all the time.
Let's consider this case: The CCM is
[1 0 0]
[0 1 0]
[0 0 0]
and contrast is set to zero. The gamma table has all the entries above
the black level set to 186 (due to zero contrast) and all the entries
below the black level set to 0. CCM is applied before gamma, a
non-black level pixel has the blue component set to 0 with the CCM
above. Now, when the gamma lookup is applied, the red and green
components are set to 186, while the blue component is set to 0. The
resulting pixel is then yellow rather than grey (as it should be with
zero contrast).
There are two ways to fix this: Either clamping pixel colour channels to
the black level in debayering or setting the below black level entries
in the gamma lookup table to the lowest value of the gamma table rather
than 0. Both should have the same effect. Let's opt for the latter for
its simplicity.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Hans de Goede <johannes.goede@oss.qualcomm.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
If the histogram size is non-zero but lower than the number of bins,
yHistValsPerBin is zero and then the AGC processing crashes on division
by it. Let's check yHistValsPerBin for being zero and stop AGC
processing in such a case. The condition also covers the cases where
histogramSize or yHistValsPerBinMod are zero.
Tested-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Hans de Goede <hansg@kernel.org>
Tested-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The result type of the std::accumulate() call is uint64_t; let's use the
same type for the initial value (rather than the default int) to prevent
summing up the values in a different integer type.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Hans de Goede <hansg@kernel.org>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Tested-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The R/G/B sums computed in AWB simple IPA may be zero or perhaps even
negative. Let's make sure the sums are always positive, to prevent
division by zero or completely nonsense results.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Hans de Goede <hansg@kernel.org>
Tested-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
At the moment the blc code uses 255 as starting blacklevel for sensors
where there is no blacklevel info in the sensor-helper.
There are a number of issues with this:
1. When the first frame is bad (e.g. mostly white) which happens sometimes
the initial blacklevel will be kept leading to a divide by zero problem
in the AGC code (this divide by 0 problem is avoided in the AGC code by
not running the AGC algorithm).
2. Not runnning the AGC algorithm means that the gain/exposure do not
change, which causes the BLC algorithm to not run on the next frames,
so we keep the bad 255 blacklevel which stops AGC from running which
stops BLC from running. Leaving things stuck at a 255 blacklevel
resulting in an unusuable image.
3. Sometimes the auto-blc code leads to an unrealistic high
blacklevel detection which results in lower image quality.
To fix this start with a blacklevel of 16, which is the highest
(4096 / 256) blacklevel used for any sensor listing a blackLevel_
value in the sensor-helper class.
Note 2. could alternatively be fixed by disabling the check for
exposure/gain changes when the blacklevel is unrealistic high,
but that still leaves the other problems.
Signed-off-by: Hans de Goede <hansg@kernel.org>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Generating statistics for every single frame is not really necessary.
However a roundtrip through ipa_->processStats() still need to be done
every frame, even if there are no stats to make the IPA generate metadata
for every frame.
Add a valid flag to the statistics struct to let the IPA know when there
are no statistics for the frame being processed and modify the IPA to
only generate metadata for frames without valid statistics.
This is a preparation patch for skipping statistics generation for some
frames.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
At the moment when the overall image brightness is considered too high
the AGC code will lower the gain all the way down to againMin before
considering lowering the exposure.
What should happen instead is lower the gain no lower than 1.0 and after
that lower the exposure instead of lowering the gain.
Otherwise there might be a heavily overexposed image (e.g. all white)
which then is made less white by a gain < 1.0 which is no good.
When there is no sensor-helper, assume the driver reported default-gain
value is close to a gain of 1.0 .
While at it also remove the weird limitation to only lower the gain
when exposure is set to the maximum. As long as the gain is higher
than the default gain, the gain should be lowered first.
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Isaac Scott <isaac.scott@ideasonboard.com>
Signed-off-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Saturation control is added on top of the colour correction matrix. A
method of saturation adjustment that can be fully integrated into the
colour correction matrix is used. The control is available only if Ccm
algorithm is enabled.
The control uses 0.0-2.0 value range, with 1.0 being unmodified
saturation, 0.0 full desaturation and 2.0 quite saturated.
The saturation is adjusted by converting to Y'CbCr colour space,
applying the saturation value on the colour axes, and converting back to
RGB. ITU-R BT.601 conversion is used to convert between the colour
spaces, for no particular reason.
The colour correction matrix is applied before gamma and the given
matrix is suitable for such a case. Alternatively, the transformation
used in libcamera rpi ccm.cpp could be used.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
A colour correction matrix (CCM) is applied like this to an RGB pixel
vector P:
CCM * P
White balance must be applied before CCM. If CCM is used, software ISP
makes a combined matrix by multiplying the CCM by a white balance gains
CCM-like matrix (WBG). The multiplication should be as follows to do it
in the correct order:
CCM * (WBG * P) = (CCM * WBG) * P
The multiplication order in Lut software ISP algorithm is reversed,
resulting in colour casts. Let's fix the order.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Automatic black level setting in software ISP updates the determined
black level value when exposure or gain change. It stores the last
exposure and gain values to detect the change.
BlackLevel::configure() resets the stored black level value but not the
exposure and gain values. This can prevent updating the black value and
cause bad image output, e.g. after suspending and resuming a camera, if
exposure and gain remain unchanged.
Let's store exposure and gain in IPAActiveState. Although the values
are not supposed to be used outside BlackLevel class, storing them in
the context has the advantage of their automatic reset together with the
other context contents and having them in `blc' struct indicates their
relationship to the black value computation.
Bug: https://bugs.libcamera.org/show_bug.cgi?id=259
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Robert Mader <robert.mader@collabora.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Report exposure and gain in metadata.
This is more complicated than it could be expected because the exposure
value should be in microseconds but it's handled using V4L2_CID_EXPOSURE
control, which doesn't specify the unit, see
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/control.html.
So the unit conversion is done in the way rkisp1 IPA uses.
This requires getting and passing IPACameraSensorInfo around. To avoid
naming confusion and to improve consistency with rkisp1 IPA,
sensorCtrlInfoMap parameter is renamed to sensorControls.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch applies color correction matrix (CCM) in debayering if the
CCM is specified. Not using CCM must still be supported for performance
reasons.
The CCM is applied as follows:
[r1 g1 b1] [r]
[r2 g2 b2] * [g]
[r3 g3 b3] [b]
The CCM matrix (the left side of the multiplication) is constant during
single frame processing, while the input pixel (the right side) changes.
Because each of the color channels is only 8-bit in software ISP, we can
make 9 lookup tables with 256 input values for multiplications of each
of the r_i, g_i, b_i values. This way we don't have to multiply each
pixel, we can use table lookups and additions instead. Gamma (which is
non-linear and thus cannot be a part of the 9 lookup tables values) is
applied on the final values rounded to integers using another lookup
table.
Because the changing part is the pixel value with three color elements,
only three dynamic table lookups are needed. We use three lookup tables
to represent the multiplied matrix values, each of the tables
corresponding to the given matrix column and pixel color.
We use int16_t to store the precomputed multiplications. This seems to
be noticeably (>10%) faster than `float' for the price of slightly less
accuracy and it covers the range of values that sane CCMs produce. The
selection and structure of data is performance critical, for example
using bytes would add significant (>10%) speedup but would be too short
to cover the value range.
The color lookup tables can be represented either as unions,
accommodating tables for both the CCM and non-CCM cases, or as separate
tables for each of the cases, leaving the tables for the other case
unused. The latter is selected as a matter of preference.
The tables are copied (as before), which is not elegant but also not a
big problem. There are patches posted that use shared buffers for
parameters passing in software ISP (see software ISP TODO #5) and they
can be adjusted for the new parameter format.
Color gains from white balance are supposed not to be a part of the
specified CCM. They are applied on it using matrix multiplication,
which is simple and in correspondence with future additions in the form
of matrix multiplication, like saturation adjustment.
With this patch, the reported per-frame slowdown when applying CCM is
about 45% on Debix Model A and about 75% on TI AM69 SK.
Using std::clamp in debayering adds some performance penalty (a few
percent). The clamping is necessary to eliminate out of range values
possibly produced by the CCM. If it could be avoided by adjusting the
precomputed tables some way then performance could be improved a bit.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Applying color correction matrix (CCM) in software ISP is optional due
to performance reasons. CCM is applied if and only if `Ccm' algorithm
is present in the tuning file.
Software ISP debayering is a performance critical piece of code and we
do not want to use dynamic conditionals there. Therefore we pass
information about CCM application to debayering configuration and let it
select the right versions of debayering functions using templates. This
is a trick similar to the previously used one for adding or not adding
an alpha channel to the output.
Debayering gets this information but it ignores it in this patch.
Actual processing with CCM is added in the followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds color correction matrix (CCM) algorithm to software ISP.
It is based on the corresponding algorithm in rkisp1.
The primary difference against hardware pipelines is that applying the
CCM is optional. Applying CCM causes a significant slowdown, time
needed to process a frame raises by 40-90% on tested platforms. If CCM
is really needed, it can be applied, if not, it's better to stick
without it. This can be configured by presence or omission of Ccm
algorithm in the tuning file.
CCM is changed only if the determined temperature changes by at least
100 K (an arbitrarily selected value), to avoid recomputing the matrices
and lookup tables all the time.
Since the CCM is float, rather than double, to use the same type as in
the rkisp1 pipeline, the type of color gains is changed from double to
float.
The outputs of the algorithm are not used yet, they will be enabled in
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The AWB algorithm has data to determine color temperature of the image.
Let's compute the temperature from it and store it into the context.
This piece of information is currently unused but it will be needed in a
followup patch introducing support for color correction matrix.
Let's store the white balance related information under `awb' subsection
of the active state, as the hardware pipelines do.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
