The algorithm uses the data type std::vector<std::array<double, 4>> to
represent the large sparse matrices that are XY (X, Y being the ALSC
grid size) high but with only 4 non-zero elements on each row.
Replace this slightly long type name by SparseArray<double>.
No functional changes.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The Array2D class is a very thin wrapper round std::vector that can be
used almost identically in the code, but it carries its 2D size with
it so that we aren't passing it around all the time.
All the std::vectors that were X * Y in size (X and Y being the ALSC
grid size) have been replaced. The sparse matrices that are XY * 4 in
size have not been as they are somewhat different, are used
differently, require more code changes, and actually make things more
confusing if everything looks like an Array2D but are not the same.
There should be no change in algorithm behaviour at all.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Remove any hard-coded assumptions about the target hardware platform
from the ALSC algorithm. Instead, use the "target" string provided by
the camera tuning config and generalised statistics structures to
determing parameters such as grid and region sizes.
The ALSC calculations use run-time allocated arrays/vectors on every
frame. Allocating these might add a non-trivial run-time penalty.
Replace these dynamic allocations with a set of reusable pre-allocated
vectors during the init phase.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add a new Controller::HardwareConfig structure that captures the
hardware statistics grid/histogram sizes and pipeline widths. This
ensures there is a single centralised places for these parameters.
Add a getHardwareConfig() helper function to retrieve these values for a
given hardware target.
Update the statistics populating routine in the IPA to use the values
from this structure instead of the hardcoded numbers.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The target string may be used by algorithms to determine the running
hardware target.
Store the target string provided by the camera tuning files in the
controller state. Add a getTarget() member function to retrieve this
string.
Validate the correct hardware target ("bcm2835") during the IPA
initialisation phase.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Make a copy of the CameraMode structure on a switch mode call. This
replaces the existing lastSensitivity_ field.
Limit the AGC gain calculations to the minimum value given by the
CameraMode structure. The maximum value remains unclipped as any gain
over the sensor maximum will be made up by digital gain.
Rename clipShutter to limitShutter for consistency, and have the latter
limit the shutter speed to both upper and lower bounds.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Use the new analogue gain and shutter speed limit fields in the ipa
code when reporting back the control value limits and calculating the
analogue gain code to use. This also replaces the now unused (and
removed) maxSensorGainCode_ field.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add fields in the CameraMode structure to capture the mode specific
limits for analogue gain and shutter speed. For convenience, also add
fields for minimum and maximum frame durations.
Populate these new fields when setting up the CameraMode structure.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When compiled with LTO (the default on Ubuntu), the global static
objects camHelpers and algorithms cause a crash in raspberrypi_ipa_proxy
at runtime as they're not allocated by the time the registration
routines execute.
This is a fairly crude fix which just converts the global static objects
into local static objects inside an equivalently named function.
Signed-off-by: Dave Jones <dave.jones@canonical.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Tested-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Some updates to the tuning for the imx296 sensors.
For the colour variant:
* Minor change to the AWB curve, making things a little less green.
* Updated CCMs that reduce colour saturation to a more accurate level.
Thanks to Dr. Rolf Henkel for these measurements and calculations.
* Sharpening has been toned down quite a lot.
* rpi.focus algorithm added so that the focus measure can be accessed.
The sharpening and focus changes are applied to the mono version of
the sensor too as we expect similar characteristics.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The maxVal variable in the computeInitialY function needs to be a
uint64_t, otherwise the subsequent multiplications in the function
can overflow on relatively high resolution images (when the counts in
the regions go over 16 bits).
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The existing mechanism of setting a timeout value simply uses the
maximum possible frame length advertised by the sensor mode. This can be
problematic when, for example, the IMX477 sensor can use a frame length
of over 600 seconds. However, for typical usage the frame length will
never go over several 100s of milliseconds, making the timeout very
impractical.
Store a list of the last 10 frame length values requested by the AGC. On
startup, and at the end of every frame, take the maximum frame length
value from this list and return that to the pipeline handler through the
setCameraTimeoutValue() signal. This allows the timeout value to better
track the actual sensor usage.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add an explicit helper function setCameraTimeout() in the pipeline
handler to set the Unicam timeout value. This function is signalled from
the IPA to set up an appropriate timeout. This replaces the
maxSensorFrameLengthMs value parameter returned back from
IPARPi::start().
Adjust the timeout to be 5x the maximum frame duration reported by the
IPA.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The RkISP1 implementation of the LensShadingCorrection algorithm has been
made adaptive to the scene color temperature in commit 14c869c00f ("ipa:
rkisp1: Take into account color temperature during LSC algorithm").
The LSC algorithm interpolates the correction factors using the
table's reference color temperatures. When calculating the interpolation
coefficients, an unintended integer division makes both coefficient
zeros resulting in a completely black image.
Fix this by type casting to double one of the division operands.
Fixes: 14c869c00f ("ipa: rkisp1: Take into account color temperature during LSC algorithm")
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
The VC4 ISP uses a pipeline bit-depth of 13-bits. The AGC algorithm needs to
know this bit-depth when computing the Y value for the image.
Instead of hardcoding the VC4 bit-depth in the AGC source code, normalise all
region sums to 16-bits when filling the Statistics structure. AWB and ALSC are
agnostic about pipeline depth, so do not need changing.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Repurpose the StatisticsPtr type from being a shared_ptr<bcm2835_isp_stats> to
shared_ptr<RPiController::Statistics>. This removes any hardware specific header
files and structures from the algorithms source code.
Add a new function in the Raspberry Pi IPA to populate the generic statistics
structure from the values provided by the hardware in the bcm2835_isp_stats
structure.
Update the Lux, AWB, AGC, ALSC, Contrast, and Focus algorithms to use the
generic statistics structure appropriately in their calculations. Additionally,
remove references to any hardware specific headers and defines in these source
files.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Tested-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Release the statistics buffer after running the through the AWB calculations.
Only the "counted" statistics are copied out to a local structure, so keeping
the statistics buffer allows the algorithm to see the "uncounted" statistics as
well.
This is currently handled by hard-coding the total number of statistics regions
regions based on the structure definition in the bcm2835_isp_stats structure.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
At present, the controller algorithms access the bcm2835_isp_stats structure,
which is hardware specific. It would be desirable to abstract out the statistics
structure to remove hardware specific headers from the algorithms source files.
Define a new templated RegionStats class that encompasses region based
statistics generated by the ISP. For the VC4 ISP, this can be used to hold
RGB sums and focus FoM values.
Define a new Statistics structure that holds all the VC4 ISP statistics output.
This includes AGC histograms, AGC/AWB region sums and focus FoM regions.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add support for Raspberry Pi Camera 3 modules (Sony IMX708 camera sensor) to the
Raspberry Pi IPA. These modules are available in either normal or wide angle
lens, both with IR or no IR cut options, giving a total for 4 variants. Provide
IQ tuning files for all four variants.
The IMX708 camera helper additionally parses PDAF and HDR histogram data that
is provided in the embedded data stream from Unicam.
Signed-off-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add the current frame's lens position (in dioptres) to the DeviceStatus
structure. This value is obtained from the AfAlgorithm::getLensPosition()
member function. Return this lens position back to the pipeline handler to
store in the metadata field of the request.
As a drive-by, fixup some inaccurate comments in the DeviceStatus structure.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Nick Hollinghurst nick.hollinghurst@raspberrypi.com
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Provide the first version of the Raspberry Pi autofocus algorithm. This
implementation uses a hybrid of contrast detect autofocus (CDAF) and phase
detect autofocus (PDAF) statistics. PDAF is always preferred over CDAF due to
having less "hunting" behavior.
Signed-off-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add a new AfAlgorithm class that defines the interface for the Raspberry Pi
autofocus algorithm.
Add an AfStatus structure that returns the results of the autofocus algorithm
back to the IPA.
Add a structure for PDAF statistics provided by supported sensors for use with
an autofocus algorithm. This structure is currently based on the IMX708's PDAF
"Type 1" statistics.
Signed-off-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Pass the available lens controls to the IPA through the configure() function.
Validate that the V4L2_CID_FOCUS_ABSOLUTE does exist. If it doesn't, log a
warning message, and do not advertise focus related controls from the IPA.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Support for the OmniVision OV8858 sensor is scheduled for inclusion in
the Linux kernel in version v6.3.
Add support for the sensor in libcamera by providing static properties
and a camera sensor helper in libipa.
The camera sensor helper expresses analogue gain increments in 1/128
step which differs from what is reported in the sensor documentation in
section "5.8 manual exposure compensation/ manual gain compensation" [0]
A more detailed analysis of the sensor gain model is reported at:
https://patchwork.linuxtv.org/project/linux-media/patch/20221106171129.166892-2-nicholas@rothemail.net/#142267
Record with a \todo note a reference to discussion on the gain model
implementation.
Signed-off-by: Nicholas Roth <nicholas@rothemail.net>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add support for Omnivision OV4689 image sensor to libipa.
The sensor implements a linear gain equation with analogue gain
control values 0-2047 corresponding to 0x-16x physical gain range.
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Signed-off-by: Mikhail Rudenko <mike.rudenko@gmail.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
The imx477_scientific.json file was added to the source tree with
incorrect executable permission. Fix it.
Fixes: bf66a1d251 ("ipa: raspberrypi: Add a "scientific" tuning for the IMX477")
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add support for OnSemi AR0521 5Mpx image sensor to libipa.
The sensor analogue gain is implemented as a coarse and a fine factor,
with the coarse gain being a power of two and the fine gain being a
value in the [1.0, 2.0[ range. The mapping between gain codes and gain
values is tabulated in the datasheet, and the table values are very
close but not identical to the mathematical model. Compute the gain
using the model to keep the code shorter, if this causes precision
issues the calculation could be replaced with a table.
Signed-off-by: Jacopo Mondi <jacopo@jmondi.org>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add a tuning file for the IMX477 more suited to scientific applications.
The key differences from the original tuning file are:
- Disable ALSC block completely
- Pure rec709 gamma curve, and no contrast enhance
- New CT curve and CCMs based on the illumination spectrum of a black body
radiator up to about 3600 K and the CIE illuminant D for higher color
temperatures.
Further details on the changes can be found at:
https://forums.raspberrypi.com/viewtopic.php?t=343449
All credit for these changes go to Dr. Rolf Henkel.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Rolf Henkel <cpixip@pixelcircus.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The latest camera modules have a very slightly different IR filter, so
the tuning file is slightly revised to give best results with both old
and new camera modules.
The original tuning file is retained as imx477_v1.json in case anyone
should wish to continue using it.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Fix a bug in the IPA frame dropping (for rate control) logic, where the
metadata for the current context was copied from itself (i.e. a no-op), instead
of being copied from the previous context.
This bug does not occur in normal conditions, only when running with a low
exposure time and unconstrained framerate, which happens in a particular
picamera2 test.
Fixes: 546154b134 ("pipeline: ipa: raspberrypi: Use IPA cookies")
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The digital gain calculation uses a total exposure value computed with the
current AGC state. However, this is wrong in the case of manual shutter/gain
controls, as the total exposure value used must be the value computed when the
AGC sent the manual shutter/gain controls to the pipeline handler to action.
To fix this, the IPA now adds the historical AgcStatus structure to the metadata
(tagged with "agc.delayed_status"). This historical AgcStatus structure contains
the total exposure value calculated when the AGC sent the manual shutter/gain
controls to the pipeline handler.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Tested-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Pass an IPA cookie from the pipeline handler to the IPA and eventually back to
the pipeline handler through the setDelayedControls signal. This cookie is used
to index the RPiController::Metadata object to be used for the frame.
The IPA cookie is then returned from DelayedControls when the frame with the
applied controls has been returned from the sensor, and eventually passed back
to the IPA from the signalIspPrepare signal.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Tested-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Allow the IPA to cycle through an array of RPiController::Metadata objects, one
per prepare()/process() invocation, when running the controller algorithms. This
allows historical metadata objects to be retained, and subsequently passed into
the controller algorithms on future frames. At present, only a single index into
this array is used.
This change provides a route to fixing a problem with the AGC algorithm, where
if a manual shutter/gain is requested, the algorithm does not currently retain
any context of the total exposure that it has calculated. As a result, the
wrong digital gain would be applied when the frame with the manual shutter/gain
is processed by the ISP.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Tested-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add a new member function to RPiController::Metadata that copies unique
key/value pairs from one object to the other. This is different from
std::map::merge that would remove the key/value pairs from the source object.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
