The black level is likely to get updated, if ever, only after exposure
or gain changes. Don't compute its possible updates if exposure and
gain are unchanged.
It's probably not worth trying to implement something more
sophisticated. Better to spend the effort on supporting tuning files.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This is the last step to fully convert software ISP to Algorithm-based
processing.
The newly introduced frameContext.sensor parameters are set, and the
updated code moved, before calling Algorithm::process() to have the
values up-to-date in stats processing.
Resolves software ISP TODO #10.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Use the standard libcamera mechanism to report the "current" controls
rather than delaying updates by counting from the last update.
A problem is that with software ISP we cannot be sure about the sensor
delay. The original implementation simply skips exposure updates for 2
frames, which should be enough in most cases. After this change, we
assume the delay being exactly 2 frames, which may or may not be correct
and may work with outdated values if the delay is shorter.
According to Kieran, the wrong parts are also wrong on the
IPU3/RKISP1/Mali pipelines and only RPi have this correct. We need to
fix this, by correctly specifying the gains in the libipa camera sensor
helpers. The sooner the better because this change could introduce a
risk of increasing oscillations.
This patch also prepares moving exposure+gain to an algorithm module
where the original delay mechanism would be a (possibly unnecessary)
complication.
Resolves software ISP TODO #11 + #12.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
It's more natural to represent color gains as floating point numbers
rather than using a particular pixel-related representation.
double is used rather than float because it's a more common floating
point type in libcamera algorithms. Otherwise there is no obvious
reason to select one over the other here.
The constructed color tables still use integer representation for
efficiency.
Black level still uses pixel (integer) values, for consistency with
other libcamera parts.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
After black level handling has been moved to an algorithm module, white
balance and the construction of color tables can be moved to algorithm
modules too.
This time, the moved code is split between stats processing and
parameter construction methods. It is also split to two algorithm
modules:
- White balance computation.
- Gamma table computation and color lookup tables construction. While
this applies the color gains computed by the white balance algorithm,
it is not directly related to white balance. And we may want to
modify the color lookup tables in future according to other parameters
than just gamma and white balance gains.
Gamma table computation and color lookup tables construction could be
split to separate algorithms too. But there is no big need for that now
so they are kept together for simplicity.
This is the only part of the software ISP algorithms that sets the
parameters so emitting setIspParams can be moved to prepare() method.
A more natural representation of the gains (and black level) would be
floating point numbers. This is not done here in order to minimize
changes in code movements. It will be addressed in a followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Acked-by: Umang Jain <umang.jain@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The black level determination, already present as a separate class, can
be moved to the prepared Algorithm processing structure. It is the
first of the current software ISP algorithms that is called in the stats
processing sequence, which means it is also the first one that should be
moved to the new structure. Stats processing starts with calling
Algorithm-based processing so the call order of the algorithms is
retained.
Movement of this algorithm is relatively straightforward because all it
does is processing stats.
The comment about getting black level from the tuning files is dropped.
The black level will be taken from CameraSensorHelper if available and
that will be implemented in one of the followup patches.
Black level is now recomputed on each stats processing. In a future
patch, after DelayedControls are used, this will be changed to recompute
the black level only after exposure/gain changes.
The black level depends on the sensor used, the computed value can be
reused in a followup capture sessions with the same sensor. Thus it is
sufficient to (re)set the initial value in BlackLevel::init.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds Algorithm::process call for the defined algorithms.
This is preparation only since there are currently no Algorithm based
algorithms defined.
As software ISP currently doesn't produce any metadata, a dummy and
unused metadata instance is created to satisfy Algorithm::process API.
This should be changed in future.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds Algorithm::queueRequest call for the defined algorithms.
As there are currently no control knobs in software ISP nor the
corresponding queueRequest call chain, the patch also introduces the
queueRequest methods and calls from the pipeline to the IPA.
This is preparation only since there are currently no Algorithm based
algorithms defined and no current software ISP algorithms support
control knobs.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds Algorithm::configure call for the defined algorithms.
This is preparation only since there are currently no Algorithm based
algorithms defined.
A part of this change is passing IPAConfigInfo instead of ControlInfoMap
to configure() calls as this is what Algorithm::configure expects.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
A previous preparation patch implemented passing frame ids to stats
processing but without actual meaningful frame id value passed there.
This patch extends that by actually providing the frame id and passing
it through to the stats processor.
The frame id is taken from the request sequence number, the same as in
hardware pipelines.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds frame and bufferId arguments to stats related calls.
Although the parameters are currently unused, because frame ids are not
tracked and used and the stats buffer is passed around directly rather
than being referred by its id, they bring the internal APIs closer to
their counterparts in hardware pipelines.
It serves as a preparation for followup patches that will introduce:
- Frame number tracking in order to switch to DelayedControls
(software ISP TODO #11 + #12).
- A ring buffer for stats in order to improve passing the stats
(software ISP TODO #2).
Frame and buffer ids are unrelated for the given purposes but since they
are passed together at the same places, the change is implemented as a
single patch rather than two, basically the same, patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Software ISP image processing algorithms are currently defined in a
simplified way, different from other libcamera pipelines. This is not
good for several reasons:
- It makes the software ISP code harder to understand due to its
different structuring.
- Adding more algorithms may make the code harder to understand
generally.
- Mass libcamera code changes may not be easily applicable to software
ISP.
- Algorithm sharing with other pipelines is not easily possible.
This patch introduces basic software ISP IPA skeletons structured
similarly to the other pipelines. The newly added files are currently
not used or compiled and the general skeleton structures don't contain
anything particular. It is just a preparation step for a larger
refactoring and the code will be actually used and extended as needed in
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
IPA modules use custom namespaces for all their internal components to
avoid namespace clashes. The simple IPA module for the software ISP uses
libcamera::ipa::soft for this purpose. It however defines an internal
class named BlackLevel in the root of the libcamera namespace, making it
prone to clashes. Move it to the ipa::soft namespace along with the rest
of the code.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Now that controls can be queried for array information, print it in
--list-controls when applicable.
Example output (with dummy controls added to vimc):
$ cam -c 1 --list-controls
Using camera platform/vimc.0 Sensor B as cam0
Control: ColourGains: [1.000000..4.000000]
Size: 2
Control: Brightness: [-1.000000..1.000000]
Control: AfWindows: [(0, 0)/1x1..(0, 0)/100x100]
Size: n
Control: Contrast: [0.000000..2.000000]
Control: Saturation: [0.000000..2.000000]
Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Now that enum names can be obtained from ControlId, use that information
to print out the list of supported enum values in --list-controls.
Example output (with a dummy AwbMode ControlInfo added to vimc):
$ cam -c 1 --list-controls
Using camera platform/vimc.0 Sensor B as cam0
Control: AwbMode:
- AwbTungsten (2)
- AwbFluorescent (3)
- AwbDaylight (5)
Control: Brightness: [-1.000000..1.000000]
Control: Contrast: [0.000000..2.000000]
Control: Saturation: [0.000000..2.000000]
Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add to ControlId information about the names and values of enum, in the
event that the ControlId is an enum type. This allows applications to
query the ControlId for the names of the enum values, so that they can
be displayed on a UI, for example. Without this, it was necessary to use
macros of NameOfControlNameValueMap, which is difficult to use and is
very inflexible.
There already exists a map from name -> value in generated code. Reuse
this and pass it to the ControlId constructor, which in turn generates
the reverse map. The reverse map is then exposed to applications.
Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add a loader that is capable of loading polynomial coefficients from the
tuning files. The polynomial is sampled at load time to reduce the
computational overhead at runtime.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
For the LSC algorithm to dynamically calculate the LSC tables based on
the sensor size and the crop rectangle it needs access to that data.
Provide access to it by adding the sensorInfo object to the context.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Now, that the generic interpolator is available, use it to do the
interpolation of the lens shading tables. This makes the algorithm
easier to read and remove some duplicate code.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
The MatrixInterpolator is great for interpolation of matrices for
different color temperatures. It has however one limitation - it can
only handle matrices. For LSC it would be great to interpolate the LSC
tables (or even polynomials) using the same approach. Add a generic
Interpolator class based on the existing MatrixInterpolator. This class
can be adapted to any other type using partial template specialization.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
When accessing a nonexistent key on a dict the YamlObject returns an
empty element. This element can happily be cast to a string which is
unexpected. For example the following statement:
yamlDict["nonexistent"].get<string>("default")
is expected to return "default" but actually returns "". Fix this by
introducing an empty type to distinguish between an empty YamlObject and
a YamlObject of type value containing an empty string. For completeness
add an isEmpty() function and an explicit cast to bool to be able to
test for that type.
Extend the tests accordingly.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Using `DMA_BUF_IOCTL_SYNC` is required for DMABUFs in order to ensure
correct output. Not doing so currently results in occasional tearing
and/or backlashes in GL/VK clients that use the buffers directly for
rendering.
An alternative approach to have the sync code in `MappedFrameBuffer` was
considered but rejected for now, in order to allow clients more
flexibility.
While the new helper is added to an annoymous namespace, add
timeDiff to the same namespace and remove the static definition as a
drive by.
Signed-off-by: Robert Mader <robert.mader@collabora.com>
Tested-by: Milan Zamazal <mzamazal@redhat.com> # Debix
Tested-by: Hans de Goede <hdegoede@redhat.com> # IPU6 + ov2740
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com> # Lenovo X13s + OV5675
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Replace the two open-coded implementations of a link representation
with the operator<< overload string representation to simplify
the code and unify appearance of reporting MediaLinks.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Various parts of libcamera print the representation of a MediaLink by
inline joining the parts to make a string representation.
This repeated use case can be supported with a common helper to print
the MediaLink in a common manner using the existing toString() and
operator<< overload style to make it easier to report on MediaLink
types.
This implementation will report in the following style:
'imx283 1-001a'[0] -> 'video-mux'[0]
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The handling for the sequence number validation within
V4L2VideoDevice::dequeueBuffer makes use of a std::optional, which can
be used as a boolean in conditional statements. This has the impact in
this use case that it can be mis-read to be interpretting the value for
firstFrame_ which is assigned as the buf.sequence.
Remove this potential for confusion by making it clear that the first
frame handling is only performed when firstFrame_ does not have a value
assigned.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
[Kieran: Rework commit message]
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Mimic the letterbox behaviour of the Qt viewfinder by rendering the
image centered. This is done by adding a projection matrix to the vertex
shader to scale the rendered rectangle.
Another option would have been to keep using glViewport() (which would
have needed to be moved to paintGL(), as Qt resets the viewport to span
the full widget before calling). Hidpi displays would then need special
handling of the device pixel ratio, which is done automatically by Qt
when it sets the default viewport. Using a projection matrix avoids this
complication.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
There's no need to call glClearColor() twice before drawing any GL
content. Drop the first call. This doesn't introduce any functional
change.
While at it, pass floats instead of doubles to glClearColor(), as
required by the function.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Starting in Qt 6.7.0, vertex buffers and shader programs are unbound
just before calling QOpenGLWidget::paintGL(). This breaks rendering in
the GL viewfinder in two ways.
First, we bind the vertex buffer only once at initialization time. There
is therefore no vertex buffer mapped at rendering time, preventing both
the vertex shader from having access to the vertex and texture
coordinates.
Then, we bind the shader program only when rendering the first frame.
There is thus no shader program bound for all subsequent frames,
breaking rendering.
Fix this by binding the vertex buffer where needed, when setting
attribute buffers for the shader program, and binding the shader program
for every frame.
As we use a single vertex buffer, we could bind it at the beginning of
paintGL() and keep it bound indefinitely. That would however fail to
clearly indicate in the source code where the vertex buffer is needed,
making the code more difficult to understand as it would rely on
implicit assumptions. Release the vertex buffer explicitly when we don't
need it anymore to avoid this.
While at it, fix a coding style violation by adding missing curly
brackets.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When the widget's aspect ratio doesn't match the camera aspect ratio,
the viewfinder is rendered letter-boxed. The side rectangles are not
painted by the viewfinder, and Qt thus renders the parent widget
background to fill that space.
To make it black, we have two options:
- The simplest option is to set the widget's autoFillBackground property
to true. This causes Qt to paint the whole widget with its background
colour before calling paintEvent(). As the camera image typically
covers most (if not all) of the viewfinder widget, this is less
efficient.
- The more complicated option is to paint the letterbox rectangles
manually. We can additionally set the widget's WA_OpaquePaintEvent
attribute to instruct Qt to skip painting the parent widget. This
reduces CPU usage by about 1% (and may reduce GPU usage as well).
Note that the WA_OpaquePaintEvent attribute has to be disabled when we
render the stopped icon, as the icon has a transparent background.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The V4L2 adaptation layer can already support streaming with components
such as OpenCV, however it is not accepting, or handling any requests to
configure the frame rate.
In V4L2 the frame rate is set by configuring the timeperframe component
of the v4l2_streamparm structure through the VIDIOC_S_PARM ioctl.
Extend the V4L2 compatibility layer to accept the VIDIOC_S_PARM ioctls
and provide an interface for setting controls on the V4L2Camera class to
set the requested rate when starting the camera.
Signed-off-by: Nejc Galof <galof.nejc@gmail.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Milan Zamazal
Paul Elder
André Apitzsch
Stefan Klug
Robert Mader
Kieran Bingham
will whang
Laurent Pinchart
Luca Weiss
Nejc Galof