Extend the Simple IPA IPC to support returning a metadata ControlList
when the process call has completed.
A new signal from the IPA is introduced to report the metadata,
similarly to what the hardware pipelines do.
Merge the metadata reported by the ISP into any completing request to
provide to the application. Completion of a request is delayed until
this is done; this doesn't apply to canceled requests.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch applies color correction matrix (CCM) in debayering if the
CCM is specified. Not using CCM must still be supported for performance
reasons.
The CCM is applied as follows:
[r1 g1 b1] [r]
[r2 g2 b2] * [g]
[r3 g3 b3] [b]
The CCM matrix (the left side of the multiplication) is constant during
single frame processing, while the input pixel (the right side) changes.
Because each of the color channels is only 8-bit in software ISP, we can
make 9 lookup tables with 256 input values for multiplications of each
of the r_i, g_i, b_i values. This way we don't have to multiply each
pixel, we can use table lookups and additions instead. Gamma (which is
non-linear and thus cannot be a part of the 9 lookup tables values) is
applied on the final values rounded to integers using another lookup
table.
Because the changing part is the pixel value with three color elements,
only three dynamic table lookups are needed. We use three lookup tables
to represent the multiplied matrix values, each of the tables
corresponding to the given matrix column and pixel color.
We use int16_t to store the precomputed multiplications. This seems to
be noticeably (>10%) faster than `float' for the price of slightly less
accuracy and it covers the range of values that sane CCMs produce. The
selection and structure of data is performance critical, for example
using bytes would add significant (>10%) speedup but would be too short
to cover the value range.
The color lookup tables can be represented either as unions,
accommodating tables for both the CCM and non-CCM cases, or as separate
tables for each of the cases, leaving the tables for the other case
unused. The latter is selected as a matter of preference.
The tables are copied (as before), which is not elegant but also not a
big problem. There are patches posted that use shared buffers for
parameters passing in software ISP (see software ISP TODO #5) and they
can be adjusted for the new parameter format.
Color gains from white balance are supposed not to be a part of the
specified CCM. They are applied on it using matrix multiplication,
which is simple and in correspondence with future additions in the form
of matrix multiplication, like saturation adjustment.
With this patch, the reported per-frame slowdown when applying CCM is
about 45% on Debix Model A and about 75% on TI AM69 SK.
Using std::clamp in debayering adds some performance penalty (a few
percent). The clamping is necessary to eliminate out of range values
possibly produced by the CCM. If it could be avoided by adjusting the
precomputed tables some way then performance could be improved a bit.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Applying color correction matrix (CCM) in software ISP is optional due
to performance reasons. CCM is applied if and only if `Ccm' algorithm
is present in the tuning file.
Software ISP debayering is a performance critical piece of code and we
do not want to use dynamic conditionals there. Therefore we pass
information about CCM application to debayering configuration and let it
select the right versions of debayering functions using templates. This
is a trick similar to the previously used one for adding or not adding
an alpha channel to the output.
Debayering gets this information but it ignores it in this patch.
Actual processing with CCM is added in the followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
There may be pending messages in SoftwareIsp message queue when
SoftwareIsp stops. The call to IPAProxySoft::stop() will dispatch them
before SoftwareIsp::stop() finishes. But this is dependent on
IPAProxySoft::stop() implementation, let's break this dependency and
dispatch messages to SoftwareIsp explicitly in SoftwareIsp::stop().
This also allows dropping `running_' flag. Since the SoftwareIsp
messages get processed and invoke IPA calls before the IPA proxy is set
to ProxyStopping state and the SoftwareIsp worker thread is no longer
running, it's guaranteed that no new messages come to SoftwareIsp and
attempt to call the stopped IPA proxy.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When SoftwareIsp stops, input and output buffers queued to it may not
yet be fully processed. They will be eventually returned but stop means
stop, there should be no processing related actions invoked afterwards.
Let's stop forwarding processed input buffers from SoftwareIsp slots
when SoftwareIsp is stopped. Let's track the queued input buffers and
return them back for capture in SoftwareIsp::stop().
The returned input buffers are marked as cancelled. This is not
necessary at the moment but it gives the pipeline handlers chance to
deal with this if they need to.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When SoftwareIsp stops, input and output buffers queued to it may not
yet be fully processed. They will be eventually returned but stop means
stop, there should be no processing related actions invoked afterwards.
Let's stop forwarding processed output buffers from the SoftwareIsp
slots once SoftwareIsp is stopped. Let's track the queued output
buffers and mark those still pending as cancelled in SoftwareIsp::stop
and return them to the pipeline handler.
Dealing with input buffers is addressed in a separate patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Software ISP runs debayering in a separate thread and debayering may
emit statsReady when software ISP (including the IPA) is being stopped.
The signal waits in a queue and gets invoked later, resulting in an
assertion error when attempting to invoke a method on the stopped IPA:
FATAL default soft_ipa_proxy.cpp:456 assertion
"state_ == ProxyRunning" failed in processStatsThread()
Let's prevent this problem by forwarding the ISP stats signal from
software ISP only when the IPA is running. To track this,
SoftwareISP::running_ variable is introduced.
Making processing of the other signals in SoftwareISP more robust is
addressed in the followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reported-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
inputBufferReady ready signal in the simple pipeline is handled in the
pipeline handler thread. outputBufferReady and ispStatsReady signals
should be handled there too.
Rather than relying on the user of the SoftwareIsp instance, let
SoftwareIsp inherits Object. SoftwareIsp serves as a signal proxy, the
signals above are emitted from signal handlers. This means that if
SoftwareIsp inherits Object then the slots are invoked in SoftwareIsp
thread. Which is the camera manager thread because the SoftwareIsp
instance is created there.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch introduces support for applying runtime controls to software
ISP. It enables the contrast control as the first control that can be
used.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds Algorithm::queueRequest call for the defined algorithms.
As there are currently no control knobs in software ISP nor the
corresponding queueRequest call chain, the patch also introduces the
queueRequest methods and calls from the pipeline to the IPA.
This is preparation only since there are currently no Algorithm based
algorithms defined and no current software ISP algorithms support
control knobs.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds Algorithm::configure call for the defined algorithms.
This is preparation only since there are currently no Algorithm based
algorithms defined.
A part of this change is passing IPAConfigInfo instead of ControlInfoMap
to configure() calls as this is what Algorithm::configure expects.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
A previous preparation patch implemented passing frame ids to stats
processing but without actual meaningful frame id value passed there.
This patch extends that by actually providing the frame id and passing
it through to the stats processor.
The frame id is taken from the request sequence number, the same as in
hardware pipelines.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
This patch adds frame and bufferId arguments to stats related calls.
Although the parameters are currently unused, because frame ids are not
tracked and used and the stats buffer is passed around directly rather
than being referred by its id, they bring the internal APIs closer to
their counterparts in hardware pipelines.
It serves as a preparation for followup patches that will introduce:
- Frame number tracking in order to switch to DelayedControls
(software ISP TODO #11 + #12).
- A ring buffer for stats in order to improve passing the stats
(software ISP TODO #2).
Frame and buffer ids are unrelated for the given purposes but since they
are passed together at the same places, the change is implemented as a
single patch rather than two, basically the same, patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The converter interface uses the unsigned int output stream index to map
to the output frame buffers. This is cumbersome to implement new
converters because one has to keep around additional book keeping
to track the streams with their correct indexes.
The v4l2_converter_m2m and simple pipeline handler are adapted to
use the new interface. This work roped in software ISP as well,
which also seems to use indexes (although it doesn't implement converter
interface) because of a common conversionQueue_ queue used for
converter_ and swIsp_.
The logPrefix is no longer able to generate an index from a stream, and
is updated to be more expressive by reporting the stream configuration
instead, for example, reporting "1920x1080-MJPEG" in place of
"stream0".
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Tested-by: Andrei Konovalov <andrey.konovalov.ynk@gmail.com> # sm8250 RB5
Users of the DmaHeap class really just want some way to allocate
dma-buffers from userspace. This can also be done by using /dev/udmabuf
instead of using /dev/dma_heap/*.
Rename DmaHeap class to DmaBufAllocator in preparation of adding
/dev/udmabuf support.
And update the DmaHeap class docs to match including replacing references
to "dma-heap type" with "dma-buf provider".
This is a pure automated rename on the code ('s/DmaHeap/DmaBufAllocator/')
+ file renames + doc updates. There are no functional changes.
The DmaBufAllocator objects in vc4.cpp and software_isp.cpp are left named
dmaHeap_ to keep the changes to those 2 files to a minimum.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Bryan O'Donoghue <bryan.odonoghue@linaro.org> # Lenovo-x13s
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Constructing the color mapping tables is related to stats rather than
debayering, where they are applied. Let's move the corresponding code
to stats processing.
The same applies to the auxiliary gamma table. As the gamma value is
currently fixed and used in a single place, with the temporary exception
mentioned below, there is no need to share it anywhere anymore.
It's necessary to initialize SoftwareIsp::debayerParams_ to default
values. These initial values are used for the first two frames, before
they are changed based on determined stats. To avoid sharing the gamma
value constant in artificial ways, we use 0.5 directly in the
initialization. This all is not a particularly elegant thing to do,
such a code belongs conceptually to the similar code in stats
processing, but doing better is left for larger refactoring.
This is a preliminary step towards building this functionality on top of
libipa/algorithm.h, which should follow.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Andrei Konovalov <andrey.konovalov.ynk@gmail.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Source files in libcamera start by a comment block header, which
includes the file name and a one-line description of the file contents.
While the latter is useful to get a quick overview of the file contents
at a glance, the former is mostly a source of inconvenience. The name in
the comments can easily get out of sync with the file name when files
are renamed, and copy & paste during development have often lead to
incorrect names being used to start with.
Readers of the source code are expected to know which file they're
looking it. Drop the file name from the header comment block.
The change was generated with the following script:
----------------------------------------
dirs="include/libcamera src test utils"
declare -rA patterns=(
['c']=' \* '
['cpp']=' \* '
['h']=' \* '
['py']='# '
['sh']='# '
)
for ext in ${!patterns[@]} ; do
files=$(for dir in $dirs ; do find $dir -name "*.${ext}" ; done)
pattern=${patterns[${ext}]}
for file in $files ; do
name=$(basename ${file})
sed -i "s/^\(${pattern}\)${name} - /\1/" "$file"
done
done
----------------------------------------
This misses several files that are out of sync with the comment block
header. Those will be addressed separately and manually.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Black may not be represented as 0 pixel value for given hardware, it may
be higher. If this is not compensated then various problems may occur
such as low contrast or suboptimal exposure.
The black pixel value can be either retrieved from a tuning file for the
given hardware, or automatically on the fly. The former is the right
and correct method, while the latter can be used when a tuning file is
not available for the given hardware. Since there is currently no
support for tuning files in software ISP, the automatic, hardware
independent way, is always used. Support for tuning files should be
added in future but it will require more work than this patch.
The patch looks at the image histogram and assumes that black starts
when pixel values start occurring on the left. A certain amount of the
darkest pixels is ignored; it doesn't matter whether they represent
various kinds of noise or are real, they are better to omit in any case
to make the image looking better. It also doesn't matter whether the
darkest pixels occur around the supposed black level or are spread
between 0 and the black level, the difference is not important.
An arbitrary threshold of 2% darkest pixels is applied; there is no
magic about that value.
The patch assumes that the black values for different colors are the
same and doesn't attempt any other non-primitive enhancements. It
cannot completely replace tuning files and simplicity, while providing
visible benefit, is its goal. Anything more sophisticated is left for
future patches.
A possible cheap enhancement, if needed, could be setting exposure +
gain to minimum values temporarily, before setting the black level. In
theory, the black level should be fixed but it may not be reached in all
images. For this reason, the patch updates black level only if the
observed value is lower than the current one; it should be never
increased.
The purpose of the patch is to compensate for hardware properties.
General image contrast enhancements are out of scope of this patch.
Stats are still gathered as an uncorrected histogram, to avoid any
confusion and to represent the raw image data. Exposure must be
determined after the black level correction -- it has no influence on
the sub-black area and must be correct after applying the black level
correction. The granularity of the histogram is increased from 16 to 64
to provide a better precision (there is no theory behind either of those
numbers).
Reviewed-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add a CPU based SwStats implementation for SoftwareISP / SoftIPA use.
This implementation offers a configure function + functions to gather
statistics on a line by line basis. This allows CPU based software
debayering to call into interleave debayering and statistics gathering
on a line by line basis while the input data is still hot in the cache.
This implementation also allows specifying a window over which to gather
statistics instead of processing the whole frame.
Doxygen documentation by Dennis Bonke.
Tested-by: Bryan O'Donoghue <bryan.odonoghue@linaro.org> # sc8280xp Lenovo x13s
Tested-by: Pavel Machek <pavel@ucw.cz>
Reviewed-by: Pavel Machek <pavel@ucw.cz>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Co-developed-by: Andrey Konovalov <andrey.konovalov@linaro.org>
Signed-off-by: Andrey Konovalov <andrey.konovalov@linaro.org>
Co-developed-by: Pavel Machek <pavel@ucw.cz>
Signed-off-by: Pavel Machek <pavel@ucw.cz>
Co-developed-by: Dennis Bonke <admin@dennisbonke.com>
Signed-off-by: Dennis Bonke <admin@dennisbonke.com>
Co-developed-by: Marttico <g.martti@gmail.com>
Signed-off-by: Marttico <g.martti@gmail.com>
Co-developed-by: Toon Langendam <t.langendam@gmail.com>
Signed-off-by: Toon Langendam <t.langendam@gmail.com>
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
