The DW100 Dewarp engine is present on i.MX8MP SoC and possibly others.
This patch provides a dedicated converter module that allows easy
integration of such a dewarper into a pipeline handler.
In this patch only the ScalerCrop control is implemented. Support for
additional functionality will be added in later patches.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Using a custom vertex map the dw100 dewarper is capable of doing
complex and useful transformations on the image data. This class
implements a pipeline featuring:
- Arbitrary ScalerCrop
- Full transform support (Flip, 90deg rotations)
- Arbitrary move, scale, rotate
ScalerCrop and Transform is implemented to provide a interface that is
standardized libcamera wide. The rest is implemented on top for more
flexible dw100 specific features.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
To actually use requests with the m2m device, requests need to be
allocated on the underlying media device. This can only be done if the
media device is opened which means acquiring it. Add a function to check
if the m2m device supports requests by acquiring the media device,
asking it and then releasing it again. Also add a function to allocate
requests that acquires the internal media device and releases it after
allocating the requests.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add V4L2 request support to the V4L2M2MConverter class. Extend the
functions related to buffer queuing with an optional request parameter
that gets passed to the lower layers.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Isaac Scott <isaac.scott@ideasonboard.com>
The V4L2 requests API provides support to atomically tie controls to a
set of buffers. This is especially common for m2m devices. Such a
request is represented by an fd that is allocated via
MEDIA_IOC_REQUEST_ALLOC and then passed to the various V4L2 functions.
Implement a V4L2Request class to wrap such an fd and add the
corresponding utility functions.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
With the upcoming addition of V4L2 requests support, the converters need
to keep a handle to the corresponding media device.
Prepare for that by changing the constructor parameter from a raw
pointer to a shared pointer.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Isaac Scott <isaac.scott@ideasonboard.com>
Adapt the PipelineHandler::acquireMediaDevice() support function to
return a shared pointer instead of the underlying raw pointer.
Propagate this update to all pipeline handlers that use the MediaDevice
and store a std::shared_ptr<MediaDevice> accordingly.
This is required to support media devices that are potentially shared
among multiple pipeline handlers, like a dewarper implemented as v4l2
m2m device.
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Paul Elder <paul.elder@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Isaac Scott <isaac.scott@ideasonboard.com>
Instead of handling enums specially in the code generation templates,
create a specialization of `IPADataSerializer` that handles enums.
Every enum is serialized as a `uint32_t`, with `static_assert` to
ensure that every possible value fits. Previously, enums were
(de)serialized in `(de)serializer_field()` based on the size of
their underlying types. Afer this change, every enum is uniformly
handled as a `uint32_t`.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Simplify a bit by storing the `EventNotifier` objects directly in the
`std::map` instead of wrapping them in unique_ptr. An other advantage
is that it removes one allocation per fence.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Generating statistics for every single frame is not really necessary.
However a roundtrip through ipa_->processStats() still need to be done
every frame, even if there are no stats to make the IPA generate metadata
for every frame.
Add a valid flag to the statistics struct to let the IPA know when there
are no statistics for the frame being processed and modify the IPA to
only generate metadata for frames without valid statistics.
This is a preparation patch for skipping statistics generation for some
frames.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Even though there is an abstract class to represent the interface of an IPA,
the threaded and IPC versions are still multiplexed using the same type,
which uses a boolean to actually dispatch to the right function.
Instead of doing that, split the classes into "threaded" and "isolated"
variants, and make `IPAManager` choose accordingly.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch adds configuration options for environment variables used in
the IPA proxy.
The configuration snippet:
configuration:
ipa:
config_paths:
- config path 1
- config path 2
- ...
module_paths:
- module path 1
- module path 2
- ...
proxy_paths:
- proxy path 1
- proxy path 2
- ...
force_isolation: BOOL
LIBCAMERA_<IPA_NAME>_TUNING_FILE remains configurable only via the
environment variable; this is supposed to be used only for testing and
debugging and it's not clear what to do about IPA names like "rpi/vc4"
and "rpi/pisp" exactly.
There are two ways to pass the configuration to the places where it is
needed: Either to pass it as an argument to the method calls that need
it, or to pass it to the class constructors and extract the needed
configuration from there. This patch uses the second method as it is
less polluting the code.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Let's add some helpers to make accessing simple configuration values
simpler. The helpers are used in the followup patches.
GlobalConfiguration::option ensures that no value is returned rather
than a value of YamlObject::empty.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Global configuration is accessed via a GlobalConfiguration instance.
The instance is conceptually a singleton, but singletons are not welcome
in libcamera so we must store the (preferably single) instance
somewhere.
This patch creates a GlobalConfiguration instance in CameraManager and
defines the corresponding access method. CameraManager is typically
instantiated only once or a few times, it is accessible in many places
in libcamera and the configuration can be retrieved from it and passed
to other places if needed (it's read-only once created). Using
CameraManager for the purpose is still suboptimal and we use it only due
to lack of better options. An alternative could be Logger, which is
still a singleton and it's accessible from everywhere. But with Logger,
we have a chicken and egg problem -- GlobalConfiguration should log
contingent problems with the configuration when it's loaded but if it is
created in the logger then there are mutual infinite recursive calls.
One possible way to deal with this is to look at the environment
variables only during logging initialisation and apply the logging
configuration when a CameraManager is constructed. Considering there
are intentions to remove the Logger singleton, let's omit logging
configuration for now.
If there are multiple CameraManager instances, there are also multiple
GlobalConfiguration instances, each CameraManager instance is meant to
be fully independent, including configuration. They may or may not
contain the same data, depending on whether the global configuration
file in the file system was changed in the meantime.
The configuration is stored in the private CameraManager. It's
accessible within libcamera (via CameraManager) but it's not meant to be
accessed by applications.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Currently, libcamera can be configured in runtime using several
environment variables. With introducing more and more variables, this
mechanism reaches its limits. It would be simpler and more flexible if
it was possible to configure libcamera in a single file.
For example, there was a request to define pipeline precedence in
runtime. We want to compile in multiple pipelines, in order to have
them accessible within single packages in distributions. And then being
able to select among the pipelines manually as needed based on the
particular hardware or operating system environment. Having the
configuration file then allows easy switching between hardware, GPU or
CPU IPAs. The configuration file can also be used to enable or disable
experimental features and avoid the need to track local patches changing
configuration options hard-wired in the code when working on new
features.
This patch introduces basic support for configuration files.
GlobalConfiguration class reads and stores the configuration. Its
instance can be used by other libcamera objects to access the
configuration. A GlobalConfiguration instance is supposed to be stored
in a well-defined place, e.g. a CameraManager instance. It is possible
to have multiple GlobalConfiguration instances, which may or may not
make sense.
libcamera configuration can be specified using a system-wide
configuration file or a user configuration file. The user configuration
file takes precedence if present. There is currently no way to merge
multiple configuration files, the one found is used as the only
configuration file. If no configuration file is present, nothing
changes to the current libcamera behavior (except for some log
messages related to configuration file lookup).
The configuration file is a YAML file. We already have a mechanism for
handling YAML configuration files in libcamera and the given
infrastructure can be reused for the purpose. However, the
configuration type is abstracted to make contingent future change of the
underlying class easier while retaining (most of) the original API.
The configuration is versioned. This has currently no particular
meaning but is likely to have its purpose in future, especially once
configuration validation is introduced.
The configuration YAML file looks as follows:
---
version: 1
configuration:
WHATEVER CONFIGURATION NEEDED
This patch introduces just the basic idea. Actually using the
configuration in the corresponding places (everything what is currently
configurable via environment variables should be configurable in the
file configuration) and other enhancements are implemented in the
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Top-level `const` qualifiers are not useful, so avoid them. This is done
either by simply removing the top-level `const`, or making the function
return a reference to const where that is appropriate.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
The `ProcessManager` is a singleton class to handle `SIGCHLD` signals
and report the exit status to the particular `Process` instance.
However, having a singleton in a library is not favourable and it is
even less favourable if it installs a signal handler.
Using pidfd it is possible to avoid the need for the signal handler;
and the `Process` objects can watch their pidfd themselves, eliminating
the need for the `ProcessManager` class altogether.
`P_PIDFD` for `waitid()` was introduced in Linux 5.4, so this change
raises the minimum supported kernel version. `clone3()`, `CLONE_PIDFD`,
`pidfd_send_signal()` were all introduced earlier.
Furthermore, the call to the `unshare()` system call can be removed
as those options can be passed to `clone3()` directly.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Instead of creating a new vector, take the vector by value to make it
possible for the caller to use move construction when calling the function.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Instead of using a separate member variable, use `pid_ > 0` to determine
if the process is still running. Previously the value of `pid_` was not
reset to -1 when the process terminated, but since it is only meaningful
while the process is running, reset it to -1 in `Process::died()`.
Neither `pid_` nor `running_` are exposed, so this change has no effect
on the public interface or observable behaviour.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add a maxQueuedRequestsDevice constructor parameter to allow pipeline
handler classes to limit the maximum number of requests that get queued
to the device in queueRequestDevice().
The default value is set to an arbitrary number of 32 which is big
enough for all currently known use cases.
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Umang Jain <uajain@igalia.com>
Tested-by: Sven Püschel <s.pueschel@pengutronix.de>
The waiting requests of one camera should not be able to influence
queuing to another camera. Currently a request that is in the
waitingQueue and waiting for preparation blocks all requests of other
cameras even if they are already prepared. To fix that replace the
single waitingRequests_ queue with one queue per camera.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Umang Jain <uajain@igalia.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Sven Püschel <s.pueschel@pengutronix.de>
The ClockRecovery class takes pairs of timestamps from two different
clocks, and models the second ("output") clock from the first ("input")
clock.
We can use it, in particular, to get a good wallclock estimate for a
frame's SensorTimestamp.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
A `Process` object has address identity because a pointer to it is
stored inside the `ProcessManager`. However, copy/move special
methods are still generated by the compiler. So disable them to
avoid potential issues and confusion.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
`appendPOD()` does a single insertion, so if only a single `appendPOD()`
will be called on a vector before returning, then calling `reserve()`
is not that useful, so remove it.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
It is useful to multiply matrices and vectors of heterogeneous types, for
instance float and double. Extend the multiplication operator to support
this, avoiding the need to convert one of the operations. The type of the
returned vector is selected automatically to avoid loosing precision.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
It is useful to multiply matrices of heterogneous types, for instance
float and double. Extend the multiplication operator to support this,
avoiding the need to convert one of the matrices. The type of the
returned matrix is selected automatically to avoid loosing precision.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
By zero-initializing the data_ member we can make most functions
constexpr which will come in handy in upcoming patches. Note that this
is due to C++17. In C++20 we will be able to leave data_ uninitialized
for constexpr. The Matrix(std::array) version of the constructor can
not be constexpr because std::copy only became constexpr in C++20.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
The `signature()` getter can just return a reference to the private vector
member variable, and let the caller make a copy if needed. Since the
return type is const qualified, this was likely the original intention.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Extend the Simple IPA IPC to support returning a metadata ControlList
when the process call has completed.
A new signal from the IPA is introduced to report the metadata,
similarly to what the hardware pipelines do.
Merge the metadata reported by the ISP into any completing request to
provide to the application. Completion of a request is delayed until
this is done; this doesn't apply to canceled requests.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch applies color correction matrix (CCM) in debayering if the
CCM is specified. Not using CCM must still be supported for performance
reasons.
The CCM is applied as follows:
[r1 g1 b1] [r]
[r2 g2 b2] * [g]
[r3 g3 b3] [b]
The CCM matrix (the left side of the multiplication) is constant during
single frame processing, while the input pixel (the right side) changes.
Because each of the color channels is only 8-bit in software ISP, we can
make 9 lookup tables with 256 input values for multiplications of each
of the r_i, g_i, b_i values. This way we don't have to multiply each
pixel, we can use table lookups and additions instead. Gamma (which is
non-linear and thus cannot be a part of the 9 lookup tables values) is
applied on the final values rounded to integers using another lookup
table.
Because the changing part is the pixel value with three color elements,
only three dynamic table lookups are needed. We use three lookup tables
to represent the multiplied matrix values, each of the tables
corresponding to the given matrix column and pixel color.
We use int16_t to store the precomputed multiplications. This seems to
be noticeably (>10%) faster than `float' for the price of slightly less
accuracy and it covers the range of values that sane CCMs produce. The
selection and structure of data is performance critical, for example
using bytes would add significant (>10%) speedup but would be too short
to cover the value range.
The color lookup tables can be represented either as unions,
accommodating tables for both the CCM and non-CCM cases, or as separate
tables for each of the cases, leaving the tables for the other case
unused. The latter is selected as a matter of preference.
The tables are copied (as before), which is not elegant but also not a
big problem. There are patches posted that use shared buffers for
parameters passing in software ISP (see software ISP TODO #5) and they
can be adjusted for the new parameter format.
Color gains from white balance are supposed not to be a part of the
specified CCM. They are applied on it using matrix multiplication,
which is simple and in correspondence with future additions in the form
of matrix multiplication, like saturation adjustment.
With this patch, the reported per-frame slowdown when applying CCM is
about 45% on Debix Model A and about 75% on TI AM69 SK.
Using std::clamp in debayering adds some performance penalty (a few
percent). The clamping is necessary to eliminate out of range values
possibly produced by the CCM. If it could be avoided by adjusting the
precomputed tables some way then performance could be improved a bit.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Applying color correction matrix (CCM) in software ISP is optional due
to performance reasons. CCM is applied if and only if `Ccm' algorithm
is present in the tuning file.
Software ISP debayering is a performance critical piece of code and we
do not want to use dynamic conditionals there. Therefore we pass
information about CCM application to debayering configuration and let it
select the right versions of debayering functions using templates. This
is a trick similar to the previously used one for adding or not adding
an alpha channel to the output.
Debayering gets this information but it ignores it in this patch.
Actual processing with CCM is added in the followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The `V4L2BufferCache` type is not thread-safe. Its `lastUsedCounter_`
member is not used in contexts where its atomicity would matter.
So it does not need to be have an atomic type.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
`MediaEntity::ancillaryEntities()` can just return a const lvalue
reference to the underlying array, a copy need not be made. That
was likely the original intention.
Fixes: 9490c664b5 ("libcamera: Add members to MediaEntity to support ancillary entities")
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
