Generating statistics for every single frame is not really necessary.
However a roundtrip through ipa_->processStats() still need to be done
every frame, even if there are no stats to make the IPA generate metadata
for every frame.
Add a valid flag to the statistics struct to let the IPA know when there
are no statistics for the frame being processed and modify the IPA to
only generate metadata for frames without valid statistics.
This is a preparation patch for skipping statistics generation for some
frames.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Milan Zamazal <mzamazal@redhat.com>
Tested-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Hans de Goede <hansg@kernel.org>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Even though there is an abstract class to represent the interface of an IPA,
the threaded and IPC versions are still multiplexed using the same type,
which uses a boolean to actually dispatch to the right function.
Instead of doing that, split the classes into "threaded" and "isolated"
variants, and make `IPAManager` choose accordingly.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch adds configuration options for environment variables used in
the IPA proxy.
The configuration snippet:
configuration:
ipa:
config_paths:
- config path 1
- config path 2
- ...
module_paths:
- module path 1
- module path 2
- ...
proxy_paths:
- proxy path 1
- proxy path 2
- ...
force_isolation: BOOL
LIBCAMERA_<IPA_NAME>_TUNING_FILE remains configurable only via the
environment variable; this is supposed to be used only for testing and
debugging and it's not clear what to do about IPA names like "rpi/vc4"
and "rpi/pisp" exactly.
There are two ways to pass the configuration to the places where it is
needed: Either to pass it as an argument to the method calls that need
it, or to pass it to the class constructors and extract the needed
configuration from there. This patch uses the second method as it is
less polluting the code.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Let's add some helpers to make accessing simple configuration values
simpler. The helpers are used in the followup patches.
GlobalConfiguration::option ensures that no value is returned rather
than a value of YamlObject::empty.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Global configuration is accessed via a GlobalConfiguration instance.
The instance is conceptually a singleton, but singletons are not welcome
in libcamera so we must store the (preferably single) instance
somewhere.
This patch creates a GlobalConfiguration instance in CameraManager and
defines the corresponding access method. CameraManager is typically
instantiated only once or a few times, it is accessible in many places
in libcamera and the configuration can be retrieved from it and passed
to other places if needed (it's read-only once created). Using
CameraManager for the purpose is still suboptimal and we use it only due
to lack of better options. An alternative could be Logger, which is
still a singleton and it's accessible from everywhere. But with Logger,
we have a chicken and egg problem -- GlobalConfiguration should log
contingent problems with the configuration when it's loaded but if it is
created in the logger then there are mutual infinite recursive calls.
One possible way to deal with this is to look at the environment
variables only during logging initialisation and apply the logging
configuration when a CameraManager is constructed. Considering there
are intentions to remove the Logger singleton, let's omit logging
configuration for now.
If there are multiple CameraManager instances, there are also multiple
GlobalConfiguration instances, each CameraManager instance is meant to
be fully independent, including configuration. They may or may not
contain the same data, depending on whether the global configuration
file in the file system was changed in the meantime.
The configuration is stored in the private CameraManager. It's
accessible within libcamera (via CameraManager) but it's not meant to be
accessed by applications.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Currently, libcamera can be configured in runtime using several
environment variables. With introducing more and more variables, this
mechanism reaches its limits. It would be simpler and more flexible if
it was possible to configure libcamera in a single file.
For example, there was a request to define pipeline precedence in
runtime. We want to compile in multiple pipelines, in order to have
them accessible within single packages in distributions. And then being
able to select among the pipelines manually as needed based on the
particular hardware or operating system environment. Having the
configuration file then allows easy switching between hardware, GPU or
CPU IPAs. The configuration file can also be used to enable or disable
experimental features and avoid the need to track local patches changing
configuration options hard-wired in the code when working on new
features.
This patch introduces basic support for configuration files.
GlobalConfiguration class reads and stores the configuration. Its
instance can be used by other libcamera objects to access the
configuration. A GlobalConfiguration instance is supposed to be stored
in a well-defined place, e.g. a CameraManager instance. It is possible
to have multiple GlobalConfiguration instances, which may or may not
make sense.
libcamera configuration can be specified using a system-wide
configuration file or a user configuration file. The user configuration
file takes precedence if present. There is currently no way to merge
multiple configuration files, the one found is used as the only
configuration file. If no configuration file is present, nothing
changes to the current libcamera behavior (except for some log
messages related to configuration file lookup).
The configuration file is a YAML file. We already have a mechanism for
handling YAML configuration files in libcamera and the given
infrastructure can be reused for the purpose. However, the
configuration type is abstracted to make contingent future change of the
underlying class easier while retaining (most of) the original API.
The configuration is versioned. This has currently no particular
meaning but is likely to have its purpose in future, especially once
configuration validation is introduced.
The configuration YAML file looks as follows:
---
version: 1
configuration:
WHATEVER CONFIGURATION NEEDED
This patch introduces just the basic idea. Actually using the
configuration in the corresponding places (everything what is currently
configurable via environment variables should be configurable in the
file configuration) and other enhancements are implemented in the
followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Top-level `const` qualifiers are not useful, so avoid them. This is done
either by simply removing the top-level `const`, or making the function
return a reference to const where that is appropriate.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Stefan Klug <stefan.klug@ideasonboard.com>
The `ProcessManager` is a singleton class to handle `SIGCHLD` signals
and report the exit status to the particular `Process` instance.
However, having a singleton in a library is not favourable and it is
even less favourable if it installs a signal handler.
Using pidfd it is possible to avoid the need for the signal handler;
and the `Process` objects can watch their pidfd themselves, eliminating
the need for the `ProcessManager` class altogether.
`P_PIDFD` for `waitid()` was introduced in Linux 5.4, so this change
raises the minimum supported kernel version. `clone3()`, `CLONE_PIDFD`,
`pidfd_send_signal()` were all introduced earlier.
Furthermore, the call to the `unshare()` system call can be removed
as those options can be passed to `clone3()` directly.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Instead of creating a new vector, take the vector by value to make it
possible for the caller to use move construction when calling the function.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Instead of using a separate member variable, use `pid_ > 0` to determine
if the process is still running. Previously the value of `pid_` was not
reset to -1 when the process terminated, but since it is only meaningful
while the process is running, reset it to -1 in `Process::died()`.
Neither `pid_` nor `running_` are exposed, so this change has no effect
on the public interface or observable behaviour.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Add a maxQueuedRequestsDevice constructor parameter to allow pipeline
handler classes to limit the maximum number of requests that get queued
to the device in queueRequestDevice().
The default value is set to an arbitrary number of 32 which is big
enough for all currently known use cases.
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Umang Jain <uajain@igalia.com>
Tested-by: Sven Püschel <s.pueschel@pengutronix.de>
The waiting requests of one camera should not be able to influence
queuing to another camera. Currently a request that is in the
waitingQueue and waiting for preparation blocks all requests of other
cameras even if they are already prepared. To fix that replace the
single waitingRequests_ queue with one queue per camera.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Umang Jain <uajain@igalia.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Tested-by: Sven Püschel <s.pueschel@pengutronix.de>
The ClockRecovery class takes pairs of timestamps from two different
clocks, and models the second ("output") clock from the first ("input")
clock.
We can use it, in particular, to get a good wallclock estimate for a
frame's SensorTimestamp.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
A `Process` object has address identity because a pointer to it is
stored inside the `ProcessManager`. However, copy/move special
methods are still generated by the compiler. So disable them to
avoid potential issues and confusion.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
`appendPOD()` does a single insertion, so if only a single `appendPOD()`
will be called on a vector before returning, then calling `reserve()`
is not that useful, so remove it.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
It is useful to multiply matrices and vectors of heterogeneous types, for
instance float and double. Extend the multiplication operator to support
this, avoiding the need to convert one of the operations. The type of the
returned vector is selected automatically to avoid loosing precision.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
It is useful to multiply matrices of heterogneous types, for instance
float and double. Extend the multiplication operator to support this,
avoiding the need to convert one of the matrices. The type of the
returned matrix is selected automatically to avoid loosing precision.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
By zero-initializing the data_ member we can make most functions
constexpr which will come in handy in upcoming patches. Note that this
is due to C++17. In C++20 we will be able to leave data_ uninitialized
for constexpr. The Matrix(std::array) version of the constructor can
not be constexpr because std::copy only became constexpr in C++20.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
The `signature()` getter can just return a reference to the private vector
member variable, and let the caller make a copy if needed. Since the
return type is const qualified, this was likely the original intention.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Paul Elder <paul.elder@ideasonboard.com>
Extend the Simple IPA IPC to support returning a metadata ControlList
when the process call has completed.
A new signal from the IPA is introduced to report the metadata,
similarly to what the hardware pipelines do.
Merge the metadata reported by the ISP into any completing request to
provide to the application. Completion of a request is delayed until
this is done; this doesn't apply to canceled requests.
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
This patch applies color correction matrix (CCM) in debayering if the
CCM is specified. Not using CCM must still be supported for performance
reasons.
The CCM is applied as follows:
[r1 g1 b1] [r]
[r2 g2 b2] * [g]
[r3 g3 b3] [b]
The CCM matrix (the left side of the multiplication) is constant during
single frame processing, while the input pixel (the right side) changes.
Because each of the color channels is only 8-bit in software ISP, we can
make 9 lookup tables with 256 input values for multiplications of each
of the r_i, g_i, b_i values. This way we don't have to multiply each
pixel, we can use table lookups and additions instead. Gamma (which is
non-linear and thus cannot be a part of the 9 lookup tables values) is
applied on the final values rounded to integers using another lookup
table.
Because the changing part is the pixel value with three color elements,
only three dynamic table lookups are needed. We use three lookup tables
to represent the multiplied matrix values, each of the tables
corresponding to the given matrix column and pixel color.
We use int16_t to store the precomputed multiplications. This seems to
be noticeably (>10%) faster than `float' for the price of slightly less
accuracy and it covers the range of values that sane CCMs produce. The
selection and structure of data is performance critical, for example
using bytes would add significant (>10%) speedup but would be too short
to cover the value range.
The color lookup tables can be represented either as unions,
accommodating tables for both the CCM and non-CCM cases, or as separate
tables for each of the cases, leaving the tables for the other case
unused. The latter is selected as a matter of preference.
The tables are copied (as before), which is not elegant but also not a
big problem. There are patches posted that use shared buffers for
parameters passing in software ISP (see software ISP TODO #5) and they
can be adjusted for the new parameter format.
Color gains from white balance are supposed not to be a part of the
specified CCM. They are applied on it using matrix multiplication,
which is simple and in correspondence with future additions in the form
of matrix multiplication, like saturation adjustment.
With this patch, the reported per-frame slowdown when applying CCM is
about 45% on Debix Model A and about 75% on TI AM69 SK.
Using std::clamp in debayering adds some performance penalty (a few
percent). The clamping is necessary to eliminate out of range values
possibly produced by the CCM. If it could be avoided by adjusting the
precomputed tables some way then performance could be improved a bit.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Applying color correction matrix (CCM) in software ISP is optional due
to performance reasons. CCM is applied if and only if `Ccm' algorithm
is present in the tuning file.
Software ISP debayering is a performance critical piece of code and we
do not want to use dynamic conditionals there. Therefore we pass
information about CCM application to debayering configuration and let it
select the right versions of debayering functions using templates. This
is a trick similar to the previously used one for adding or not adding
an alpha channel to the output.
Debayering gets this information but it ignores it in this patch.
Actual processing with CCM is added in the followup patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The `V4L2BufferCache` type is not thread-safe. Its `lastUsedCounter_`
member is not used in contexts where its atomicity would matter.
So it does not need to be have an atomic type.
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
`MediaEntity::ancillaryEntities()` can just return a const lvalue
reference to the underlying array, a copy need not be made. That
was likely the original intention.
Fixes: 9490c664b5 ("libcamera: Add members to MediaEntity to support ancillary entities")
Signed-off-by: Barnabás Pőcze <barnabas.pocze@ideasonboard.com>
Reviewed-by: Daniel Scally <dan.scally@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
There may be pending messages in SoftwareIsp message queue when
SoftwareIsp stops. The call to IPAProxySoft::stop() will dispatch them
before SoftwareIsp::stop() finishes. But this is dependent on
IPAProxySoft::stop() implementation, let's break this dependency and
dispatch messages to SoftwareIsp explicitly in SoftwareIsp::stop().
This also allows dropping `running_' flag. Since the SoftwareIsp
messages get processed and invoke IPA calls before the IPA proxy is set
to ProxyStopping state and the SoftwareIsp worker thread is no longer
running, it's guaranteed that no new messages come to SoftwareIsp and
attempt to call the stopped IPA proxy.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When SoftwareIsp stops, input and output buffers queued to it may not
yet be fully processed. They will be eventually returned but stop means
stop, there should be no processing related actions invoked afterwards.
Let's stop forwarding processed input buffers from SoftwareIsp slots
when SoftwareIsp is stopped. Let's track the queued input buffers and
return them back for capture in SoftwareIsp::stop().
The returned input buffers are marked as cancelled. This is not
necessary at the moment but it gives the pipeline handlers chance to
deal with this if they need to.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
When SoftwareIsp stops, input and output buffers queued to it may not
yet be fully processed. They will be eventually returned but stop means
stop, there should be no processing related actions invoked afterwards.
Let's stop forwarding processed output buffers from the SoftwareIsp
slots once SoftwareIsp is stopped. Let's track the queued output
buffers and mark those still pending as cancelled in SoftwareIsp::stop
and return them to the pipeline handler.
Dealing with input buffers is addressed in a separate patch.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Software ISP runs debayering in a separate thread and debayering may
emit statsReady when software ISP (including the IPA) is being stopped.
The signal waits in a queue and gets invoked later, resulting in an
assertion error when attempting to invoke a method on the stopped IPA:
FATAL default soft_ipa_proxy.cpp:456 assertion
"state_ == ProxyRunning" failed in processStatsThread()
Let's prevent this problem by forwarding the ISP stats signal from
software ISP only when the IPA is running. To track this,
SoftwareISP::running_ variable is introduced.
Making processing of the other signals in SoftwareISP more robust is
addressed in the followup patches.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reported-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Prepare the move of the Vector class from libipa to libcamera by copying
the relevant files into the corresponding libcamera directories. The
files are copied without modification.
Signed-off-by: Stefan Klug <stefan.klug@ideasonboard.com>
Acked-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Add a data() function to the Matrix class to access the internal data.
This is useful for code that needs to use the matrix contents as a
linear array, as shown by the RkISP1::Ccm::process() function that needs
to copy the matrix data to a local variable. Simplify that function by
using the new accessor.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Reviewed-by: Milan Zamazal <mzamazal@redhat.com>
inputBufferReady ready signal in the simple pipeline is handled in the
pipeline handler thread. outputBufferReady and ispStatsReady signals
should be handled there too.
Rather than relying on the user of the SoftwareIsp instance, let
SoftwareIsp inherits Object. SoftwareIsp serves as a signal proxy, the
signals above are emitted from signal handlers. This means that if
SoftwareIsp inherits Object then the slots are invoked in SoftwareIsp
thread. Which is the camera manager thread because the SoftwareIsp
instance is created there.
Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
The PipelineHandler::configurationFile() function prints an error
message when no configuration file is found. It can be useful for
pipeline handlers to silence the lookup operation and handle errors
themselves. Add a silent parameter to the function to enable this.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Tested-by: Julien Vuillaumier <julien.vuillaumier@nxp.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Without the change the build fails on upcoming `gcc-15` as:
In file included from ../src/libcamera/dma_buf_allocator.cpp:9:
../include/libcamera/internal/dma_buf_allocator.h:66:19: error: 'uint64_t' has not been declared
66 | void sync(uint64_t step);
| ^~~~~~~~
Signed-off-by: Sergei Trofimovich <slyich@gmail.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
