Commit Graph

599 Commits

Author SHA1 Message Date
Jan Kiszka
06aae60b65 container: Disable git safe.directory when running without kas-container
This is for the sake of the gitlab-ci runner which does not properly
aligns the ownership of the repo it checks out with the UID:GID of
our builder user. Reason not yet understood and hard to debug (logging
of the runner is incomplete).

Work around this issue by disabling safe.directory checks in case the
container is called without kas-container as wrapper (means, when it is
called without setting "--user=root"). This preserves git's checks for
the common interactive case, the more critical one.

Reported-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-07 10:38:39 +01:00
Jan Kiszka
492b2c56ab container: Rework uid/gid alignment with caller
Already create the builder user/group during container image build and
only align the IDs in the entrypoint if started with a non-zero USER_ID.
The primary gain is code simplification because this removes some
dynamics from the entrypoint.

As this refactoring avoids that gitlab-ci runners start the container as
root, it was also supposed to resolve the mismatch between the owner of
the checked-out repo and builder user. Unfortunately, this does not work
yet, and the reason is still unclear.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-07 10:38:27 +01:00
Jan Kiszka
a596e2ac33 kas-isar: Rework isar-specific entrypoint setup
Leave the isar-only commands commented-out in the container-entrypoint
and simply remove the comment when building kas-isar. This is simpler,
more readable and also more robust against changes of the entrypoint
file.

While at it, avoid a separate layer for modifying container-entrypoint.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-06 10:30:04 +01:00
Jan Kiszka
6b1df5f425 tests: Avoid useless directory creations
The pattern

mkdir test_commands
rm -rf test_commands

makes no sense. If there were test_commands before, mkdir would have
failed. Simply build the path and copy the source content in.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-05 21:18:35 +01:00
Jan Kiszka
039a320631 Add a SECURITY.md
This file shall describe the vulnerability disclosure process for kas
and the security context in which kas should be seen. Reporting
vulnerabilities via github has already been activated.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-04 14:03:00 +01:00
Jan Kiszka
66893480c7 Release 3.2.1 2023-02-02 08:41:17 +01:00
Jan Kiszka
795ecdbcd6 doc: Fix description of bblayers_conf_header and local_conf_header
For a long time already (c9326cc1ed), the documentation suggests a
different ordering of entries generated by *_conf_header than we
actually create. Finally fix this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-01 08:57:07 +01:00
Hannah Kiekens
b27ab659f3 Dockerfile: Add unzip package to kas-base
Was lost in the refactoring of a6b18abc8a.

Signed-off-by: Hannah Kiekens <hannah.kiekens@mind.be>
[Jan: refer to the causing commit]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-24 07:11:12 +01:00
Jan Kiszka
34cf4c9c8f docs: Fix description of container image generation
Since a6b18abc8a, we only have a single Dockerfile and a --target
instead.

We actually also need to touch the unmodified proxy line, wrapping it
around, so that doc8 remains happy.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-17 18:39:36 +01:00
Jan Kiszka
5b15a4d52a scripts: Add an update reminder for readthedocs
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-17 18:27:08 +01:00
Jan Kiszka
dc44638cd8 Release 3.2 2023-01-17 12:49:44 +01:00
Jan Kiszka
e9fac8dda7 ci: Update docker actions
Silences lots of deprecation warnings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-13 18:11:00 +01:00
Frieder Schrempf
f3521f82fa kas-container: Fix engine detection when docker is an alias for podman
The current detection mechanism assumes that if the docker command
is available the engine behind is also docker. In fact nowadays a
lot of people use /usr/bin/docker as an alias for podman. In that
case the script expects a docker engine and misses to use
podman-specific settings.

To fix this, run "docker -v" and check if the output indicates a
podman engine or a real docker engine. Otherwise error out as the
enginge is not supported.

Signed-off-by: Frieder Schrempf <frieder.schrempf@kontron.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-09 13:09:14 +01:00
Jan Kiszka
f5655dcb0c ci: Update to actions/checkout@v3 and actions/setup-python@v4
Silences some NodeJS 12 action deprecation warnings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 20:22:22 +01:00
Felix Moessbauer
1c2c859a4a add test for KAS_REPO_REF_DIR logic
This patch adds a test for cloning with KAS_REPO_REF_DIR.
It explicitly tests the case that two repos with a single upstream URI
are added and also tests mercurial.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: ensure Python 3.5 compatibility]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 20:03:25 +01:00
Felix Moessbauer
3289ec2a7e re-create tmpdir after successfull rename
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:12:04 +01:00
Felix Moessbauer
f2560588bc auto create repo refs when setting KAS_REPO_REF_DIR
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.

Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.

The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.

The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:11:06 +01:00
Felix Moessbauer
e7896c33d9 do not short-circuit on initial clone of repos
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:10:51 +01:00
Jan Kiszka
c816b483fa ci: Adjust images to cover all Python versions
Only Ubuntu 20.04 is still supported with 3.5 and 3.6, so pin to it.
Expand the test to 3.10 and 3.11 at this chance. Quoting is needed now
to avoid that 3.10 becomes 3.1.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:59:27 +01:00
Felix Moessbauer
88a2b3d47b add test for resolve-env option in dump plugin
This patch adds a test for the --resolve-env option in the dump plugin.
Both cases resolve and non-resolve are tested, each for yaml and json
output format.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:41:00 +01:00
Felix Moessbauer
20a69016da dump plugin: add option to capture current env
This patch adds the option --resolve-env to the dump plugin.
When enabled, all variables in the 'env' section of the kas config file
are set to the captured value (at time of executing the dump plugin).

This helps to debug build issues on CI runners by precisely capturing
the relevant environment.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:40:04 +01:00
Felix Moessbauer
33250866ec test dump plugin support for resolved refs
This patch extends the test of the dump plugin.
In addition, we test if relative refspecs are expanded and if the
generated output can be used as input to kas again.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:51 +01:00
Felix Moessbauer
ffabaa19f3 extend dump plugin to support resolving revisions
This patch adds a flag --resolve-refs to the dump plugin.
Once enabled, all relative refspecs (e.g. branch or tag names) are
resolved and replaced by their exact revision (before patches are
applied).
When re-running kas on the flattened and expanded config file, the build
is executed against exactly the versions of the dependencies that where
used when dumping the configuration. This helps to keep track of build
versions for future use (e.g. reproducible builds) or for
version-tracking tools like renovate.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:43 +01:00
Felix Moessbauer
23dcf955a2 add revision attribute to repo class
This patch adds a programmatic attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).

Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:25 +01:00
Felix Moessbauer
53369faf55 add support for the dump plugin in kas-container
This patch adds support to use the dump plugin in kas-container.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:14 +01:00
Felix Moessbauer
bfd6bd6195 add test of dump plugin
This patch adds a basic test of the dump plugin. We check if the configuration
is flattened and external references are included.
In addition, we check if no other files are referenced anymore.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:03 +01:00
Felix Moessbauer
5b3e238128 add plugin to dump flattened config
This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.

While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.

The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:38:41 +01:00
Felix Moessbauer
9bb1af5ecd kas-container: write debug messages to stderr
The kas-container trace / debug infrastructure has to write to stderr to
not interfer with kas output to stdout. Note, that the kas debug
messages already write to stderr. By that, the kas-container script can
be used in pipes similar to a direct invocation of kas.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-28 08:35:11 +01:00
Felix Moessbauer
1a2de8daa7 forward DISTRO_APT_PREMIRRORS environment variable
This patch forwards the DISTRO_APT_PREMIRRORS environment variable into
the container when using the kas-container script. This variable is
commonly used to externally set mirrors for debian apt, e.g. in a CI
environment.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-15 11:54:34 +01:00
Benedikt Niedermayr
7215e58eab libkas: Fix typo
Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-11 13:36:12 +01:00
Benedikt Niedermayr
4795c21fb6 tests: Add test case for environment variables
These additional test cases check if the env section
can:

- export variables with their default value
- add variables with 'None' assigned only to
  BB_ENV_PASSTHROUGH_ADDITIONS or to the deprecated
  BB_ENV_EXTRAWHITE
- bitbake: make sure that variables actually make it into bitbake

The BB_ENV_EXTRAWHITE variable is still present, so a test case has also
been added for this.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: ensure python 3.5 compatibility, adjust copyright header]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-11 13:35:45 +01:00
Benedikt Niedermayr
c38517df83 add conditional environment variables
Until now all environment variables in the kas config's
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-10 19:03:48 +01:00
Felix Moessbauer
102c4667d1 forward build-concurrency related env vars
This patch forwards environment variables to control the concurrency and
parallelism of the build to bitbake.
With that, the user can reduce the concurrency in case of running short
on memory. Further, by reducing the concurrency on very large and NUMA
machines, the build sometimes can be significantly speed up.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-10-19 11:02:17 +02:00
Jan Kiszka
cf95cfc57b kas-container: Do not advertise deprecated --docker-args anymore
We still accept it, but we should stop talking about it.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-10-19 07:28:33 +02:00
Jose Quaresma
c0d00aa637 Dockerfile: avoid deploy the python pip cache
This patch prevents deployment of the pip cache folder that is only
needed during installation. Which is also done with the apt,
where at the end of the install operation we delete the
cache and all temporary folders.

The reducion of the image size is insignificant:
| builder@1373d0b95f44:/build$ sudo du -sh /root/.cache/pip
| 1.6M    /root/.cache/pip

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-10-14 11:51:48 +02:00
Jose Quaresma
044de41a9a Dockerfile: adds a comment about the installation on the downstream layers
The install package list are actually taking 1:1 from their documentation,
so there some packages that can already installed by other downstream layers.
This will not change any image sizes on all the layers in use.

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-10-14 11:51:22 +02:00
Wadim Klincov
4f51a41ce7 Use correct variable on error output
Use correct variable on error output.

Signed-off-by: Wadim Klincov <wadim@klincov.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-09-30 14:04:33 +02:00
Claudius Heine
26a7ab8f48 kas-container: add --log-driver=none as default runtime arg
Container engines per default log all container output into journald
(Podman) or a json file (Docker).

A build with kas-container/bitbake containes lots of useless build
output like progress lines, etc.

In case of Podman, where journald is used, this means that all this is
spammed into the journald buffer and possible also send to ttys or even
serial ttys. This might make sense for containers that output only
status information about running services, but for a build process, this
is mostly just spamming and causes important information to be lost.

Since the stdout/stderr output of the container is used for
kas-container ordinarily, the background logging from the container
image can be deactivated.

Therefor add the `--log-driver=none` parameter to all container
runtimes command lines.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-09-19 15:56:23 +02:00
Jan Kiszka
10aa308334 Dockerfile: Test kas installation by printing its version
Same effect but less verbose and potentially confusing output.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 15:16:50 +02:00
Jan Kiszka
edfc30751d image-tests: Update to recent Yocto and Isar revisions
This moves poky to 4.0.3 and Isar to the next revision of the day. The
latter is needed as we have no release with sbuild support yet.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 15:12:20 +02:00
Jan Kiszka
18532df94d .dockerignore: Update
Travis-ci is long gone, now we have github actions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:39:41 +02:00
Jan Kiszka
a6b18abc8a Dockerfile: Rework generation of kas images
While kas and kas-isar surely have a common ground like the kas tool
itself, kas-isar does not need all the toolchain packages that are
essential Yocto dependencies. Splitting up the images earlier allows to
shrink kas-isar by almost 400 MB.

Use this chance to model both images as different build stages of the
same Dockerfile. That is simpler than creating a temporary "kas-base"
image via a separate Dockerfile.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:34:37 +02:00
Jan Kiszka
46b1c199a3 Dockerfile: Drop bc as Yocto dependency
Prior to the official first release of kas, this was once introduced to
fulfill kernel build dependencies. That is not longer be needed with
modern Yocto versions (if it ever actually was).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:32:40 +02:00
Jan Kiszka
3863d46f17 Dockerfile: Refresh Yocto build dependency list
Use the list from Kirkstone. This should cover the needs back to Dunfell
at least as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 13:47:14 +02:00
Jan Kiszka
d73f6f4694 Dockerfile.isar: Clean up dependency installation
autoconf, automake, libtool were never official dependencies. Their
addition comes from the early Isar days, likely misunderstanding the
actual needs already back then.

e2fsprogs and multistrap were only needed up to Isar v0.5 from 2018.
It's fair to drop support for these historic versions now.

debhelper was added along sbuild support in 54ab356f9f but never
became an official upstream dependency - drop it again.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 11:15:44 +02:00
Michael Adler
271f0bf6ca feat(shell): print bitbake command
This is useful for people who start an interactive 'kas shell' and would
like to know the exact build command which 'kas build' would use.

Signed-off-by: Michael Adler <michael.adler@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-10 09:41:53 +02:00
Jan Kiszka
de7acb5f0d scripts: Switch release script to twine
The old procedure finally broke over a new distro version of the
maintainer. Migrate to twine for uploading, that seems to work fine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-05 13:39:35 +02:00
Jan Kiszka
d1834dd6bf Release 3.1 2022-08-05 13:17:58 +02:00
Jan Kiszka
9afb1d8239 tests: Fix style warnings around assert usage
assert is not a function. Latest pycodestyle and flake8 warn about this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-04 19:09:12 +02:00
Felix Moessbauer
fa8414b660 remove obsolete schroot mntpoint in kas-container
This patch is an addendum to 54ab356f and removes
the external mount used for the schroot.
This is no longer required as the schroot is now
placed in TMPDIR which is already mounted from the host.

This patch does not break backward compatibility, as neither
an ISAR release nor a KAS release happened while this
code path was active.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:01:20 +02:00