Commit Graph

473 Commits

Author SHA1 Message Date
Jan Kiszka
6a97879ad0 Dockerfile: Carry oe-git-proxy locally
This imports revision aa9b9dc9a9 from
https://git.yoctoproject.org/git/poky to avoid fetching it - and having
to add the missing content validation to prevent supply-chain attacks.

Reported-by: Raphael Lisicki <raphael.lisicki@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-30 21:56:04 +02:00
Jasper Orschulko
9732ec16f2 for_all_repos: Exit on command failure
KAS should exit with the appropriate exit code, if the command parsed to the for_all_repos
plugin fails to apply on one of the repositories.

Example use-case: This allows for CI pipelines or wrapper scripts to take appropriate action (e.g. fail),
if an issue occured within the KAS for_all_repos command.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-26 18:56:38 +02:00
Jasper Orschulko
fca78e5601 for_all_repos: Fix test case
An upcoming change will introduce returning non-zero status on the KAS
command if the for_all_repos command fails for one of the repos defined
in the kas configuration. This caused an issue with the non-git "this"
repo defined in the test.yml.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: adjusted commit log to reflect different commit ordering]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-26 18:55:52 +02:00
Jan Kiszka
6eda3ba48a kas-container: Simplify translation of file argument list
Since dabda7617f, all elements of KAS_FILES are already absolute.
Therefore, a single replacement rule is sufficient to translate them to
paths for the container.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-24 19:19:36 +02:00
Jan Kiszka
c7bd8393aa tests: Add include test case for multiple files in sub-dirs
This stresses the previously broken case of the incorrectly detected
repository root while including files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:26 +02:00
Jan Kiszka
2077900b4e includehandler: Fix repo path detection for local includes
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.

Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.

As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:14 +02:00
Jan Kiszka
8a1c27bc92 libkas: Gracefully handle empty repo set
We will fail differently then, but with a nicer error message (missing
init-build-env script) than

ValueError: Set of coroutines/Futures is empty.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 23:17:41 +02:00
Jan Kiszka
4c81f96742 context: Add support for relative KAS_WORK/BUILD/REPO_REF_DIR paths
No one asked for this so far but the result of providing a relative
path, e.g. to KAS_WORK_DIR, is so confusing that it is better to simply
resolve all those vars to absolute paths.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 12:43:09 +02:00
Ross Burton
ff99bf2226 Dockerfile: install lz4
OpenEmbedded Core is about to depend on lz4c, pzstd, and zstd being on
the host.  The Dockerfile was already installing zstd which provides
zstd/pzstd, so also install lz4 for lz4c.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 12:41:21 +02:00
Steffen Hieber
41d9f68ec3 Dockerfile: install git-lfs from buster-backports.
Debian Buster comes with version 2.7.1 of git-lfs which does not support
git+ssh yet, therefore we need a newer version of git-lfs for cloning a
repository which uses LFS.

Signed-off-by: Steffen Hieber <steffen.hieber@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-06 12:56:10 +02:00
Jan Kiszka
bb134014f2 Dockerfile: Replace unneeded backports pinning with explicit release selection
Backport repos have a priority of 100 by default, thus can just be added
with extra pinning. Use explicit release selection on the installation
of qemu-user-static instead, that's simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-06 11:54:43 +02:00
Jan Kiszka
ed19ad0059 kas-container: Fix indention
No functional changes.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Drew Reed
6b65de3e2f Enable gerrit/gitlab/github refspecs
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree.  This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes.  When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.

Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Jan Kiszka
635b1023a1 Declare proxy_config obsolete
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.

Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-12 08:03:40 +02:00
Jasper Orschulko
15de0142d4 Add environment variable SSH_PRIVATE_KEY_FILE
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-10 11:25:40 +02:00
Jan Kiszka
4d2b16e04d includehandler: Avoid duplicate cloning of repos in command line includes
If command line includes refer to the same repo, missing_repos was
aggregating those multiple times, causing multiple clone procedures to
be triggered. This generally failed. Therefore, build missing_repos
while checking for duplicates.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-03 18:08:12 +02:00
Jan Kiszka
8b3ff93bae kas-container: Update and improve help output
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-06-29 13:45:40 +02:00
Jan Kiszka
28891711c0 Release 2.5 2021-05-26 07:27:23 +02:00
Jan Kiszka
46eabc7be3 kas-container: Add support for checkout and for-all-repos
Allow invoking those two new sub-commands also via the container
interface.

Closes #51
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-20 16:39:39 +02:00
Rotem Bar
0e470dc644 kas-container: Fix mounting of custom KAS_REPO_REF_DIR
kas-container support for KAS_REPO_REF_DIR was broken. The path provided
by the env variable was the path on the local machine, not in the
container.

Signed-off-by: Rotem Bar <rotemb@hailo.ai>
[Jan: massaged commit log]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-10 07:30:17 +02:00
Claudius Heine
7c6e3c3994 kas-container: add --git-credential-store options
This options allows to specify a git credential store file, which is
then mounted into the container and used by kas as a
git-credential-helper.

Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: remove debug echo]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-06 14:09:10 +02:00
Claudius Heine
71e7f5523d kas: add GIT_CREDENTIAL_HELPER environment variable
This allows to set a git credential helper in the .gitconfig, in order
to handle git authentications. For instance when bitbake recipes require
to clone from git via https.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-06 14:08:40 +02:00
Konrad Weihmann
b250c341fb repos: strip dot from layer name
when importing nested layer in combination with the top level layer like

meta-security
meta-security/meta-tpm

one can use

layers:
    .:
    meta-tpm:

in the spec file to import the parent layer.
This leads to the effect that

.
meta-tpm             = "HEAD:<hash>"

is shown in the build console and in /etc/build
when using image-buildinfo.bbclass.
To fix that just strip the trailing dot from each layer name.

Add test to avoid future regressions

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
[Jan: use only a single rstrip call, fix style]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-05 15:43:58 +02:00
Jon Mason
eb63b1873b Dockerfile: move sudo to standard Dockerfile
To expand the usability of the standard Docker image, move sudo
enablement from Docker.isar.

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-04-27 07:25:59 +02:00
Jose Quaresma
6e82e0d12d kas-container: add support to set a custom container images location
There are situation that the user have a local customized container images
with tools installed that are not provided in the default kas container.

- For a local container the user can tag it and use the existing variables:

 docker tag local-container my/container:1.0
 export KAS_CONTAINER_IMAGE_PATH=my
 export KAS_CONTAINER_IMAGE_NAME=container
 export KAS_IMAGE_VERSION=1.0

- This patch improve the user interface by allowing to set a custom image
 with just a single environment variable.

 export KAS_CONTAINER_IMAGE=local-container

Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-04-16 14:20:58 +02:00
Claudius Heine
06240c23da kas-container: mount /repo as read-write for shell command
There are some bitbake commands like `recipetool` that allows to easily
create new or append to existing recipes. This of course only works if
the main repository that is worked with is writeable.

However it is mounted into the container only as read-only.

This patch mounts the repository writeable when the `shell` command is use
and read-only in case of the `build` command.

It also adds `--repo-ro` and `--repo-rw` to allow overwriting the
default behaviour.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-04-16 08:04:31 +02:00
Silvano Cirujano Cuesta
6d1b65eb7b Add skopeo and umoci to ISAR image
Skopeo and Umoci are needed for the new ISAR SDK Container target.
A "manual" backport of Skopeo (and only Skopeo) is needed, since the
repository "buster-backports" doesn't provide it.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
[Jan: add golang-github-containers-{image,common} to buster list]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-04-06 11:00:32 +02:00
Henning Schild
243c51d24d Use issue template to point people to the list
We often see Issues being used while they maybe should not. Introducing
an issue template might help avoid confusion and inviting contributors
to the list without disabling Issues.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-25 19:43:01 +01:00
Jan Kiszka
3f98520ab6 kas-container: Improve KAS_WORK/BUILD_DIR retrieval
Perform link resolution unconditionally to account for cases when the
default paths are links. Use default values to simplify the assignments.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-21 09:17:30 +01:00
Jan Kiszka
03da3f9436 docs: Improve link to container images
We were linking version 2.2 here.

Reported-by: Vivien Didelot <vivien.didelot@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-19 16:24:25 +01:00
Jan Kiszka
6a792c0a3e docs: Tune layout of env var table
Otherwise the descriptions of KAS_WORK_DIR and KAS_BUILD_DIR are merged
together when rendering on kas.readthedocs.io.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-19 08:01:29 +01:00
Peter Hatina
9d1d2adbd5 docs: Fix typo in excluded layers example
Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-12 11:18:24 +01:00
Peter Hatina
f32290d425 Introduce KAS_BUILD_DIR environment variable
This variable can override default build path `${KAS_WORK_DIR}/build`.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-10 07:53:09 +01:00
Peter Hatina
24c13a491b Apply patches before doing an environment setup
We may need to patch the environment setup, too.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-01 13:58:49 +01:00
Henning Schild
6b52357dc7 kas-container: add an argument to get version information
This script is deployed in many copies, give people a way to identify
which one they have cached/installed.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: simplify and use basename for the program]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-25 21:19:02 +01:00
Jan Kiszka
8fe75298e4 Release 2.4 2021-02-25 19:09:02 +01:00
Henning Schild
8234936902 Dockerfile: do not allow pip to fetch any deps anymore
We want all our deps from debian to make sure we do not get random
versions from pip. So we call pip install in such a way, and test that
kas can include all its deps later.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 20:54:06 +01:00
Jan Kiszka
5253d3c1d2 Dockerfile: Pull all Python dependencies from Debian
This provides a clearer and more stable environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 13:14:29 +01:00
Jan Kiszka
5981664b25 setup: Set upper version limit for dependencies
Let's avoid that any of our dependencies decides to break interfaces
that we use and that users run into surprises. As a safe bet, limit us
to the currently know-to-work major versions.

Closes #34
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 09:01:08 +01:00
Jan Kiszka
6c5e77800c kas-container: Simplify docker arguments
--privileged implies all caps, so no need to list some explicitly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-16 10:36:14 +01:00
Jan Kiszka
6f8d5d6c78 checkcode: Address shellcheck warnings
...and stop ignoring them all.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 21:00:08 +01:00
Jan Kiszka
6fcdf85d9a release.sh: Address shellcheck remarks
No real issues, but let's silence them to catch future ones. One change
is a nice simplification in fact.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:59:55 +01:00
Henning Schild
111906f804 kas-container: make sure that we pass shellcheck
There is one actual change where code gets touched, the rest are just
comments to satisfy shellcheck.
All the lines with the ignores should be reviewed later, there might be
problems behind the findings.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: add one more SC2086 suppression]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:42:49 +01:00
Jan Kiszka
4042ef4859 kas-container: Add support for multi-word --command arguments
Yet another special dance needed to preserve argument boundaries while
moving them around: This trick enables

kas-container shell kas.yml --command "echo it works."

by keeping the command argument separate and injecting it via
appropriate quoting into the argument array.

Closes: #42
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:29:56 +01:00
Maximilian Schmidt
15aaf4677b Add support for rootless podman with userns keep-id
Running `./kas-container` as rootless podman container fails with:

```
groupadd: Permission denied.
groupadd: cannot lock /etc/group; try again later.
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
```

This patch add a check whether the correct USER_ID is already in use.

Co-authored-by: Nicolas Riebesel <nicolas.riebesel@gmx.com>
Signed-off-by: Maximilian Schmidt <maximilian@schmidt.so>
[Jan: fixed indention]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-20 08:16:36 +01:00
Henning Schild
c50b3d21d8 kas: drop bitbakes "-k" from the default args
That gets us in line with the default of bitbake itself. If users still
want to pass it, they will have to do so from now on.

While that continue switch can seem like a sane default for interactive
builds, it can also be pretty wasteful in CI. Where it would not just
eat compute ressources but also peoples time, because it potentially
delays build failures significantly.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-19 18:14:59 +01:00
Jan Kiszka
e8b6a4d89b Add support for multi-arch containers
Create the kas and kas-isar containers also for arm64. This allows using
them for arm-native builds, e.g. in AWS instances.

We need to drop a couple of packages from arm64 images which do not
exist on that arch. If the kas image is usable for Yocto builds is
untested, though. The Isar image works.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:39:35 +01:00
Jan Kiszka
c92d845639 ci: Push directly from build step
Refactor the workflows so that we can use build-push-action also for
publishing. This is a precondition for switching to buildx, needed for
multi-arch images.

The trick needed for this is using an environment variable to control
whether we should push when building the master branch, initializing it
according to the tag status.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:39:12 +01:00
Jan Kiszka
2f60b982a7 Dockerfile.isar: Drop grub package
The leakage of grub artifacts from the building Debian version and the
usage its grub tools was fixed in Isar 0.6 already. No one is expected
to use that old version with latest kas containers, and the presence of
these packages also collide with providing non-x86 versions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:36:38 +01:00
Jan Kiszka
2f4b89c77f Dockerfile: Fold two related ENV settings together
One layer less.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-02 15:53:06 +01:00