Commit Graph

441 Commits

Author SHA1 Message Date
Peter Hatina
f32290d425 Introduce KAS_BUILD_DIR environment variable
This variable can override default build path `${KAS_WORK_DIR}/build`.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-10 07:53:09 +01:00
Peter Hatina
24c13a491b Apply patches before doing an environment setup
We may need to patch the environment setup, too.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-01 13:58:49 +01:00
Henning Schild
6b52357dc7 kas-container: add an argument to get version information
This script is deployed in many copies, give people a way to identify
which one they have cached/installed.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: simplify and use basename for the program]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-25 21:19:02 +01:00
Jan Kiszka
8fe75298e4 Release 2.4 2021-02-25 19:09:02 +01:00
Henning Schild
8234936902 Dockerfile: do not allow pip to fetch any deps anymore
We want all our deps from debian to make sure we do not get random
versions from pip. So we call pip install in such a way, and test that
kas can include all its deps later.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 20:54:06 +01:00
Jan Kiszka
5253d3c1d2 Dockerfile: Pull all Python dependencies from Debian
This provides a clearer and more stable environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 13:14:29 +01:00
Jan Kiszka
5981664b25 setup: Set upper version limit for dependencies
Let's avoid that any of our dependencies decides to break interfaces
that we use and that users run into surprises. As a safe bet, limit us
to the currently know-to-work major versions.

Closes #34
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-24 09:01:08 +01:00
Jan Kiszka
6c5e77800c kas-container: Simplify docker arguments
--privileged implies all caps, so no need to list some explicitly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-02-16 10:36:14 +01:00
Jan Kiszka
6f8d5d6c78 checkcode: Address shellcheck warnings
...and stop ignoring them all.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 21:00:08 +01:00
Jan Kiszka
6fcdf85d9a release.sh: Address shellcheck remarks
No real issues, but let's silence them to catch future ones. One change
is a nice simplification in fact.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:59:55 +01:00
Henning Schild
111906f804 kas-container: make sure that we pass shellcheck
There is one actual change where code gets touched, the rest are just
comments to satisfy shellcheck.
All the lines with the ignores should be reviewed later, there might be
problems behind the findings.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: add one more SC2086 suppression]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:42:49 +01:00
Jan Kiszka
4042ef4859 kas-container: Add support for multi-word --command arguments
Yet another special dance needed to preserve argument boundaries while
moving them around: This trick enables

kas-container shell kas.yml --command "echo it works."

by keeping the command argument separate and injecting it via
appropriate quoting into the argument array.

Closes: #42
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-29 20:29:56 +01:00
Maximilian Schmidt
15aaf4677b Add support for rootless podman with userns keep-id
Running `./kas-container` as rootless podman container fails with:

```
groupadd: Permission denied.
groupadd: cannot lock /etc/group; try again later.
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
```

This patch add a check whether the correct USER_ID is already in use.

Co-authored-by: Nicolas Riebesel <nicolas.riebesel@gmx.com>
Signed-off-by: Maximilian Schmidt <maximilian@schmidt.so>
[Jan: fixed indention]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-20 08:16:36 +01:00
Henning Schild
c50b3d21d8 kas: drop bitbakes "-k" from the default args
That gets us in line with the default of bitbake itself. If users still
want to pass it, they will have to do so from now on.

While that continue switch can seem like a sane default for interactive
builds, it can also be pretty wasteful in CI. Where it would not just
eat compute ressources but also peoples time, because it potentially
delays build failures significantly.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-19 18:14:59 +01:00
Jan Kiszka
e8b6a4d89b Add support for multi-arch containers
Create the kas and kas-isar containers also for arm64. This allows using
them for arm-native builds, e.g. in AWS instances.

We need to drop a couple of packages from arm64 images which do not
exist on that arch. If the kas image is usable for Yocto builds is
untested, though. The Isar image works.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:39:35 +01:00
Jan Kiszka
c92d845639 ci: Push directly from build step
Refactor the workflows so that we can use build-push-action also for
publishing. This is a precondition for switching to buildx, needed for
multi-arch images.

The trick needed for this is using an environment variable to control
whether we should push when building the master branch, initializing it
according to the tag status.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:39:12 +01:00
Jan Kiszka
2f60b982a7 Dockerfile.isar: Drop grub package
The leakage of grub artifacts from the building Debian version and the
usage its grub tools was fixed in Isar 0.6 already. No one is expected
to use that old version with latest kas containers, and the presence of
these packages also collide with providing non-x86 versions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-03 18:36:38 +01:00
Jan Kiszka
2f4b89c77f Dockerfile: Fold two related ENV settings together
One layer less.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-02 15:53:06 +01:00
Jan Kiszka
cc159c471f Dockerfile: Make DEBIAN_FRONTEND non-persistent
Only use this setting during non-interactive build, not in the possibly
interactive image.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-02 15:50:17 +01:00
Jan Kiszka
a18c414b81 CONTRIBUTING: Drop reference to non-existing maintainer list
This project does not have this.

Reported-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-17 18:13:19 +01:00
Silvano Cirujano Cuesta
0892b37ad6 repos: refactor repo path calculation
Refactored repo path calculation eliminates duplicated code and aligns
the matching logic (with respect to url and path) with the
documentation.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:12 +01:00
Silvano Cirujano Cuesta
1f1a21992d repos: fix path if no url, but path given
The `repo` documentation says:
"If the url is missing and the path defined, this entry references the
directory the path points to. [...] In case of a relative path name
`kas_work_dir` is prepended."

But if no `url` is given and a relative path is given in `path`, then
the layer paths being added to `bblayer.conf` are missing the
`kas_work_dir` prefix.

This patch fixes it.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:04 +01:00
Jan Kiszka
87af46ff8c Silence "Exception ignored when trying to write to the signal wakeup fd"
Disable warn_on_full_buffer via set_wakeup_fd. This avoids tons of

    Exception ignored when trying to write to the signal wakeup fd:
    BlockingIOError: [Errno 11] Resource temporarily unavailable

that are currently issued after running complex builds. Only works since
Python 3.7, though, so this remains best-effort. Fortunately, the Debian
in our container image is 3.7.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-02 12:40:19 +01:00
Jan Kiszka
368cdffc24 Release 2.3.3 2020-11-28 12:36:46 +01:00
Quirin Gylstorff
e905f0002d Dockerfile.isar: Escape special characters
Escape '&' '\&' as sed replace & with the search term.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-23 16:45:07 +01:00
Jan Kiszka
ba439d4200 Release 2.3.2 2020-11-20 12:20:24 +01:00
Jan Kiszka
3d0c9b8299 Add release artifact to .gitignore
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-20 12:05:51 +01:00
Jan Kiszka
7f89747645 scripts/release: Fix KAS_IMAGE_VERSION_DEFAULT update
Remove the leading tab, it's no longer there.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-20 12:03:25 +01:00
Jan Kiszka
f50316e1c9 Release 2.3.1 2020-11-20 11:16:18 +01:00
Jan Kiszka
94f20cc9ad ci: Switch to environment files
Fixes:

"Unable to process command '::set-env name=RELEASE_VERSION::2.3'
successfully.
The `set-env` command is disabled. Please upgrade to using Environment
Files [...]."

To make the next release smoother, hopefully.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-20 11:03:59 +01:00
Jan Kiszka
816dffa898 scripts/release: Update announcement footer
We have the packages on github now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-20 08:39:52 +01:00
Jan Kiszka
7bbccbfb47 Release 2.3 2020-11-20 08:19:32 +01:00
Jan Kiszka
6110863a0c userguide: Fix typo
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-18 19:42:10 +01:00
Jan Kiszka
49e90b2728 CONTRIBUTING: Add newline to fix rendering
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-18 15:15:45 +01:00
Jan Kiszka
b36b2dbedd Rename forallrepos plugin to for_all_repos
Comes closer to how the command is spelled. We can't uses dashes as that
is not compatible with the python syntax.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 18:26:12 +01:00
Paul Barker
9176d1f5b9 tests: Add tests for new for-all-repos & checkout commands
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:57:59 +01:00
Paul Barker
a89b011b80 for-all-repos: Extend environment to include repo config
The environment for executing the command in each repository is extended
to include the variables KAS_REPO_NAME, KAS_REPO_PATH, KAS_REPO_URL and
KAS_REPO_REFSPEC. This allows more complex actions to be automated via
this command.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:57:46 +01:00
Paul Barker
d42df24afe repos: Keep repo name as read from config file
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:52:00 +01:00
Jan Kiszka
d5eb961a44 tests: Stress the build_system property
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:29 +01:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Jan Kiszka
f502f92afe Dockerfile.isar: Improve binfmt_misc setup
The init script is a bit noisy, so replace it with a direct call of
update-binfmts. That tool unfortunately fails to report missing kernel
support via a proper error code. Catch that by doing an extra check on
the mounted filesystem.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-15 10:03:56 +01:00
Paul Barker
a8b5c79326 checkout: New plugin
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:57 +01:00
Paul Barker
baacd3a8b9 for-all-repos: New plugin
This plugin runs a specified command in all checked out repositories.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:51 +01:00
Paul Barker
1253d22dee tests: Extract common changedir fixture
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:35 +01:00
Paul Barker
bcb748c22a kas: Factor out common setup & cleanup steps
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.

By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:56:52 +01:00
Paul Barker
451a51f94a shell: Handle --keep-config-unchanged with existing skip mechanism
This makes the body of the Shell.run function closely match Build.run so
that we can factor out common code.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:58 +01:00
Paul Barker
c408e89ef1 kas: Eliminate Context.keep_config flag
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:35 +01:00
Paul Barker
e6d1d1718d flake8: Ignore obsolete warning W503
"Line break occurred before a binary operator (W503)" goes against PEP8
and has been superseded by W504.

See https://www.flake8rules.com/rules/W503.html.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:20 +01:00
Jan Kiszka
4744c8946b Declare Python 3.9 compatibility
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:56:31 +01:00
Jan Kiszka
85e3b1e74f Account for removal of aynchio.Tasks.all_tasks
As we still support 3.5..3.7, we need to catch the case that
asyncio.get_running_loop and asyncio.all_tasks are not yet available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:53:33 +01:00