Commit Graph

562 Commits

Author SHA1 Message Date
Claudius Heine
26a7ab8f48 kas-container: add --log-driver=none as default runtime arg
Container engines per default log all container output into journald
(Podman) or a json file (Docker).

A build with kas-container/bitbake containes lots of useless build
output like progress lines, etc.

In case of Podman, where journald is used, this means that all this is
spammed into the journald buffer and possible also send to ttys or even
serial ttys. This might make sense for containers that output only
status information about running services, but for a build process, this
is mostly just spamming and causes important information to be lost.

Since the stdout/stderr output of the container is used for
kas-container ordinarily, the background logging from the container
image can be deactivated.

Therefor add the `--log-driver=none` parameter to all container
runtimes command lines.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-09-19 15:56:23 +02:00
Jan Kiszka
10aa308334 Dockerfile: Test kas installation by printing its version
Same effect but less verbose and potentially confusing output.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 15:16:50 +02:00
Jan Kiszka
edfc30751d image-tests: Update to recent Yocto and Isar revisions
This moves poky to 4.0.3 and Isar to the next revision of the day. The
latter is needed as we have no release with sbuild support yet.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 15:12:20 +02:00
Jan Kiszka
18532df94d .dockerignore: Update
Travis-ci is long gone, now we have github actions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:39:41 +02:00
Jan Kiszka
a6b18abc8a Dockerfile: Rework generation of kas images
While kas and kas-isar surely have a common ground like the kas tool
itself, kas-isar does not need all the toolchain packages that are
essential Yocto dependencies. Splitting up the images earlier allows to
shrink kas-isar by almost 400 MB.

Use this chance to model both images as different build stages of the
same Dockerfile. That is simpler than creating a temporary "kas-base"
image via a separate Dockerfile.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:34:37 +02:00
Jan Kiszka
46b1c199a3 Dockerfile: Drop bc as Yocto dependency
Prior to the official first release of kas, this was once introduced to
fulfill kernel build dependencies. That is not longer be needed with
modern Yocto versions (if it ever actually was).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 14:32:40 +02:00
Jan Kiszka
3863d46f17 Dockerfile: Refresh Yocto build dependency list
Use the list from Kirkstone. This should cover the needs back to Dunfell
at least as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 13:47:14 +02:00
Jan Kiszka
d73f6f4694 Dockerfile.isar: Clean up dependency installation
autoconf, automake, libtool were never official dependencies. Their
addition comes from the early Isar days, likely misunderstanding the
actual needs already back then.

e2fsprogs and multistrap were only needed up to Isar v0.5 from 2018.
It's fair to drop support for these historic versions now.

debhelper was added along sbuild support in 54ab356f9f but never
became an official upstream dependency - drop it again.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-26 11:15:44 +02:00
Michael Adler
271f0bf6ca feat(shell): print bitbake command
This is useful for people who start an interactive 'kas shell' and would
like to know the exact build command which 'kas build' would use.

Signed-off-by: Michael Adler <michael.adler@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-10 09:41:53 +02:00
Jan Kiszka
de7acb5f0d scripts: Switch release script to twine
The old procedure finally broke over a new distro version of the
maintainer. Migrate to twine for uploading, that seems to work fine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-05 13:39:35 +02:00
Jan Kiszka
d1834dd6bf Release 3.1 2022-08-05 13:17:58 +02:00
Jan Kiszka
9afb1d8239 tests: Fix style warnings around assert usage
assert is not a function. Latest pycodestyle and flake8 warn about this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-04 19:09:12 +02:00
Felix Moessbauer
fa8414b660 remove obsolete schroot mntpoint in kas-container
This patch is an addendum to 54ab356f and removes
the external mount used for the schroot.
This is no longer required as the schroot is now
placed in TMPDIR which is already mounted from the host.

This patch does not break backward compatibility, as neither
an ISAR release nor a KAS release happened while this
code path was active.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:01:20 +02:00
Johann Neuhauser
bce8a916fa config: Symplify .config.yaml file handling
If no configuration file is specified, we try to load a single
.config.yaml from KAS_WORK_DIR and set the top_repo_path to the
repo root of this file with fallback to the containing directory.

This process is identical to loading explicitly specified
configuration files and can therefore be combined into one case.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:55 +02:00
Johann Neuhauser
cbdba7e8e1 config: Fix repo-relative include file handling if no config file is given
If we do not specify a configuration file for the "kas build" command,
the repository anchor is set to the current working directory,
which breaks the resolution of repo-relative include file paths
in the IncludeHandler class if repo root != current working directory.

Sets the repository anchor to the root path of the repository with
fallback to KAS_WORK_DIR if we use .config.yaml from KAS_WORK_DIR.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:49 +02:00
Jasper Orschulko
a9cc7d06b3 for-all-repos: Add option to keep current env
Extend support for preserving the current environment to the
for-all-repos plugin with the --preserve-env flag.
This eases the usage of dynamic configuration done via environment
variables within the for-all-repos plugin, e.g. when calling a script.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-22 14:28:00 +02:00
Peter Hoyes
682db50916 libcmds: Remove protected accesses to Config
There is one remaining protected access in SetupReposStep: an assignment
to ctx.config._config. Replace this with a call to
ctx.config.find_missing_repos, which already handles this assignment.
Remove the TODO comment.

There is one remaining protected access in FinishSetupRepos: a read of
ctx.config._config for debugging purposes. Replace this with a call to
ctx.config.get_config(). Remove the TODO comment.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-12 09:21:07 +02:00
Peter Hoyes
7647d7ad3e tests: Add failing test with repository includes
Add a new test to test_commands.py which:
 * Has an incomplete definition of externalrepo
 * Defines a local repository, subrepo
 * Includes a file from subrepo
 * The included file completes the definition of externalrepo
 * It also attempts to include a file from externalrepo

This test previously failed with the error message "No such file or
directory: '.../test_commands/tests/test_layers/test.yml'"

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-12 09:21:07 +02:00
Peter Hoyes
34983c13ee libcmds: Only checkout "missing" repositories in SetupReposStep
There is a subtle bug in SetupReposStep that occurs when:
 * One or more repositories are reported as "missing" in InitSetupRepos.
 * SetupReposStep checks out the missing repositories.
 * SetupReposStep updates the repo_dict with all repositories in the
   configuration, some of which may only be partially defined and/or
   not checked out.
 * SetupReposStep passes this new, incorrect list of checked out
   repositories to the include handler in the next pass.
 * The include handler attempts to include a file from a partially
   defined repository, likely resulting in a "File not found" error
   because it is assumed to be a local repository.

To fix this, keep track of the repositories that have been checked out
by only adding repositories that have been reported as "missing" to
ctx.config.repo_dict

Now that ctx.config.repo_dict only contains checked out repositories, it
is necessary to check missing repo names against the config dict
instead.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-12 09:21:07 +02:00
Peter Hoyes
958b0ab2f8 config: Public interface amendments
Add a repo_paths argument to find_missing_repos, with the default value
unchanged.

Factor out the contents of the loop in _get_repo_dict in config.py to
get_repo so that Repo instances can be created one at a time. The
behavior of _get_repo_dict is unchanged.

Add get_repos_config so it is possible to check whether a repo name
exists just in the config, as opposed to whether a Repo instance has
been created.

Add get_config to allow the config dict to be read in its entirety for
debugging purposes.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fix minor style issue]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-12 09:20:51 +02:00
Henning Schild
71cf5dc17b add NETRC_FILE to allow passing credentials into kas home
Tools like wget and git can read credentials from $HOME/.netrc for
servers that require authentication. Allow users to pass in a .netrc
file into the kas home dir to support i.e. bitbake https fetching with
auth.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix in command-line.rst]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-06-23 17:58:37 +02:00
Henning Schild
a8b69f5bd4 libcmds: stop creating empty .wgetrc and .netrc
Having empty files does not add any value. We do have a new HOME so
files from the old HOME are not used anyways. And writing an empty
.wgetrc will not disable the use of /etc/wgetrc. .netrc is supported by
many tools in slightly different ways, but also here if there was a
global config the one from HOME would likely be appended and would not
prevent use of the global one.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-06-21 12:57:53 +02:00
Diego Sueiro
720aebc79b Dockerfile: Add telnet
Some emulators use telnet to provide console access to the system.

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-06-21 12:57:24 +02:00
Florian Bezdeka
2f149509c2 kas-container: Add support for podman >= 4.1
Starting with podman 4.1 the --userns=keep-id flag is no longer ignored
for privileged containers leading to an error when trying to start up
such a container (in our case: for ISAR builds):

  Error: keep-id is only supported in rootless mode

To address that we have to move the --userns=keep-id part to a non-ISAR
specific path.

Reported-by: Wadim Klincov <wadim@klincov.com>
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-06-08 16:45:18 +02:00
Claudius Heine
42e83cb7a3 repos: add --whitespace=nowarn to git apply command
Patches contain tailing white space in empty lines around a change, those
are generally expected by git-apply and cause no issue.

However in cases where a patch file is added, removed or modified
in a patch, that patch would necessarily include `- ` or `+ ` lines,
since they operate on patch files.

Applying those patches with git-apply will cause git to print out
warnings about tailing white spaces per default, but still applies the
patch. Those warnings will be picked up by kas, and since they are
coming from stderr, print them out as `ERRORS`.

To solve this add `--whitespace=nowarn` as a parameter to the git-apply
call to silence those warnings.

In case of kas, it is to be expected that patches are added, removed or
changed in meta layers, so those supposedly errors will cause confusion.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-05-16 09:12:48 +02:00
Zhibin Dong
be4501b872 Dockerfile: reduce the image size
At my debian11/amd64 machine, the size of newer image is 889MB, while the
older one is 908MB.

Refs: #81
Signed-off-by: Zhibin Dong <zhibin.dong@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-05-16 09:12:06 +02:00
Felix Moessbauer
94e0e999f2 use relative layer dirs to make build relocatable
This patch replaces the absolute paths that are injected into BBLAYERS
by relative ones. These are relative to TOPDIR.
By that, the whole build directory becomes relocatable.

This is of value when using a shared sstate cache and build machines
with varying build locations (e.g. gitlab-ci runners).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-04-27 15:57:08 +02:00
Alban Bedel
b3405be5e8 libcmds: Add support for authentication with gitlab CI
Gitlab CI predefine many variables in its environment, among them the
sever hostname and a token that can be used to authenticate with the
server. If we find these variables in the environment add the
credentials to .netrc which in turn allow git and other tools to
access resources found on the CI server.

Signed-off-by: Alban Bedel <alban.bedel@aerq.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-04-26 18:10:17 +02:00
Harald Seiler
5ecef9f919 Allow "deleting" url/path of repo in override
Make "null" an allowed type for the `url` and `path` properties of a
repository.  This allows "deleting" them in an override kas-file to e.g.
point a repository at a local directory instead of a git remote:

"Original" file:

    repos:
      meta-foo:
        url: https://example.org/foo.git
        refspec: badc0ffee

Override:

    repos:
      meta-foo:
        url:
        path: /path/to/local/foo

Without the explicit "nulling" of the URL, kas would start messing with
the repo in /path/to/local/foo (changing remote URL, switching to a
different rev, etc.).  With this change, we can instead force kas to use
the recipes in that directory untouched.

Signed-off-by: Harald Seiler <hws@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-04-11 18:22:38 +02:00
Marcus Folkesson
0929101104 kas-container: fix typo in help text
cleanstate -> cleansstate

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-04-06 08:12:04 +02:00
Anders Montonen
7909ee2b18 kas-container: add '--ssh-agent' option
The option mounts the SSH_AUTH_SOCK ssh agent socket, and sets the
environment variable in the container.

Signed-off-by: Anders Montonen <Anders.Montonen@iki.fi>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-03-29 09:25:23 +02:00
Konrad Weihmann
e9bdbe8cfa setup.py: bundle kas-container script
so it will be shipped with the pypi release and users can
use this script from a secure and verifiable source, as pypi
if providing checksum for package verification, unlike as if
the script would be fetched just from github and set executable.

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-03-29 09:24:37 +02:00
Jan Kiszka
772cb5748b build: Resolve handling of -- separator in the absence of a config file
The fixed 'kas build -- -e', i.e. the build with the default
.config.yaml while trying to forward arguments to bitbake.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-03-14 12:57:11 +01:00
Jan Kiszka
680c67d756 Release 3.0.2 2022-03-08 15:52:55 +01:00
Jan Kiszka
c0aa2b9c99 Dockerfile.isar: Add missing escape in output redirection rule
Resolves

/kas/container-entrypoint: line 37: GOSU=gosu builder1: Read-only file system

or even writing a file "GOSU=gosu builder1" when calling "shell".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-03-08 15:21:01 +01:00
Jan Kiszka
229310958b Release 3.0.1 2022-03-08 14:31:59 +01:00
Jan Kiszka
f3aa8303eb Dockerfile.isar: Silence chatty sbuild-adduser
Unfortunately, just filtering stdout is not sufficient because this tool
also improperly uses stderr for printing

  cp /usr/share/doc/sbuild/examples/example.sbuildrc /builder/.sbuildrc

But we can assume all will be fine with it in our environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-03-03 18:48:52 +01:00
Jan Kiszka
41b52b1263 Release 3.0 2022-03-03 12:42:57 +01:00
Henning Schild
16a697736c repos: git fetch always with quiet flag
git-fetch prints its progress on stderr which makes harmless progress
output be rendered like ERRORs. Messages we likely do not care about and
certainly do not want to see as errors.
This gets us consistent with most of the other git operations using -q.

Closes: #78
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-26 09:21:13 +01:00
Ryan Fairfax
32514c8f4f shell: Add option to keep current environment
When doing interactive development via kas shell it is often desirable
to keep the user's customized configuration. The new --preserve-env
argument has been added to support this scenario via an opt in flag.

This flag is blocked  when not running from a TTY or via kas-container
and kas issues a warning to the user about potential unintended side
effects when invoked.

Signed-off-by: Ryan Fairfax <rfairfax@linux.microsoft.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-26 09:20:53 +01:00
Peter Hoyes
3d35fe2f60 libkas: Update BB_ENV_PASSTHROUGH_ADDITIONS with env_vars
Poky master has renamed BB_ENV_EXTRAWHITE ->
BB_ENV_PASSTHROUGH_ADDITIONS ahead of the kirkstone release
https://git.yoctoproject.org/poky/commit/?id=492214104a55ec56cc483c9b045038e355c6af7b

Update env var logic so that it updates either BB_ENV_EXTRAWHITE (for
honister or earlier) or BB_ENV_PASSTHROUGH_ADDITIONS (for master or
kirkstone), depending on which is defined in the bitbake environment.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-22 19:17:17 +01:00
Tobias Schaffner
1c53b817b5 fix: Raise an error if no refspec is provided
Only local repositories may be configured without refspec.
Raise an error if a repository url is given but refspec is not.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-21 20:05:31 +01:00
Claudius Heine
d7af4dc678 docs/userguide.rst: clarify local file include paths
How relative paths in local file includes are handles is more complex
than this one line makes out to be.

This change clarifies the documentation here.

Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: dropped duplicate blank at beginning of sentence]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-10 12:42:50 +01:00
Schmidl, Tobias
87ce7f90d6 kas-container: pass http_proxy et.al through sudo
If used in a podman environment, it's important to pass the http proxy
information down through sudo. This does not happen by default. We also
want to pass the entire environment down to avoid missing any other
environment variable.

Signed-off-by: Tobias Schmidl <tobiasschmidl@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-03 07:05:23 +01:00
Felix Moessbauer
94641a7b72 add pigz package to enable parallel compression of tar.gz archives
This patch adds the pigz package to enable parallel compression
and decompression support for the sstate artifacts.
This significantly speeds up compression of large artifacts on
build systems with many cores.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-01 19:32:35 +01:00
Uladzimir Bely
54ab356f9f Dockerfile.isar: support of sbuild
Upcoming support of sbuild in Isar requires some additional tools
to be preinstalled.

Also, `builder` user should be in `sbuild` group.

Additionally, to use external volume for schroot overlay because
the 'upper' overlayfs layer of sbuild can't be based on another
overlayfs filesystem that happens in case of using Docker.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-01-28 12:57:40 +01:00
Jan Kiszka
6abf837eab kas-container: Run enable_isar_mode only once
If both --isar is provide and build_system is set, enable_isar_mode will
be called twice that leads to KAS_CONTAINER_COMMAND gaining two "sudo" -
harmless but unneeded.

Reported-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-01-27 12:03:03 +01:00
Jan Kiszka
3318a5c79c ci: Add basic tests for built kas and kas-isar containers
Break up the build into two stages, performing basic tests with poky and
isar on the two kas container types before uploading them. The tests
consist of building zlib-native with poky and cowsay for x86 bullseye
with Isar.

Note that the ordering "build kas", "test kas", "push kas",
"build kas-isar" is important because "build kas-isar" will not use the
locally built and imported kas:next image and rather pull the previous
one from the registry.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-01-11 22:05:29 +01:00
Jan Kiszka
0730c971b8 checkcode.sh: Add container-entrypoint
Closes #67

Reported-by: Marius Kriegerowski <marius.kriegerowski@gfz-potsdam.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-01-11 15:06:04 +01:00
Jan Kiszka
df9b3af111 container-entrypoint: Address shellcheck findings
Reported-by: Marius Kriegerowski <marius.kriegerowski@gfz-potsdam.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-01-11 15:04:47 +01:00