There is one remaining protected access in SetupReposStep: an assignment
to ctx.config._config. Replace this with a call to
ctx.config.find_missing_repos, which already handles this assignment.
Remove the TODO comment.
There is one remaining protected access in FinishSetupRepos: a read of
ctx.config._config for debugging purposes. Replace this with a call to
ctx.config.get_config(). Remove the TODO comment.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add a new test to test_commands.py which:
* Has an incomplete definition of externalrepo
* Defines a local repository, subrepo
* Includes a file from subrepo
* The included file completes the definition of externalrepo
* It also attempts to include a file from externalrepo
This test previously failed with the error message "No such file or
directory: '.../test_commands/tests/test_layers/test.yml'"
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
There is a subtle bug in SetupReposStep that occurs when:
* One or more repositories are reported as "missing" in InitSetupRepos.
* SetupReposStep checks out the missing repositories.
* SetupReposStep updates the repo_dict with all repositories in the
configuration, some of which may only be partially defined and/or
not checked out.
* SetupReposStep passes this new, incorrect list of checked out
repositories to the include handler in the next pass.
* The include handler attempts to include a file from a partially
defined repository, likely resulting in a "File not found" error
because it is assumed to be a local repository.
To fix this, keep track of the repositories that have been checked out
by only adding repositories that have been reported as "missing" to
ctx.config.repo_dict
Now that ctx.config.repo_dict only contains checked out repositories, it
is necessary to check missing repo names against the config dict
instead.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add a repo_paths argument to find_missing_repos, with the default value
unchanged.
Factor out the contents of the loop in _get_repo_dict in config.py to
get_repo so that Repo instances can be created one at a time. The
behavior of _get_repo_dict is unchanged.
Add get_repos_config so it is possible to check whether a repo name
exists just in the config, as opposed to whether a Repo instance has
been created.
Add get_config to allow the config dict to be read in its entirety for
debugging purposes.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fix minor style issue]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Tools like wget and git can read credentials from $HOME/.netrc for
servers that require authentication. Allow users to pass in a .netrc
file into the kas home dir to support i.e. bitbake https fetching with
auth.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix in command-line.rst]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Having empty files does not add any value. We do have a new HOME so
files from the old HOME are not used anyways. And writing an empty
.wgetrc will not disable the use of /etc/wgetrc. .netrc is supported by
many tools in slightly different ways, but also here if there was a
global config the one from HOME would likely be appended and would not
prevent use of the global one.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Some emulators use telnet to provide console access to the system.
Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Starting with podman 4.1 the --userns=keep-id flag is no longer ignored
for privileged containers leading to an error when trying to start up
such a container (in our case: for ISAR builds):
Error: keep-id is only supported in rootless mode
To address that we have to move the --userns=keep-id part to a non-ISAR
specific path.
Reported-by: Wadim Klincov <wadim@klincov.com>
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Patches contain tailing white space in empty lines around a change, those
are generally expected by git-apply and cause no issue.
However in cases where a patch file is added, removed or modified
in a patch, that patch would necessarily include `- ` or `+ ` lines,
since they operate on patch files.
Applying those patches with git-apply will cause git to print out
warnings about tailing white spaces per default, but still applies the
patch. Those warnings will be picked up by kas, and since they are
coming from stderr, print them out as `ERRORS`.
To solve this add `--whitespace=nowarn` as a parameter to the git-apply
call to silence those warnings.
In case of kas, it is to be expected that patches are added, removed or
changed in meta layers, so those supposedly errors will cause confusion.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
At my debian11/amd64 machine, the size of newer image is 889MB, while the
older one is 908MB.
Refs: #81
Signed-off-by: Zhibin Dong <zhibin.dong@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch replaces the absolute paths that are injected into BBLAYERS
by relative ones. These are relative to TOPDIR.
By that, the whole build directory becomes relocatable.
This is of value when using a shared sstate cache and build machines
with varying build locations (e.g. gitlab-ci runners).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Gitlab CI predefine many variables in its environment, among them the
sever hostname and a token that can be used to authenticate with the
server. If we find these variables in the environment add the
credentials to .netrc which in turn allow git and other tools to
access resources found on the CI server.
Signed-off-by: Alban Bedel <alban.bedel@aerq.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make "null" an allowed type for the `url` and `path` properties of a
repository. This allows "deleting" them in an override kas-file to e.g.
point a repository at a local directory instead of a git remote:
"Original" file:
repos:
meta-foo:
url: https://example.org/foo.git
refspec: badc0ffee
Override:
repos:
meta-foo:
url:
path: /path/to/local/foo
Without the explicit "nulling" of the URL, kas would start messing with
the repo in /path/to/local/foo (changing remote URL, switching to a
different rev, etc.). With this change, we can instead force kas to use
the recipes in that directory untouched.
Signed-off-by: Harald Seiler <hws@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The option mounts the SSH_AUTH_SOCK ssh agent socket, and sets the
environment variable in the container.
Signed-off-by: Anders Montonen <Anders.Montonen@iki.fi>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
so it will be shipped with the pypi release and users can
use this script from a secure and verifiable source, as pypi
if providing checksum for package verification, unlike as if
the script would be fetched just from github and set executable.
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The fixed 'kas build -- -e', i.e. the build with the default
.config.yaml while trying to forward arguments to bitbake.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Resolves
/kas/container-entrypoint: line 37: GOSU=gosu builder1: Read-only file system
or even writing a file "GOSU=gosu builder1" when calling "shell".
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Unfortunately, just filtering stdout is not sufficient because this tool
also improperly uses stderr for printing
cp /usr/share/doc/sbuild/examples/example.sbuildrc /builder/.sbuildrc
But we can assume all will be fine with it in our environment.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
git-fetch prints its progress on stderr which makes harmless progress
output be rendered like ERRORs. Messages we likely do not care about and
certainly do not want to see as errors.
This gets us consistent with most of the other git operations using -q.
Closes: #78
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When doing interactive development via kas shell it is often desirable
to keep the user's customized configuration. The new --preserve-env
argument has been added to support this scenario via an opt in flag.
This flag is blocked when not running from a TTY or via kas-container
and kas issues a warning to the user about potential unintended side
effects when invoked.
Signed-off-by: Ryan Fairfax <rfairfax@linux.microsoft.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Poky master has renamed BB_ENV_EXTRAWHITE ->
BB_ENV_PASSTHROUGH_ADDITIONS ahead of the kirkstone release
https://git.yoctoproject.org/poky/commit/?id=492214104a55ec56cc483c9b045038e355c6af7b
Update env var logic so that it updates either BB_ENV_EXTRAWHITE (for
honister or earlier) or BB_ENV_PASSTHROUGH_ADDITIONS (for master or
kirkstone), depending on which is defined in the bitbake environment.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Only local repositories may be configured without refspec.
Raise an error if a repository url is given but refspec is not.
Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
How relative paths in local file includes are handles is more complex
than this one line makes out to be.
This change clarifies the documentation here.
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: dropped duplicate blank at beginning of sentence]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If used in a podman environment, it's important to pass the http proxy
information down through sudo. This does not happen by default. We also
want to pass the entire environment down to avoid missing any other
environment variable.
Signed-off-by: Tobias Schmidl <tobiasschmidl@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the pigz package to enable parallel compression
and decompression support for the sstate artifacts.
This significantly speeds up compression of large artifacts on
build systems with many cores.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Upcoming support of sbuild in Isar requires some additional tools
to be preinstalled.
Also, `builder` user should be in `sbuild` group.
Additionally, to use external volume for schroot overlay because
the 'upper' overlayfs layer of sbuild can't be based on another
overlayfs filesystem that happens in case of using Docker.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If both --isar is provide and build_system is set, enable_isar_mode will
be called twice that leads to KAS_CONTAINER_COMMAND gaining two "sudo" -
harmless but unneeded.
Reported-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Break up the build into two stages, performing basic tests with poky and
isar on the two kas container types before uploading them. The tests
consist of building zlib-native with poky and cowsay for x86 bullseye
with Isar.
Note that the ordering "build kas", "test kas", "push kas",
"build kas-isar" is important because "build kas-isar" will not use the
locally built and imported kas:next image and rather pull the previous
one from the registry.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
To complete the set of clean tasks that OE and Isar offer, add
cleansstate to purge SSTATE_DIR and cleanall also delete DL_DIR.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This helps reaping zombies if processes do not perform proper cleanups.
Known to stumble is bazel so far, see
https://github.com/bazelbuild/bazel/issues/13823. But as the overhead of
an init service is negligible and problems around this are hard to
debug, we better add this option by default.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We were using the host PID namespace as workaround for problems related
to binfmt and its missing namespace support. As it turns out after
running a bunch of tests this is no longer necessary.
This patch "reverts" 6b025e4910 ("kas-docker: Podman: Fixing isar builds
failing with exec format errors").
Test matrix:
kas layer podman 3.4.4 podman 3.0.1 podman 3.4.4
on Fedora 35 on Debian 11 on Debian 11
(podman from testing)
xenomai-images
ISAR_CROSS_COMPILE = 1 OK OK OK
ISAR_CROSS_COMPILE = 0 OK OK OK
iot-2000 OK FAIL [1] FAIL [1]
[1] The iot-2000 layer is not ISAR based, so we do not run in privileged
mode for such builds which seems to make a difference when using the
--userns=keep-id argument. As it works on Fedora and the error message
indicates "creating of systemd unit failed" it might by systemd related.
podman run --rm -t -i --userns=keep-id debian:buster-slim
Fedora: OK
Debian: Fail
Error: OCI runtime error: error creating systemd unit
`libpod-<snip>.scope`: got `failed`
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is added to avoid confusion in the creation of downstream layers.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
[Jan: fixed over-long line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Currently the containers are based on buster, but use buster-backports,
and (in case if the Isar container) even plain bullseye with pinning as
additional deb soure.
This changes that to use bullseye only.
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Permits installing via wheel, even if this dependency is not yet
installed.
Reported-by: Marius Kriegerowski <marius.kriegerowski@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Adding `.` was not intuitively for me, therefore an explicit comment on
how to add both the repo and a subdir might be useful.
Signed-off-by: Tobias Schmidl <tobias@schmidl.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
kas wrongly assumes so far that setting $HOME will also make ssh use the
kas provided $HOME/.ssh/ folder. But ssh will pick up the homedir from
/etc/passwd instead. This could cause kas to overwrite the users
~/.ssh/config when using SSH_PRIVATE_KEY*. We can try to cure ssh config
isolation, but that may cause surprises for users so far silently
relying on it.
For now, as a stable fix, avoid to cause damage to ~/.ssh/config in
cases where this is likely not desired, namely when there is already
config file. Warn if that file does not contain the generated content
from a previous run.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Use the more powerful configuration file .readthedocs.yaml to ensure
that all requirements are installed. This now includes also a distro
package, python-newt, something that cannot be installed via a
requirements.txt.
Fixes the doc build for 2.6.x which become broken with the addition of
the menu plugin.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Check if KAS_REPO_REF_DIR exists to avoid an error during
the execution of readlink.
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If KAS_WORK_DIR is set in the executing shell
and does not exists kas-container fails silently
during readlink -f on KAS_BUILD_DIR if the default
"${KAS_WORK_DIR}/build" is used.
Create KAS_WORK_DIR to ensure the subsequent
code execution.
This can be tested by setting KAS_WORK_DIR, e.g.
```
KAS_WORK_DIR="$(pwd)/kas_work" kas-container ...
```
Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The github CI environment causes the copied /kas folder to become
world-writable. This is at least undesired, so fix up the permissions.
This is done the classic way, i.e. via a separate chmod, rather than
using the new and not yet widely available "COPY --chmod".
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Moving this file to /usr/local/bin broke user configurations that had
/usr/bin/ hard-coded, and it also broke inside the kas sandbox which
does not have /usr/local/bin in its PATH. So move things back to the
original location.
Do that by setting a link to the /kas/contrib folder, rather than
copying the file once more.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
