kas wrongly assumes so far that setting $HOME will also make ssh use the
kas provided $HOME/.ssh/ folder. But ssh will pick up the homedir from
/etc/passwd instead. This could cause kas to overwrite the users
~/.ssh/config when using SSH_PRIVATE_KEY*. We can try to cure ssh config
isolation, but that may cause surprises for users so far silently
relying on it.
For now, as a stable fix, avoid to cause damage to ~/.ssh/config in
cases where this is likely not desired, namely when there is already
config file. Warn if that file does not contain the generated content
from a previous run.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
repo.url will point the repo.path in that case, but we want the user to
see an empty KAS_REPO_URL, as also documented.
Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Irrespective of build system (OE / Yocto / Isar), they all have
BBPATH = "${TOPDIR}"
BBFILES ?= ""
in their bblayers_conf_header fragments. This could be dropped if kas
wrote those standard assignments already. So let's do that, weakly so
that those could still be overwritten via bblayers_conf_header.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence
kas menu -> save & exit
kas build
kas shell
kas ...
and also makes rebuilding the self-configured image simpler.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This uses the kconfiglib python module to implement simple interactive
configuration for complex layers with many kas options and even variable
input. See the menu.py documentation header for details on the Kconfig
format and its translation to kas.
Rather than using the limited menuconfig implementation of kconfiglib,
this comes with an own, python-newt based version. The permits smooth
integration in the workflow, e.g. to offer an option that directly
triggers a build after completing a configuration. And it comes with a
nicer layout.
Unfortunately, python-newt can't be fulfilled via pip. But it is
available as proper package via common distros. So warn if the package
is missing.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to define plugins which do not need have any of common args.
Move setup_parser_common_args into libkas for this as it now becomes a
library function.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
- Easier to reference as schema for IDEs
- Can be added to schemastore.org
Signed-off-by: ciarancourtney <ciaran.courtney@activeenergy.ie>
[Jan: rebased, merged in schema installation]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Since 2077900b4e, the existence of a repo key is enforced by the
config schema. So we can drop the redundant check in the include
handler.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Commit 2077900b4e was wrong in claiming that the documentation already
demanded repo-relative paths for local includes. It only did that for
dictionary-based include entries. Aligning both to repo-relative paths
remain a value, but we definitely have to update the documentation now.
While at it, also update the code comments accordingly. That also still
talked about the repo key being option, something that was never the
case in practice.
Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Using ssh to authenticate against repositories to check out is enabled.
When switching to https and authentication via the git credential helper
it is necessary to first setup the home directory before checking out the
repositories.
Signed-off-by: Ralf Anton Beier <ralf_beier@me.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When packaging kas for OpenSUSE, rpmlint threw the following warnings:
RPMLINT report:
===============
kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/__main__.py 644 /usr/bin/env python3
kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/kas.py 644 /usr/bin/env python3
This text file contains a shebang or is located in a path dedicated for
executables, but lacks the executable bits and cannot thus be executed. If
the file is meant to be an executable script, add the executable bits,
otherwise remove the shebang or move the file elsewhere.
2 packages and 0 specfiles checked; 0 errors, 2 warnings.
The Python files inside the kas module don't need shebang lines as they
are not executed directly so we can just remove them.
Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
KAS should exit with the appropriate exit code, if the command parsed to the for_all_repos
plugin fails to apply on one of the repositories.
Example use-case: This allows for CI pipelines or wrapper scripts to take appropriate action (e.g. fail),
if an issue occured within the KAS for_all_repos command.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.
Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.
As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We will fail differently then, but with a nicer error message (missing
init-build-env script) than
ValueError: Set of coroutines/Futures is empty.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
No one asked for this so far but the result of providing a relative
path, e.g. to KAS_WORK_DIR, is so confusing that it is better to simply
resolve all those vars to absolute paths.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree. This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes. When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.
Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.
Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If command line includes refer to the same repo, missing_repos was
aggregating those multiple times, causing multiple clone procedures to
be triggered. This generally failed. Therefore, build missing_repos
while checking for duplicates.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to set a git credential helper in the .gitconfig, in order
to handle git authentications. For instance when bitbake recipes require
to clone from git via https.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
when importing nested layer in combination with the top level layer like
meta-security
meta-security/meta-tpm
one can use
layers:
.:
meta-tpm:
in the spec file to import the parent layer.
This leads to the effect that
.
meta-tpm = "HEAD:<hash>"
is shown in the build console and in /etc/build
when using image-buildinfo.bbclass.
To fix that just strip the trailing dot from each layer name.
Add test to avoid future regressions
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
[Jan: use only a single rstrip call, fix style]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This variable can override default build path `${KAS_WORK_DIR}/build`.
Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
That gets us in line with the default of bitbake itself. If users still
want to pass it, they will have to do so from now on.
While that continue switch can seem like a sane default for interactive
builds, it can also be pretty wasteful in CI. Where it would not just
eat compute ressources but also peoples time, because it potentially
delays build failures significantly.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Refactored repo path calculation eliminates duplicated code and aligns
the matching logic (with respect to url and path) with the
documentation.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The `repo` documentation says:
"If the url is missing and the path defined, this entry references the
directory the path points to. [...] In case of a relative path name
`kas_work_dir` is prepended."
But if no `url` is given and a relative path is given in `path`, then
the layer paths being added to `bblayer.conf` are missing the
`kas_work_dir` prefix.
This patch fixes it.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Disable warn_on_full_buffer via set_wakeup_fd. This avoids tons of
Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable
that are currently issued after running complex builds. Only works since
Python 3.7, though, so this remains best-effort. Fortunately, the Debian
in our container image is 3.7.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Comes closer to how the command is spelled. We can't uses dashes as that
is not compatible with the python syntax.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The environment for executing the command in each repository is extended
to include the variables KAS_REPO_NAME, KAS_REPO_PATH, KAS_REPO_URL and
KAS_REPO_REFSPEC. This allows more complex actions to be automated via
this command.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.
Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin runs a specified command in all checked out repositories.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.
By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This makes the body of the Shell.run function closely match Build.run so
that we can factor out common code.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As we still support 3.5..3.7, we need to catch the case that
asyncio.get_running_loop and asyncio.all_tasks are not yet available.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Only tasks should be provided to it, no longer coroutines. Python 3.9
starts to complain about the deprecated pattern.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
