When executing async commands, bail out in case the fail flag is set
and a command returns a non zero exit code. This is required to
ensure that further downstream commands are not executed as well.
Previously, the execution continued till the first error of the command
execution itself happened (e.g. command not found).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The common case that an include file cannot be found is now wrapped into
a LoadConfigException. By that, the user gets a more meaningful error
message on the console (instead of a stacktrace).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch handles common Kconfig errors (file not found, invalid
config) and reports them as user errors. By that, the root cause of the
error is easier to spot by the user.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the KasUserError exception class to distinguish between
internal kas exceptions and user or configuration errors. Exceptions
previously raised on user errors are ported over by deriving
KasUserError. In case of user errors, only the exception message is
shown, but no stacktrace. This makes it easier for users to locate the
issue as the reason is now stated in the last line of the output.
Kas internal exceptions are not subject to this change to help the
developers to find the root cause more easily.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The 'kas' function does not return any value, hence we must not use the
return value.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
KAS by default will output a lot of information (INFO) messages for
all operations, which makes it difficult to spot warnings thru all
that 'noise'.
Add a command line argument so that the default log level can be
modified.
For backward compatibility, the --debug parameter is still supported
but marked as deprecated in the help message.
Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fixes]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
To make it easier to display (and modify) the default log level,
especially with the introduction of the new (future commit) argument
--log-level.
Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The auto-generated lockfiles should be terminated with a newline to be
POSIX compliant. Previously, this only worked for output via stdout but
not for inplace operations. This is fixed by appending the newline to
the active output target.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the --lock option to the dump plugin. When enabled, the
output only contains the resolved refspecs of each repo (as valid kas
format). By that, floating branches can be used in the projects kas files
and these can be pinned to fixed revisions, when required.
When using --lock in combination with --inplace, a lockfile named
<filename>.lock.<ext> is created next to the <filename>.<ext>. In case
multiple files are added to the kas CLI, the lockfile is only created
for the first file (by considering the merged information from all files).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fold in Python 3.6 support]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When checking out repositories, check if a file <filename>.lock.<ext>
exists next to the file specified first on the kas CLI. In case this
file exists and the --update option is not specified, automatically
append this file to the kas CLI before performing any other kas
operations.
When --update is specified, the lockfile is ignored.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.
Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the required meta-schema identifiers to allow automatic
validation against a fixed version of json-schema.
Reported-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows the configuration of the git option 'credential.usehttppath' if
the used credential helper requires this.
Signed-off-by: Christoph Freundl <Christoph.Freundl@ifm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch puts the code examples of the dump plugin into code-blocks in
the documentation. By that, the documentation is easier to read.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.
Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.
The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.
The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the option --resolve-env to the dump plugin.
When enabled, all variables in the 'env' section of the kas config file
are set to the captured value (at time of executing the dump plugin).
This helps to debug build issues on CI runners by precisely capturing
the relevant environment.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a flag --resolve-refs to the dump plugin.
Once enabled, all relative refspecs (e.g. branch or tag names) are
resolved and replaced by their exact revision (before patches are
applied).
When re-running kas on the flattened and expanded config file, the build
is executed against exactly the versions of the dependencies that where
used when dumping the configuration. This helps to keep track of build
versions for future use (e.g. reproducible builds) or for
version-tracking tools like renovate.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a programmatic attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).
Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.
While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.
The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Until now all environment variables in the kas config's
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.
This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.
Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch forwards environment variables to control the concurrency and
parallelism of the build to bitbake.
With that, the user can reduce the concurrency in case of running short
on memory. Further, by reducing the concurrency on very large and NUMA
machines, the build sometimes can be significantly speed up.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is useful for people who start an interactive 'kas shell' and would
like to know the exact build command which 'kas build' would use.
Signed-off-by: Michael Adler <michael.adler@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If no configuration file is specified, we try to load a single
.config.yaml from KAS_WORK_DIR and set the top_repo_path to the
repo root of this file with fallback to the containing directory.
This process is identical to loading explicitly specified
configuration files and can therefore be combined into one case.
Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If we do not specify a configuration file for the "kas build" command,
the repository anchor is set to the current working directory,
which breaks the resolution of repo-relative include file paths
in the IncludeHandler class if repo root != current working directory.
Sets the repository anchor to the root path of the repository with
fallback to KAS_WORK_DIR if we use .config.yaml from KAS_WORK_DIR.
Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Extend support for preserving the current environment to the
for-all-repos plugin with the --preserve-env flag.
This eases the usage of dynamic configuration done via environment
variables within the for-all-repos plugin, e.g. when calling a script.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
There is one remaining protected access in SetupReposStep: an assignment
to ctx.config._config. Replace this with a call to
ctx.config.find_missing_repos, which already handles this assignment.
Remove the TODO comment.
There is one remaining protected access in FinishSetupRepos: a read of
ctx.config._config for debugging purposes. Replace this with a call to
ctx.config.get_config(). Remove the TODO comment.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
There is a subtle bug in SetupReposStep that occurs when:
* One or more repositories are reported as "missing" in InitSetupRepos.
* SetupReposStep checks out the missing repositories.
* SetupReposStep updates the repo_dict with all repositories in the
configuration, some of which may only be partially defined and/or
not checked out.
* SetupReposStep passes this new, incorrect list of checked out
repositories to the include handler in the next pass.
* The include handler attempts to include a file from a partially
defined repository, likely resulting in a "File not found" error
because it is assumed to be a local repository.
To fix this, keep track of the repositories that have been checked out
by only adding repositories that have been reported as "missing" to
ctx.config.repo_dict
Now that ctx.config.repo_dict only contains checked out repositories, it
is necessary to check missing repo names against the config dict
instead.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add a repo_paths argument to find_missing_repos, with the default value
unchanged.
Factor out the contents of the loop in _get_repo_dict in config.py to
get_repo so that Repo instances can be created one at a time. The
behavior of _get_repo_dict is unchanged.
Add get_repos_config so it is possible to check whether a repo name
exists just in the config, as opposed to whether a Repo instance has
been created.
Add get_config to allow the config dict to be read in its entirety for
debugging purposes.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fix minor style issue]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Tools like wget and git can read credentials from $HOME/.netrc for
servers that require authentication. Allow users to pass in a .netrc
file into the kas home dir to support i.e. bitbake https fetching with
auth.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix in command-line.rst]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Having empty files does not add any value. We do have a new HOME so
files from the old HOME are not used anyways. And writing an empty
.wgetrc will not disable the use of /etc/wgetrc. .netrc is supported by
many tools in slightly different ways, but also here if there was a
global config the one from HOME would likely be appended and would not
prevent use of the global one.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Patches contain tailing white space in empty lines around a change, those
are generally expected by git-apply and cause no issue.
However in cases where a patch file is added, removed or modified
in a patch, that patch would necessarily include `- ` or `+ ` lines,
since they operate on patch files.
Applying those patches with git-apply will cause git to print out
warnings about tailing white spaces per default, but still applies the
patch. Those warnings will be picked up by kas, and since they are
coming from stderr, print them out as `ERRORS`.
To solve this add `--whitespace=nowarn` as a parameter to the git-apply
call to silence those warnings.
In case of kas, it is to be expected that patches are added, removed or
changed in meta layers, so those supposedly errors will cause confusion.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch replaces the absolute paths that are injected into BBLAYERS
by relative ones. These are relative to TOPDIR.
By that, the whole build directory becomes relocatable.
This is of value when using a shared sstate cache and build machines
with varying build locations (e.g. gitlab-ci runners).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Gitlab CI predefine many variables in its environment, among them the
sever hostname and a token that can be used to authenticate with the
server. If we find these variables in the environment add the
credentials to .netrc which in turn allow git and other tools to
access resources found on the CI server.
Signed-off-by: Alban Bedel <alban.bedel@aerq.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make "null" an allowed type for the `url` and `path` properties of a
repository. This allows "deleting" them in an override kas-file to e.g.
point a repository at a local directory instead of a git remote:
"Original" file:
repos:
meta-foo:
url: https://example.org/foo.git
refspec: badc0ffee
Override:
repos:
meta-foo:
url:
path: /path/to/local/foo
Without the explicit "nulling" of the URL, kas would start messing with
the repo in /path/to/local/foo (changing remote URL, switching to a
different rev, etc.). With this change, we can instead force kas to use
the recipes in that directory untouched.
Signed-off-by: Harald Seiler <hws@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The fixed 'kas build -- -e', i.e. the build with the default
.config.yaml while trying to forward arguments to bitbake.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
git-fetch prints its progress on stderr which makes harmless progress
output be rendered like ERRORs. Messages we likely do not care about and
certainly do not want to see as errors.
This gets us consistent with most of the other git operations using -q.
Closes: #78
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When doing interactive development via kas shell it is often desirable
to keep the user's customized configuration. The new --preserve-env
argument has been added to support this scenario via an opt in flag.
This flag is blocked when not running from a TTY or via kas-container
and kas issues a warning to the user about potential unintended side
effects when invoked.
Signed-off-by: Ryan Fairfax <rfairfax@linux.microsoft.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
