Commit Graph

329 Commits

Author SHA1 Message Date
Jan Kiszka
0d268197a4 plugins: for_all_repos: Add support for commit/branch
Add corresponding environment variables for the new keys and declare the
existing KAS_REPO_REFSPEC as deprecated in lock-step with the key it
mirrors.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 17:37:02 +02:00
Jan Kiszka
241d03f159 Switch overrides and dump plugin to new commit/branch schema
Lock files are now using the commit key, rather than legacy refspec, and
the plugin writes out flattened configs that are in line with the input
configs (only write refspec if input repo was using refspec).

This also requires slight adjustments of the related test case. Enhance
the dump test at this chance to actually validate the written commit ID
against the expected one.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 17:32:03 +02:00
Jan Kiszka
c01a11833f repos: Warn if a repo uses legacy refspec
We are deprecating its usage but not yet removing it. Warn once per
parsed repo and kas run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 17:29:44 +02:00
Jan Kiszka
bb48adad1f repos: Introduce commit and branch and alternative to refspec key
This allows to clearly express if a refspec is a commit ID or a
branch/symbolic reference. Mixing repos with old refspec with repos that
uses new commit/branch is supported, but not mixing the keys in the same
repo.

This commit lays the ground by extending the schema and ensuring that
the code which at minimum needs to know about these new keys does so.
Existing tests still pass, new ones will follow.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 16:55:07 +02:00
Jan Kiszka
51846faef4 repos: hg: Add real support for resolve_branch_cmd
Will be needed to unify invalid branch name detection.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Jan Kiszka
f1f358bed5 repos: git: Always replicate refs/ into local branches on checkout
If a refspec addresses an upstream branch which commit happens to be
local already, checkout will not properly replicate that into a local
branch. Extend contains_refspec_cmd so that it does not falsely report
the remotes/origin/ ref as locally available and rather triggers its
fetching.

The related test case was papering over this issue so far. Adjust it
accordingly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Jan Kiszka
420bf399b3 repos: git: Strip of heads/ prefix when checking out branches
If a remote branch is referenced as refs/heads/<branch>, also drop the
heads/ from the local branch name to have a consistent naming.

This wasn't noticed so far because the primary use case of refs/ is
addressing symbolic refs in upstream, and the related test case
(test_refspec_absolute) was papering over another hidden kas issues
while using refs/heads/.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Jan Kiszka
a317a0e957 repos: Rename boolean branch variable/argument to is_branch
Makes it clearer that this is a boolean, not the branch name.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Felix Moessbauer
e9032bc278 kas-container: track repo dir when using menu cmd
When using the kas menu command, we need to track the repository
location outside of the kas container as well. This is required to
prepare the environment for a later kas-container build, as the repo
path inside the container is different from the outside one. For that,
the location of KAS_REPO_DIR is passed via an env-var into the container.
There, this value is picked up by the menu plugin and written to the
.config.yaml file as `_source_host_dir`. When running kas-container build,
we check if the .config.yaml file contains this node and set the
KAS_REPO_DIR accordingly. The schema is extended accordingly.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: rework _source_dir_host extraction to make it usable for menu as well]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:29:12 +02:00
Felix Moessbauer
76db0706f3 menu plugin: track root repo dir
This patch adds a new top-level schema node `_source_dir` which is used
to track the top repo dir on auto-generated kas files. This node is
generated by the kas menu plugin and provides the absolute path to the
top repo dir at time of invoking the plugin. When later calling any other
kas operation that performs a checkout, this node is evaluated and the
top repo dir is set accordingly. This tracking is required when the
build command is executed from another dir than the top repo dir, as the
repo-dir cannot be computed by kas in this case (the .config.yaml file
is in the KAS_WORK_DIR, not in the repo dir).

Proposed-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: aligned format-changelog text, simplified load_config returning]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 11:07:53 +02:00
Jan Kiszka
a1f38f9979 includehandler: Derive validator from schema
No need to hard-code this, validator_for() can find the right one as
well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-26 17:36:17 +02:00
Jan Kiszka
3aea34cab6 schema-kas.json: Fix dialect URL
Use the proper one according to
https://json-schema.org/understanding-json-schema/reference/schema.html.
Will help jsonschema to find the right validator.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-26 17:34:42 +02:00
Felix Moessbauer
5e4da5020b repos: restore error handling of patch apply
In ac4373088 the error handling was made more strict. By that, failures
during patch apply instantly bailed out instead of returning and
providing the more useful error message. This patch restores the error
handling by marking the commands as not-fail and encapsulating the error
into a more specific exception with a useful error message.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:49:53 +02:00
Felix Moessbauer
d3963ff494 refactor: remove dead code paths on task exception
As the tasks with fail=True property return via an exception for a
long time, the handling of the return code is not required anymore.
This simplifies the code by removing dead code paths.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:49:49 +02:00
Felix Moessbauer
ecd670e9ae fix: collect exceptions on task errors
When async tasks fail, all exceptions need to be collected to not get
subsequent exceptions about invalid future states. This is achieved by
gathering the task results, instead of just waiting for them. By
gathering the results, also user-requested cancellation (e.g. via
ctrl-c) works without throwing tons of additional exceptions.
Since ac437308 we more likely run into that case, which unvealed the bug.

By properly handling the exception, a TaskResultError is returned
instead of the underlying CommandExecError. This change is reflected in
the corresponding unit test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:36:31 +02:00
Felix Moessbauer
ac43730889 bail out on error in async command marked as fail
When executing async commands, bail out in case the fail flag is set
and a command returns a non zero exit code. This is required to
ensure that further downstream commands are not executed as well.

Previously, the execution continued till the first error of the command
execution itself happened (e.g. command not found).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-10 09:02:07 +02:00
Felix Moessbauer
b21a7e1c3b includes: wrap missing include file exception
The common case that an include file cannot be found is now wrapped into
a LoadConfigException. By that, the user gets a more meaningful error
message on the console (instead of a stacktrace).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-10 09:01:51 +02:00
Felix Moessbauer
2503ab39cd menu plugin: handle common KConfig errors
This patch handles common Kconfig errors (file not found, invalid
config) and reports them as user errors. By that, the root cause of the
error is easier to spot by the user.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:56:40 +02:00
Felix Moessbauer
222f07de69 refactor: port all sys.exit over to kas exceptions
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:56:32 +02:00
Felix Moessbauer
a5750901c6 use custom exceptions to improve error handling
This patch adds the KasUserError exception class to distinguish between
internal kas exceptions and user or configuration errors. Exceptions
previously raised on user errors are ported over by deriving
KasUserError. In case of user errors, only the exception message is
shown, but no stacktrace. This makes it easier for users to locate the
issue as the reason is now stated in the last line of the output.

Kas internal exceptions are not subject to this change to help the
developers to find the root cause more easily.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:54:40 +02:00
Felix Moessbauer
6fa40363aa kas-main: do not use return type of void function
The 'kas' function does not return any value, hence we must not use the
return value.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-04 09:03:36 +02:00
Hugo Villeneuve
522a2422c4 kas: add support for --log-level argument
KAS by default will output a lot of information (INFO) messages for
all operations, which makes it difficult to spot warnings thru all
that 'noise'.

Add a command line argument so that the default log level can be
modified.

For backward compatibility, the --debug parameter is still supported
but marked as deprecated in the help message.

Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fixes]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 17:25:36 +02:00
Hugo Villeneuve
246f23cbca kas: define variable to store default log level
To make it easier to display (and modify) the default log level,
especially with the introduction of the new (future commit) argument
--log-level.

Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 17:23:42 +02:00
Felix Moessbauer
88964d63b5 improve documentation of lockfile support
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 09:05:55 +02:00
Felix Moessbauer
471f5e58d2 dump plugin: add newline to file in inplace mode
The auto-generated lockfiles should be terminated with a newline to be
POSIX compliant. Previously, this only worked for output via stdout but
not for inplace operations. This is fixed by appending the newline to
the active output target.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-01 14:19:48 +02:00
Felix Moessbauer
7c0953c08e dump plugin: add option to create lockfile
This patch adds the --lock option to the dump plugin. When enabled, the
output only contains the resolved refspecs of each repo (as valid kas
format). By that, floating branches can be used in the projects kas files
and these can be pinned to fixed revisions, when required.

When using --lock in combination with --inplace, a lockfile named
<filename>.lock.<ext> is created next to the <filename>.<ext>. In case
multiple files are added to the kas CLI, the lockfile is only created
for the first file (by considering the merged information from all files).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fold in Python 3.6 support]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:05:04 +02:00
Felix Moessbauer
fe4031ce01 add support for lockfiles on checkout
When checking out repositories, check if a file <filename>.lock.<ext>
exists next to the file specified first on the kas CLI. In case this
file exists and the --update option is not specified, automatically
append this file to the kas CLI before performing any other kas
operations.

When --update is specified, the lockfile is ignored.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:05:04 +02:00
Felix Moessbauer
3e0dd10416 add support to override refspec of repos
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.

Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:04:51 +02:00
Felix Moessbauer
93cd91bc7c add meta-schema identifiers in kas json schema
This patch adds the required meta-schema identifiers to allow automatic
validation against a fixed version of json-schema.

Reported-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Freundl, Christoph' via kas-devel
8f61dcfeb6 kas: add GIT_CREDENTIAL_USEHTTPPATH environment variable
This allows the configuration of the git option 'credential.usehttppath' if
the used credential helper requires this.

Signed-off-by: Christoph Freundl <Christoph.Freundl@ifm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Felix Moessbauer
ea4099c2c8 improve style of dump plugin documentation
This patch puts the code examples of the dump plugin into code-blocks in
the documentation. By that, the documentation is easier to read.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-03-19 19:45:28 +01:00
Jan Kiszka
b9abbd1e63 Release 3.2.3 2023-03-05 10:36:58 +01:00
Jan Kiszka
0e61fdddaf Release 3.2.2 2023-02-13 21:18:57 +01:00
Jan Kiszka
66893480c7 Release 3.2.1 2023-02-02 08:41:17 +01:00
Jan Kiszka
dc44638cd8 Release 3.2 2023-01-17 12:49:44 +01:00
Felix Moessbauer
3289ec2a7e re-create tmpdir after successfull rename
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:12:04 +01:00
Felix Moessbauer
f2560588bc auto create repo refs when setting KAS_REPO_REF_DIR
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.

Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.

The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.

The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:11:06 +01:00
Felix Moessbauer
e7896c33d9 do not short-circuit on initial clone of repos
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:10:51 +01:00
Felix Moessbauer
20a69016da dump plugin: add option to capture current env
This patch adds the option --resolve-env to the dump plugin.
When enabled, all variables in the 'env' section of the kas config file
are set to the captured value (at time of executing the dump plugin).

This helps to debug build issues on CI runners by precisely capturing
the relevant environment.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:40:04 +01:00
Felix Moessbauer
ffabaa19f3 extend dump plugin to support resolving revisions
This patch adds a flag --resolve-refs to the dump plugin.
Once enabled, all relative refspecs (e.g. branch or tag names) are
resolved and replaced by their exact revision (before patches are
applied).
When re-running kas on the flattened and expanded config file, the build
is executed against exactly the versions of the dependencies that where
used when dumping the configuration. This helps to keep track of build
versions for future use (e.g. reproducible builds) or for
version-tracking tools like renovate.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:43 +01:00
Felix Moessbauer
23dcf955a2 add revision attribute to repo class
This patch adds a programmatic attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).

Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:25 +01:00
Felix Moessbauer
5b3e238128 add plugin to dump flattened config
This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.

While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.

The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:38:41 +01:00
Benedikt Niedermayr
7215e58eab libkas: Fix typo
Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-11 13:36:12 +01:00
Benedikt Niedermayr
c38517df83 add conditional environment variables
Until now all environment variables in the kas config's
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-10 19:03:48 +01:00
Felix Moessbauer
102c4667d1 forward build-concurrency related env vars
This patch forwards environment variables to control the concurrency and
parallelism of the build to bitbake.
With that, the user can reduce the concurrency in case of running short
on memory. Further, by reducing the concurrency on very large and NUMA
machines, the build sometimes can be significantly speed up.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-10-19 11:02:17 +02:00
Michael Adler
271f0bf6ca feat(shell): print bitbake command
This is useful for people who start an interactive 'kas shell' and would
like to know the exact build command which 'kas build' would use.

Signed-off-by: Michael Adler <michael.adler@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-10 09:41:53 +02:00
Jan Kiszka
d1834dd6bf Release 3.1 2022-08-05 13:17:58 +02:00
Johann Neuhauser
bce8a916fa config: Symplify .config.yaml file handling
If no configuration file is specified, we try to load a single
.config.yaml from KAS_WORK_DIR and set the top_repo_path to the
repo root of this file with fallback to the containing directory.

This process is identical to loading explicitly specified
configuration files and can therefore be combined into one case.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:55 +02:00
Johann Neuhauser
cbdba7e8e1 config: Fix repo-relative include file handling if no config file is given
If we do not specify a configuration file for the "kas build" command,
the repository anchor is set to the current working directory,
which breaks the resolution of repo-relative include file paths
in the IncludeHandler class if repo root != current working directory.

Sets the repository anchor to the root path of the repository with
fallback to KAS_WORK_DIR if we use .config.yaml from KAS_WORK_DIR.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:49 +02:00
Jasper Orschulko
a9cc7d06b3 for-all-repos: Add option to keep current env
Extend support for preserving the current environment to the
for-all-repos plugin with the --preserve-env flag.
This eases the usage of dynamic configuration done via environment
variables within the for-all-repos plugin, e.g. when calling a script.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-22 14:28:00 +02:00