As both test_dump and test_lockfile use the kas repo via its master
branch, they were now affected by master using the new syntax from some
of the test patterns. Account for that by checking the new property in
test_dump and by using a past commit with the new syntax in
test_lockfile.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
OpenEmbedded/Yocto expands TMPDIR by TCLIBCAPPEND. That break our -
granted - simplistic clean operation which assumes TMPDIR to be
"${TOPDIR}/tmp". Relax it a bit. This is still safe as we work on the
generally separate build dir.
An alternative to this would be calling kas shell with a concrete
configuration and ask for the actual TMPDIR and also DL_DIR while at it.
Probably doable, but it would require passing of a config file on each
global clean operation.
Isar layers are not expected expand TMPDIR, so keep things as they are
there.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Sync again with the list from latest Isar user manual. All of the "new"
packages (bzip2, python3-distutils, unzip, zstd) were already implicitly
installed, but this makes sure it they will not accidentally vanish on
the next base image change.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Covers both the invalid combinations refspec with the new commit/branch
keys as well as the issuing of a warning when a repo contains refspec.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Lift most test cases to the new syntax, just with few exceptions:
- tests/test_refspec/test2.yml actually stresses the mixed usage
(but clean per repo)
- tests/test_repo_includes references kas as external repo at a point
where there was no new syntax yet; this needs to be migrated
differently later on
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add corresponding environment variables for the new keys and declare the
existing KAS_REPO_REFSPEC as deprecated in lock-step with the key it
mirrors.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Lock files are now using the commit key, rather than legacy refspec, and
the plugin writes out flattened configs that are in line with the input
configs (only write refspec if input repo was using refspec).
This also requires slight adjustments of the related test case. Enhance
the dump test at this chance to actually validate the written commit ID
against the expected one.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to clearly express if a refspec is a commit ID or a
branch/symbolic reference. Mixing repos with old refspec with repos that
uses new commit/branch is supported, but not mixing the keys in the same
repo.
This commit lays the ground by extending the schema and ensuring that
the code which at minimum needs to know about these new keys does so.
Existing tests still pass, new ones will follow.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a refspec addresses an upstream branch which commit happens to be
local already, checkout will not properly replicate that into a local
branch. Extend contains_refspec_cmd so that it does not falsely report
the remotes/origin/ ref as locally available and rather triggers its
fetching.
The related test case was papering over this issue so far. Adjust it
accordingly.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a remote branch is referenced as refs/heads/<branch>, also drop the
heads/ from the local branch name to have a consistent naming.
This wasn't noticed so far because the primary use case of refs/ is
addressing symbolic refs in upstream, and the related test case
(test_refspec_absolute) was papering over another hidden kas issues
while using refs/heads/.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When using the kas menu command, we need to track the repository
location outside of the kas container as well. This is required to
prepare the environment for a later kas-container build, as the repo
path inside the container is different from the outside one. For that,
the location of KAS_REPO_DIR is passed via an env-var into the container.
There, this value is picked up by the menu plugin and written to the
.config.yaml file as `_source_host_dir`. When running kas-container build,
we check if the .config.yaml file contains this node and set the
KAS_REPO_DIR accordingly. The schema is extended accordingly.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: rework _source_dir_host extraction to make it usable for menu as well]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds two tests of the kas menu plugin in combination
with KAS_WORK_DIR.
Reported-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a new top-level schema node `_source_dir` which is used
to track the top repo dir on auto-generated kas files. This node is
generated by the kas menu plugin and provides the absolute path to the
top repo dir at time of invoking the plugin. When later calling any other
kas operation that performs a checkout, this node is evaluated and the
top repo dir is set accordingly. This tracking is required when the
build command is executed from another dir than the top repo dir, as the
repo-dir cannot be computed by kas in this case (the .config.yaml file
is in the KAS_WORK_DIR, not in the repo dir).
Proposed-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: aligned format-changelog text, simplified load_config returning]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch refactors the menu test by moving the monkey patching out
into a fixture. When changing the cwd, we now use the monkeypatch
tooling. By that, the following tests are not affected by a failure in a
previous one, which leads to a cleaner test report.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch removes the special handling of the -h / --help flag for kas
subcommands. By that, the build system detection logic properly works
and runs the help command in the kas-isar container for ISAR kas files.
While this logically does not make any difference, it avoids to pull the
kas OE container for ISAR builds, just to show the help message.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds tests for common errors when working with patches.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In ac4373088 the error handling was made more strict. By that, failures
during patch apply instantly bailed out instead of returning and
providing the more useful error message. This patch restores the error
handling by marking the commands as not-fail and encapsulating the error
into a more specific exception with a useful error message.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As the tasks with fail=True property return via an exception for a
long time, the handling of the return code is not required anymore.
This simplifies the code by removing dead code paths.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When async tasks fail, all exceptions need to be collected to not get
subsequent exceptions about invalid future states. This is achieved by
gathering the task results, instead of just waiting for them. By
gathering the results, also user-requested cancellation (e.g. via
ctrl-c) works without throwing tons of additional exceptions.
Since ac437308 we more likely run into that case, which unvealed the bug.
By properly handling the exception, a TaskResultError is returned
instead of the underlying CommandExecError. This change is reflected in
the corresponding unit test.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This adds a test for common command exec errors when checking out
repositories. The test checks if the correct error is thrown
(CommandExecError) instead of failing silently or with an
unspecific exception.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When executing async commands, bail out in case the fail flag is set
and a command returns a non zero exit code. This is required to
ensure that further downstream commands are not executed as well.
Previously, the execution continued till the first error of the command
execution itself happened (e.g. command not found).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The common case that an include file cannot be found is now wrapped into
a LoadConfigException. By that, the user gets a more meaningful error
message on the console (instead of a stacktrace).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch configures Coverage.py to measure only the code coverage of
kas itself. In addition, we add a note in the devguide how to create and
inspect the coverage data.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch handles common Kconfig errors (file not found, invalid
config) and reports them as user errors. By that, the root cause of the
error is easier to spot by the user.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the KasUserError exception class to distinguish between
internal kas exceptions and user or configuration errors. Exceptions
previously raised on user errors are ported over by deriving
KasUserError. In case of user errors, only the exception message is
shown, but no stacktrace. This makes it easier for users to locate the
issue as the reason is now stated in the last line of the output.
Kas internal exceptions are not subject to this change to help the
developers to find the root cause more easily.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The 'kas' function does not return any value, hence we must not use the
return value.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As part of the tests of the dump plugins a variable was shadowed. By
that, one branch of the test was not executed.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch forwards the log-level parameter from kas-container to kas
inside the container. For backward compatibility, the -d parameter is
deprecated but kept and also forwarded as-is to kas. By that, a more recent
kas-container script can still be used to run an older kas container.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
KAS by default will output a lot of information (INFO) messages for
all operations, which makes it difficult to spot warnings thru all
that 'noise'.
Add a command line argument so that the default log level can be
modified.
For backward compatibility, the --debug parameter is still supported
but marked as deprecated in the help message.
Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fixes]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
To make it easier to display (and modify) the default log level,
especially with the introduction of the new (future commit) argument
--log-level.
Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch makes the creation and update of a lockfile more convenient.
When running the dump plugin in inplace mode, a lockfile is created next
to the first file on the kas cmdline. By that, the repo directory also
needs to be mounted rw. Otherwise the kas inside the container cannot
create the file.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The auto-generated lockfiles should be terminated with a newline to be
POSIX compliant. Previously, this only worked for output via stdout but
not for inplace operations. This is fixed by appending the newline to
the active output target.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This commit adds a test that check the creation, effectiveness and
update of a lockfile. Testing this functionality via the dump plugin is
sufficient, as the plugin directly uses the checkout workflow.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fix over-long lines and removed assert brackets]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the --lock option to the dump plugin. When enabled, the
output only contains the resolved refspecs of each repo (as valid kas
format). By that, floating branches can be used in the projects kas files
and these can be pinned to fixed revisions, when required.
When using --lock in combination with --inplace, a lockfile named
<filename>.lock.<ext> is created next to the <filename>.<ext>. In case
multiple files are added to the kas CLI, the lockfile is only created
for the first file (by considering the merged information from all files).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fold in Python 3.6 support]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When checking out repositories, check if a file <filename>.lock.<ext>
exists next to the file specified first on the kas CLI. In case this
file exists and the --update option is not specified, automatically
append this file to the kas CLI before performing any other kas
operations.
When --update is specified, the lockfile is ignored.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.
Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As all currently supported distros provide at least python 3.6, we
drop the support for 3.5. At time of this commit, the python versions
of the supported distros are as following:
- Debian: 3.7 (buster)
- Ubuntu: 3.7 (18.04)
- Fedora: 3.11 (Fedora 36)
- RHEL: 2.x (RHEL 7), 3.6 (RHEL 8)
- OpenSUSE: 3.6 (Leap / 15.4)
While updating the lower bound version, we also unify the upper bound in
the setup.py script with the versions tested in the CI.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
