Add corresponding environment variables for the new keys and declare the
existing KAS_REPO_REFSPEC as deprecated in lock-step with the key it
mirrors.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Lock files are now using the commit key, rather than legacy refspec, and
the plugin writes out flattened configs that are in line with the input
configs (only write refspec if input repo was using refspec).
This also requires slight adjustments of the related test case. Enhance
the dump test at this chance to actually validate the written commit ID
against the expected one.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When using the kas menu command, we need to track the repository
location outside of the kas container as well. This is required to
prepare the environment for a later kas-container build, as the repo
path inside the container is different from the outside one. For that,
the location of KAS_REPO_DIR is passed via an env-var into the container.
There, this value is picked up by the menu plugin and written to the
.config.yaml file as `_source_host_dir`. When running kas-container build,
we check if the .config.yaml file contains this node and set the
KAS_REPO_DIR accordingly. The schema is extended accordingly.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: rework _source_dir_host extraction to make it usable for menu as well]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a new top-level schema node `_source_dir` which is used
to track the top repo dir on auto-generated kas files. This node is
generated by the kas menu plugin and provides the absolute path to the
top repo dir at time of invoking the plugin. When later calling any other
kas operation that performs a checkout, this node is evaluated and the
top repo dir is set accordingly. This tracking is required when the
build command is executed from another dir than the top repo dir, as the
repo-dir cannot be computed by kas in this case (the .config.yaml file
is in the KAS_WORK_DIR, not in the repo dir).
Proposed-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: aligned format-changelog text, simplified load_config returning]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch handles common Kconfig errors (file not found, invalid
config) and reports them as user errors. By that, the root cause of the
error is easier to spot by the user.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The auto-generated lockfiles should be terminated with a newline to be
POSIX compliant. Previously, this only worked for output via stdout but
not for inplace operations. This is fixed by appending the newline to
the active output target.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the --lock option to the dump plugin. When enabled, the
output only contains the resolved refspecs of each repo (as valid kas
format). By that, floating branches can be used in the projects kas files
and these can be pinned to fixed revisions, when required.
When using --lock in combination with --inplace, a lockfile named
<filename>.lock.<ext> is created next to the <filename>.<ext>. In case
multiple files are added to the kas CLI, the lockfile is only created
for the first file (by considering the merged information from all files).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fold in Python 3.6 support]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch puts the code examples of the dump plugin into code-blocks in
the documentation. By that, the documentation is easier to read.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the option --resolve-env to the dump plugin.
When enabled, all variables in the 'env' section of the kas config file
are set to the captured value (at time of executing the dump plugin).
This helps to debug build issues on CI runners by precisely capturing
the relevant environment.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a flag --resolve-refs to the dump plugin.
Once enabled, all relative refspecs (e.g. branch or tag names) are
resolved and replaced by their exact revision (before patches are
applied).
When re-running kas on the flattened and expanded config file, the build
is executed against exactly the versions of the dependencies that where
used when dumping the configuration. This helps to keep track of build
versions for future use (e.g. reproducible builds) or for
version-tracking tools like renovate.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.
While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.
The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is useful for people who start an interactive 'kas shell' and would
like to know the exact build command which 'kas build' would use.
Signed-off-by: Michael Adler <michael.adler@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Extend support for preserving the current environment to the
for-all-repos plugin with the --preserve-env flag.
This eases the usage of dynamic configuration done via environment
variables within the for-all-repos plugin, e.g. when calling a script.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The fixed 'kas build -- -e', i.e. the build with the default
.config.yaml while trying to forward arguments to bitbake.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When doing interactive development via kas shell it is often desirable
to keep the user's customized configuration. The new --preserve-env
argument has been added to support this scenario via an opt in flag.
This flag is blocked when not running from a TTY or via kas-container
and kas issues a warning to the user about potential unintended side
effects when invoked.
Signed-off-by: Ryan Fairfax <rfairfax@linux.microsoft.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
repo.url will point the repo.path in that case, but we want the user to
see an empty KAS_REPO_URL, as also documented.
Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence
kas menu -> save & exit
kas build
kas shell
kas ...
and also makes rebuilding the self-configured image simpler.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This uses the kconfiglib python module to implement simple interactive
configuration for complex layers with many kas options and even variable
input. See the menu.py documentation header for details on the Kconfig
format and its translation to kas.
Rather than using the limited menuconfig implementation of kconfiglib,
this comes with an own, python-newt based version. The permits smooth
integration in the workflow, e.g. to offer an option that directly
triggers a build after completing a configuration. And it comes with a
nicer layout.
Unfortunately, python-newt can't be fulfilled via pip. But it is
available as proper package via common distros. So warn if the package
is missing.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to define plugins which do not need have any of common args.
Move setup_parser_common_args into libkas for this as it now becomes a
library function.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
KAS should exit with the appropriate exit code, if the command parsed to the for_all_repos
plugin fails to apply on one of the repositories.
Example use-case: This allows for CI pipelines or wrapper scripts to take appropriate action (e.g. fail),
if an issue occured within the KAS for_all_repos command.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
That gets us in line with the default of bitbake itself. If users still
want to pass it, they will have to do so from now on.
While that continue switch can seem like a sane default for interactive
builds, it can also be pretty wasteful in CI. Where it would not just
eat compute ressources but also peoples time, because it potentially
delays build failures significantly.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Comes closer to how the command is spelled. We can't uses dashes as that
is not compatible with the python syntax.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The environment for executing the command in each repository is extended
to include the variables KAS_REPO_NAME, KAS_REPO_PATH, KAS_REPO_URL and
KAS_REPO_REFSPEC. This allows more complex actions to be automated via
this command.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin runs a specified command in all checked out repositories.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.
By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This makes the body of the Shell.run function closely match Build.run so
that we can factor out common code.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The target argument is only used when initializing the build plugin so
we can make it optional. This allows us to simplify the call to Config()
in the shell plugin and in other future plugins.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.
Also fix a typo in the build command help message.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.
The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
