The environment for executing the command in each repository is extended
to include the variables KAS_REPO_NAME, KAS_REPO_PATH, KAS_REPO_URL and
KAS_REPO_REFSPEC. This allows more complex actions to be automated via
this command.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.
Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The init script is a bit noisy, so replace it with a direct call of
update-binfmts. That tool unfortunately fails to report missing kernel
support via a proper error code. Catch that by doing an extra check on
the mounted filesystem.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This plugin runs a specified command in all checked out repositories.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.
By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This makes the body of the Shell.run function closely match Build.run so
that we can factor out common code.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
"Line break occurred before a binary operator (W503)" goes against PEP8
and has been superseded by W504.
See https://www.flake8rules.com/rules/W503.html.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As we still support 3.5..3.7, we need to catch the case that
asyncio.get_running_loop and asyncio.all_tasks are not yet available.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Only tasks should be provided to it, no longer coroutines. Python 3.9
starts to complain about the deprecated pattern.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The target argument is only used when initializing the build plugin so
we can make it optional. This allows us to simplify the call to Config()
in the shell plugin and in other future plugins.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This fixes the missing sub-command entries in the User Guide section of
the docs by allowing the commands and arguments introduced by the
plugins to be seen by sphinx when building the docs.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.
Also fix a typo in the build command help message.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The introduction page does not include installation instructions.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change makes the user guide page more readable and results in a
command line usage page which is easy to navigate as all sub-commands
can be found in the left-hand outline panel.
Also fix the layout of the environment variables table by using a
replacement.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
* Add plugins module to the class reference documentation.
* Drop documentation of member classes and functions within plugins as
the these aren't intended to be called directly outside of the plugin
class itself.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
According to git-rev-parse(1) "-q" only does something in combination
with "--verify". Skip it, we do not need "--verify" because HEAD is
exactly one arg.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
It would be nice to have zstd frontend installed in the docker
environment. Since this is a relatively new and trending compression
algorithm, some packages might use it as part of their SRC's.
Also debian buster's tar has incorporated zstd support from upstream
GNU Tar version 1.31, starting from 1.30+dfsg-3. Therefore '.tar.zst'
extension files would be recognized and treated correctly by the tar
utility during unpack, etc.
Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.
Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Currently kas-container did not allow to set the `-d` parameter of kas,
this patch changes that and allows to set it via the `-d` or `-v`
parameter of kas-container.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
kas itself has a `-d` parameter that allows to print debug information,
while kas-container has the `-v` parameter. Since the `-d` parameter of
kas was first, rename the `-v` parameter of kas-container to fit kas.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.
The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.
The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The current default causes spurious pulls even when the image is already
locally available, at least with podman 2.0.x. Looking through podman
how-tos, none mentions that this prefix is required, and the
image_default_transport in containers.con is generally "docker://". So
it's safe to drop it for better default behavior.
If a custom local setup deviate, users can still override
KAS_CONTAINER_IMAGE_PATH, prepending what is desired.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.
Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.
Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch introduces new and improved ways to overwrite kas-container
variables that specify which image should be used.
`KAS_CONTAINER_IMAGE_PREFIX`: can be used to overwrite image sources.
For instance if podman is used, the default value `docker://` is used as
a prefix. But it still allows to overwrite it, by setting a custom
variable.
`KAS_CONTAINER_IMAGE_PATH`: can be used to customize the image source
path.
`KAS_CONTAINER_IMAGE_NAME`: can be set to change the container image
name. For instance if `--isar` is used, the default value is
`kas-isar`, as with `KAS_CONTAINER_IMAGE_PREFIX` this default value can
still be overwritten by setting a custom value.
And last `KAS_IMAGE_VERSION`: It was possible to overwrite this value
before this patch and its still possible. The default value is the
current release version. Setting it for instance to `latest`, would
allow to used the most recent version of the image.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Some variables used by the `kas-container` script have rather generic
terms, that might be used in bitbake recipes. This change prefixes all
variables with `KAS_` to avoid those possible variable name conflicts.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add the github/ghcr equivalent to travis/dockerhub. With this in place,
we can also deploy the next and latest images to ghcr. Adjust
Dockerfile.isar accordingly. Also update the badge shown in the README.
We model the workflow with github a bit differently than so far: next
and master branch are different workflows. The former includes testing,
the latter just updates the container image. This is based on the
(offline) workflow to push changes to next first and have no commits
in master that haven't been in next before, except for release commits.
The master workflow also triggers on tags in which case it additionally
tags the container images accordingly and pushed that as well.
Travis is removed in the same run to avoid pushing differently built
images to the different registries.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is needed for running fakeroot from bullseye and sid which triggers
an unimplemented syscall with the current version.
Due to changes in the new qemu-user-static package, the current approach
to activate its binfmt settings no longer works. Instead, switch to an
explicit call of the binfmt-support in init script.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.
Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
All images are now also available on github's new container registry.
Refer to that as default, both in kas-docker as well as in the user
manual. The latter can benefit from some refreshment at this chance.
Automatic deployment via CI will be switched separately.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is a bit tricky because the only posix shell compliant way is to
use the positional parameters list. However, that is already in use for
collecting docker arguments. But we can do the following:
Keep the extra args from the kas-docker command line in the list after
having parsed and removed all other kas-docker args. Save the number of
arguments we keep this way, then append the usual container engine args.
Before calling the engine, rotate the extra args from the front of the
positional parameters list to the back.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Eliminate manual call to os.remove by using tempfile context manager.
The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
