It would be nice to have zstd frontend installed in the docker
environment. Since this is a relatively new and trending compression
algorithm, some packages might use it as part of their SRC's.
Also debian buster's tar has incorporated zstd support from upstream
GNU Tar version 1.31, starting from 1.30+dfsg-3. Therefore '.tar.zst'
extension files would be recognized and treated correctly by the tar
utility during unpack, etc.
Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.
Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Currently kas-container did not allow to set the `-d` parameter of kas,
this patch changes that and allows to set it via the `-d` or `-v`
parameter of kas-container.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
kas itself has a `-d` parameter that allows to print debug information,
while kas-container has the `-v` parameter. Since the `-d` parameter of
kas was first, rename the `-v` parameter of kas-container to fit kas.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.
The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.
The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The current default causes spurious pulls even when the image is already
locally available, at least with podman 2.0.x. Looking through podman
how-tos, none mentions that this prefix is required, and the
image_default_transport in containers.con is generally "docker://". So
it's safe to drop it for better default behavior.
If a custom local setup deviate, users can still override
KAS_CONTAINER_IMAGE_PATH, prepending what is desired.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.
Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.
Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch introduces new and improved ways to overwrite kas-container
variables that specify which image should be used.
`KAS_CONTAINER_IMAGE_PREFIX`: can be used to overwrite image sources.
For instance if podman is used, the default value `docker://` is used as
a prefix. But it still allows to overwrite it, by setting a custom
variable.
`KAS_CONTAINER_IMAGE_PATH`: can be used to customize the image source
path.
`KAS_CONTAINER_IMAGE_NAME`: can be set to change the container image
name. For instance if `--isar` is used, the default value is
`kas-isar`, as with `KAS_CONTAINER_IMAGE_PREFIX` this default value can
still be overwritten by setting a custom value.
And last `KAS_IMAGE_VERSION`: It was possible to overwrite this value
before this patch and its still possible. The default value is the
current release version. Setting it for instance to `latest`, would
allow to used the most recent version of the image.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Some variables used by the `kas-container` script have rather generic
terms, that might be used in bitbake recipes. This change prefixes all
variables with `KAS_` to avoid those possible variable name conflicts.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Add the github/ghcr equivalent to travis/dockerhub. With this in place,
we can also deploy the next and latest images to ghcr. Adjust
Dockerfile.isar accordingly. Also update the badge shown in the README.
We model the workflow with github a bit differently than so far: next
and master branch are different workflows. The former includes testing,
the latter just updates the container image. This is based on the
(offline) workflow to push changes to next first and have no commits
in master that haven't been in next before, except for release commits.
The master workflow also triggers on tags in which case it additionally
tags the container images accordingly and pushed that as well.
Travis is removed in the same run to avoid pushing differently built
images to the different registries.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is needed for running fakeroot from bullseye and sid which triggers
an unimplemented syscall with the current version.
Due to changes in the new qemu-user-static package, the current approach
to activate its binfmt settings no longer works. Instead, switch to an
explicit call of the binfmt-support in init script.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.
Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
All images are now also available on github's new container registry.
Refer to that as default, both in kas-docker as well as in the user
manual. The latter can benefit from some refreshment at this chance.
Automatic deployment via CI will be switched separately.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This is a bit tricky because the only posix shell compliant way is to
use the positional parameters list. However, that is already in use for
collecting docker arguments. But we can do the following:
Keep the extra args from the kas-docker command line in the list after
having parsed and removed all other kas-docker args. Save the number of
arguments we keep this way, then append the usual container engine args.
Before calling the engine, rotate the extra args from the front of the
positional parameters list to the back.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Eliminate manual call to os.remove by using tempfile context manager.
The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Shortens code and makes the intent clearer.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.
From git help fetch:
> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.
Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
.yaml is the recommended extension (https://yaml.org/faq.html)
Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When trying to run kas-docker with --isar on podman the build fails
with some "exec format" errors when the target has a different
architecture than the host. The reason is that /proc/sys/fs/binfmt_misc
is not mounted as expected inside the container.
The way the /proc filesystem is mounted seems to differ between podman
and docker. See [1] for the podman issue tracker. This is a temporary
fix and should be removed when the upstream issue was resolved.
[1] https://github.com/containers/podman/issues/4725
Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The --permissive and --cap-add options are conflicting for podman.
--permissive tells podman to enable all capabilities so additional
--cap-add options are not allowed / supported.
Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The following flags were added:
--userns=keep-id:
By default podman maps the user that is executing the podman run
command into the container as root user. That is not what the kas
entrypoint inside the container expects. Using keep-id tells podman
to use the expected 1:1 mapping. The entrypoint is able to create
the build user as expected.
--security-opt label=disable:
This is for SELinux enabled systems only. When running on SELinux
enabled systems the volumes assigned to the container would not be
accessible at all. The alternative to disable the labeling would
be passing the "z" or "Z" flag to all volumes (-v) which may be
even more dangerous.
Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Tests changing the working directory could leave us stranded somewhere
where all following tests could fail. Make sure to always go back on
failure.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Also update config file version to 9 due to this format change.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
While we support
layers:
.: excluded
this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.
Add support for that be converting a non-existing layers property into
layers:
.:
internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.
Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
kas-docker uses sh shell, which doesnot support pattern substitution
like bash. Replace Bash like pattern substitution with sed.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This ensures we don't try to check all python files in '/' if the
argument to the checkcode.sh script is missed by accident.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When running the checkcode.sh script, if no flake8 config file is
present then all error codes will be emitted. This may include error
codes from flake8 plugins installed locally which are not relevant for
this project (such as the flake8-black plugin). To prevent this issue we
add a flake8 config file which enables the built-in error codes C, E, F
and W.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Mercurial has no concept of a local tracking branch so we only need to
test git repositories here.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This makes it much simpler to implement new command line arguments.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By defining the plugin name and help message as attributes of the plugin
class we can move the argument parser creation up into the
kas_get_argparser() function. This will allow us to further reduce
duplication in following commits.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
