Commit Graph

377 Commits

Author SHA1 Message Date
Paul Barker
5e3ca820b3 plugins: Refactor plugin identification and access
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.

The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:17:03 +02:00
Paul Barker
7b9bce8e46 kas: Drop undocumented external plugin support
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:14:00 +02:00
Paul Barker
1dffb986b1 plugins: Move plugins to dedicated submodule
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:13:06 +02:00
Paul Barker
af4389e606 kas: Simplify plugin lookup
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.

The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:11:49 +02:00
Jan Kiszka
3403ba4923 kas-container: Drop container image prefix
The current default causes spurious pulls even when the image is already
locally available, at least with podman 2.0.x. Looking through podman
how-tos, none mentions that this prefix is required, and the
image_default_transport in containers.con is generally "docker://". So
it's safe to drop it for better default behavior.

If a custom local setup deviate, users can still override
KAS_CONTAINER_IMAGE_PATH, prepending what is desired.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-18 23:21:20 +02:00
Jan Kiszka
e5701a03a6 repos: Make set_remote_url_cmd fail loudly
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.

Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-13 14:06:31 +02:00
Jan Kiszka
b8a6577ece Report invalid repo key
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.

Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-10 16:17:16 +02:00
Claudius Heine
7e22208849 kas-container: Fix help output
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: further massaging]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-02 08:44:04 +02:00
Jan Kiszka
29bbf059a5 userguide: Switch to kas-container
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:36:42 +02:00
Jan Kiszka
1e4b4c7363 kas-container: Fix trailing whitespace
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:36:07 +02:00
Claudius Heine
503cf183b1 kas-container: make container image variables overwritable
This patch introduces new and improved ways to overwrite kas-container
variables that specify which image should be used.

`KAS_CONTAINER_IMAGE_PREFIX`: can be used to overwrite image sources.
For instance if podman is used, the default value `docker://` is used as
a prefix. But it still allows to overwrite it, by setting a custom
variable.

`KAS_CONTAINER_IMAGE_PATH`: can be used to customize the image source
path.

`KAS_CONTAINER_IMAGE_NAME`: can be set to change the container image
name.  For instance if `--isar` is used, the default value is
`kas-isar`, as with `KAS_CONTAINER_IMAGE_PREFIX` this default value can
still be overwritten by setting a custom value.

And last `KAS_IMAGE_VERSION`: It was possible to overwrite this value
before this patch and its still possible. The default value is the
current release version. Setting it for instance to `latest`, would
allow to used the most recent version of the image.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:35:04 +02:00
Claudius Heine
8d24d00bb3 kas-container: prefix variables with 'KAS_' to avoid conflicts
Some variables used by the `kas-container` script have rather generic
terms, that might be used in bitbake recipes. This change prefixes all
variables with `KAS_` to avoid those possible variable name conflicts.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:34:00 +02:00
Claudius Heine
c876548759 kas-container: rename 'docker' mentions more generic terms
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:33:30 +02:00
Claudius Heine
bb303ced8c rename docker-entrypoint to container-entrypoint
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:32:57 +02:00
Claudius Heine
52adce2387 rename kas-docker to kas-container, leave symlink in place
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:30:10 +02:00
Jan Kiszka
0f779e6577 Switch to github workflow
Add the github/ghcr equivalent to travis/dockerhub. With this in place,
we can also deploy the next and latest images to ghcr. Adjust
Dockerfile.isar accordingly. Also update the badge shown in the README.

We model the workflow with github a bit differently than so far: next
and master branch are different workflows. The former includes testing,
the latter just updates the container image. This is based on the
(offline) workflow to push changes to next first and have no commits
in master that haven't been in next before, except for release commits.

The master workflow also triggers on tags in which case it additionally
tags the container images accordingly and pushed that as well.

Travis is removed in the same run to avoid pushing differently built
images to the different registries.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-24 13:13:12 +02:00
Jan Kiszka
c7e15bdbd6 Dockerfile.isar: Pull qemu-user-static from buster-backports
This is needed for running fakeroot from bullseye and sid which triggers
an unimplemented syscall with the current version.

Due to changes in the new qemu-user-static package, the current approach
to activate its binfmt settings no longer works. Instead, switch to an
explicit call of the binfmt-support in init script.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-24 12:56:00 +02:00
Adrian Stratulat
a5c8345ee6 repos: remove trailing slash from layer path
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.

Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-23 09:34:28 +02:00
Jan Kiszka
e4be1a20ff kas-docker, docs: Switch to github container registry
All images are now also available on github's new container registry.
Refer to that as default, both in kas-docker as well as in the user
manual. The latter can benefit from some refreshment at this chance.

Automatic deployment via CI will be switched separately.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-07 13:36:43 +02:00
Jan Kiszka
747790d05d kas-docker: Preserve whitespaces in extra bitbake args
This is a bit tricky because the only posix shell compliant way is to
use the positional parameters list. However, that is already in use for
collecting docker arguments. But we can do the following:

Keep the extra args from the kas-docker command line in the list after
having parsed and removed all other kas-docker args. Save the number of
arguments we keep this way, then append the usual container engine args.
Before calling the engine, rotate the extra args from the front of the
positional parameters list to the back.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-01 08:22:39 +02:00
Jan Kiszka
22b489b33b Release 2.2 2020-08-20 09:17:34 +02:00
Chris Laplante
9518dc4e35 libkas: make get_build_environ more exception safe
Eliminate manual call to os.remove by using tempfile context manager.

The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:33 +02:00
Chris Laplante
22f7c0b03d config: Make common repository check a bit more Pythonic
Shortens code and makes the intent clearer.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:25 +02:00
Nathan Genetzky
0e7c32cc79 repos: GitRepo fetch_cmd shouldn't default to "git fetch --all"
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.

From git help fetch:

> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.

Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-27 08:21:28 +02:00
Chris Laplante
7068767d27 kas: support .yaml extension
.yaml is the recommended extension (https://yaml.org/faq.html)

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-24 18:44:45 +02:00
Florian Bezdeka
6b025e4910 kas-docker: Podman: Fixing isar builds failing with exec format errors
When trying to run kas-docker with --isar on podman the build fails
with some "exec format" errors when the target has a different
architecture than the host. The reason is that /proc/sys/fs/binfmt_misc
is not mounted as expected inside the container.

The way the /proc filesystem is mounted seems to differ between podman
and docker. See [1] for the podman issue tracker. This is a temporary
fix and should be removed when the upstream issue was resolved.

[1] https://github.com/containers/podman/issues/4725

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:41:20 +02:00
Florian Bezdeka
7ac9debaff kas-docker: Podman: Update podman flags to make isar builds working
The --permissive and --cap-add options are conflicting for podman.
--permissive tells podman to enable all capabilities so additional
--cap-add options are not allowed / supported.

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:41:04 +02:00
Florian Bezdeka
1688d60177 kas-docker: Podman: Update podman flags to make non-isar builds working
The following flags were added:
  --userns=keep-id:
    By default podman maps the user that is executing the podman run
    command into the container as root user. That is not what the kas
    entrypoint inside the container expects. Using keep-id tells podman
    to use the expected 1:1 mapping. The entrypoint is able to create
    the build user as expected.

  --security-opt label=disable:
    This is for SELinux enabled systems only. When running on SELinux
    enabled systems the volumes assigned to the container would not be
    accessible at all. The alternative to disable the labeling would
    be passing the "z" or "Z" flag to all volumes (-v) which may be
    even more dangerous.

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:40:50 +02:00
Henning Schild
4cfbf43f07 tests: add testcase for repo layers
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:57 +02:00
Henning Schild
0672592768 tests: introduce fixture to change back to right dir
Tests changing the working directory could leave us stranded somewhere
where all following tests could fail. Make sure to always go back on
failure.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:57 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Claudius Heine
b2e1074771 checkcode: add shell check
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:31 +02:00
Claudius Heine
f2c68e6862 kas-docker: fix shellcheck warnings
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: tiny indention fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:01 +02:00
Vijai Kumar K
a48ae6a7fa kas-docker: Fix bad substitution error
kas-docker uses sh shell, which doesnot support pattern substitution
like bash. Replace Bash like pattern substitution with sed.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-19 10:38:52 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Paul Barker
f8067a449b checkcode.sh: Abort if no path is provided
This ensures we don't try to check all python files in '/' if the
argument to the checkcode.sh script is missed by accident.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:00:04 +02:00
Paul Barker
04e5a2487c Add flake8 config
When running the checkcode.sh script, if no flake8 config file is
present then all error codes will be emitted. This may include error
codes from flake8 plugins installed locally which are not relevant for
this project (such as the flake8-black plugin). To prevent this issue we
add a flake8 config file which enables the built-in error codes C, E, F
and W.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:59:51 +02:00
Paul Barker
c2bbfee8f3 tests: Check refspec switch updates git branch/commit correctly
Mercurial has no concept of a local tracking branch so we only need to
test git repositories here.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:44 +02:00
Paul Barker
24f2f66cf5 kas: Support --update argument
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:40 +02:00
Paul Barker
1569895001 kas: Support --force-checkout argument
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:32 +02:00
Paul Barker
90598db3c8 context: Store arguments in global context
This makes it much simpler to implement new command line arguments.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:27 +02:00
Paul Barker
f0094af79f kas: De-duplicate common plugin args
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:57:11 +02:00
Paul Barker
a37d72ea7c kas: Simplify setup of plugin argument parsers
By defining the plugin name and help message as attributes of the plugin
class we can move the argument parser creation up into the
kas_get_argparser() function. This will allow us to further reduce
duplication in following commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:49:11 +02:00
Paul Barker
23277836c4 tests: Check patches can still be applied after switching refspec
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:53:59 +02:00
Paul Barker
79bfe5a04d tests: Return to kas directory when test_patch finishes
This prevents subseqent tests from starting in the wrong directory.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:53:54 +02:00
Paul Barker
b998dfa722 repos: Make tracking branch resolution more robust
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.

It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:18:43 +02:00
Harald Seiler
a5dc5f8663 kas-docker: Add support for podman
On many modern Linux systems, docker no longer works because they only
have cgroups v2 support.  To use kas in a container on these systems,
one can resort to using podman.  Add support for this alternative docker
engine to kas-docker.

Now, kas-docker searches for the first available engine in "docker,
podman" or, if the KAS_DOCKER_ENGINE environment variable is set, will
force the specified engine.

Signed-off-by: Harald Seiler <hws@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-03 12:00:11 +02:00