Commit Graph

390 Commits

Author SHA1 Message Date
Jan Kiszka
204019f54e Remove up obsolete compat code for Python < 3.5
We demand 3.5 since e383e16570.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:16:01 +01:00
Jan Kiszka
9e217c2632 ci: Add Python 3.9 to test matrix
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:02:24 +01:00
Paul Barker
c2e62df73f config: Make target argument optional in Config()
The target argument is only used when initializing the build plugin so
we can make it optional. This allows us to simplify the call to Config()
in the shell plugin and in other future plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:47:05 +01:00
Paul Barker
bad24e7468 kas: Load plugins in kas_get_argparser()
This fixes the missing sub-command entries in the User Guide section of
the docs by allowing the commands and arguments introduced by the
plugins to be seen by sphinx when building the docs.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:46:21 +01:00
Paul Barker
4e153a2872 plugins: Improve documentation of existing modules
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.

Also fix a typo in the build command help message.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:45:07 +01:00
Paul Barker
c2d20ca8cb docs: Fix title of intro page
The introduction page does not include installation instructions.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:42:24 +01:00
Paul Barker
6bc2627616 docs: Move command line details to a separate page
This change makes the user guide page more readable and results in a
command line usage page which is easy to navigate as all sub-commands
can be found in the left-hand outline panel.

Also fix the layout of the environment variables table by using a
replacement.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:41:57 +01:00
Paul Barker
f69311b18b docs: Update devguide to handle new plugin structure
* Add plugins module to the class reference documentation.

* Drop documentation of member classes and functions within plugins as
  the these aren't intended to be called directly outside of the plugin
  class itself.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:41:01 +01:00
Henning Schild
c3e1594b0e tests: small correction to git args
According to git-rev-parse(1) "-q" only does something in combination
with "--verify". Skip it, we do not need "--verify" because HEAD is
exactly one arg.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-09 08:57:04 +01:00
Mert Kirpici
348894158d Dockerfile: add zstd
It would be nice to have zstd frontend installed in the docker
environment. Since this is a relatively new and trending compression
algorithm, some packages might use it as part of their SRC's.

Also debian buster's tar has incorporated zstd support from upstream
GNU Tar version 1.31, starting from 1.30+dfsg-3. Therefore '.tar.zst'
extension files would be recognized and treated correctly by the tar
utility during unpack, etc.

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-02 09:10:37 +01:00
Mert Kirpici
e79910aa29 config: avoid duplicates in BBMULTICONFIG
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-02 09:10:07 +01:00
Claudius Heine
36ab0549ce kas-container: allow the -d/-v of kas-container to set -d on kas
Currently kas-container did not allow to set the `-d` parameter of kas,
this patch changes that and allows to set it via the `-d` or `-v`
parameter of kas-container.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-22 08:40:37 +02:00
Claudius Heine
b719fc9f6a kas-container: rename -v to -d and deprecate -v
kas itself has a `-d` parameter that allows to print debug information,
while kas-container has the `-v` parameter. Since the `-d` parameter of
kas was first, rename the `-v` parameter of kas-container to fit kas.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-22 08:40:25 +02:00
Paul Barker
5e3ca820b3 plugins: Refactor plugin identification and access
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.

The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:17:03 +02:00
Paul Barker
7b9bce8e46 kas: Drop undocumented external plugin support
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:14:00 +02:00
Paul Barker
1dffb986b1 plugins: Move plugins to dedicated submodule
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:13:06 +02:00
Paul Barker
af4389e606 kas: Simplify plugin lookup
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.

The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:11:49 +02:00
Jan Kiszka
3403ba4923 kas-container: Drop container image prefix
The current default causes spurious pulls even when the image is already
locally available, at least with podman 2.0.x. Looking through podman
how-tos, none mentions that this prefix is required, and the
image_default_transport in containers.con is generally "docker://". So
it's safe to drop it for better default behavior.

If a custom local setup deviate, users can still override
KAS_CONTAINER_IMAGE_PATH, prepending what is desired.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-18 23:21:20 +02:00
Jan Kiszka
e5701a03a6 repos: Make set_remote_url_cmd fail loudly
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.

Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-13 14:06:31 +02:00
Jan Kiszka
b8a6577ece Report invalid repo key
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.

Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-10 16:17:16 +02:00
Claudius Heine
7e22208849 kas-container: Fix help output
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: further massaging]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-02 08:44:04 +02:00
Jan Kiszka
29bbf059a5 userguide: Switch to kas-container
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:36:42 +02:00
Jan Kiszka
1e4b4c7363 kas-container: Fix trailing whitespace
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:36:07 +02:00
Claudius Heine
503cf183b1 kas-container: make container image variables overwritable
This patch introduces new and improved ways to overwrite kas-container
variables that specify which image should be used.

`KAS_CONTAINER_IMAGE_PREFIX`: can be used to overwrite image sources.
For instance if podman is used, the default value `docker://` is used as
a prefix. But it still allows to overwrite it, by setting a custom
variable.

`KAS_CONTAINER_IMAGE_PATH`: can be used to customize the image source
path.

`KAS_CONTAINER_IMAGE_NAME`: can be set to change the container image
name.  For instance if `--isar` is used, the default value is
`kas-isar`, as with `KAS_CONTAINER_IMAGE_PREFIX` this default value can
still be overwritten by setting a custom value.

And last `KAS_IMAGE_VERSION`: It was possible to overwrite this value
before this patch and its still possible. The default value is the
current release version. Setting it for instance to `latest`, would
allow to used the most recent version of the image.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:35:04 +02:00
Claudius Heine
8d24d00bb3 kas-container: prefix variables with 'KAS_' to avoid conflicts
Some variables used by the `kas-container` script have rather generic
terms, that might be used in bitbake recipes. This change prefixes all
variables with `KAS_` to avoid those possible variable name conflicts.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:34:00 +02:00
Claudius Heine
c876548759 kas-container: rename 'docker' mentions more generic terms
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:33:30 +02:00
Claudius Heine
bb303ced8c rename docker-entrypoint to container-entrypoint
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:32:57 +02:00
Claudius Heine
52adce2387 rename kas-docker to kas-container, leave symlink in place
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:30:10 +02:00
Jan Kiszka
0f779e6577 Switch to github workflow
Add the github/ghcr equivalent to travis/dockerhub. With this in place,
we can also deploy the next and latest images to ghcr. Adjust
Dockerfile.isar accordingly. Also update the badge shown in the README.

We model the workflow with github a bit differently than so far: next
and master branch are different workflows. The former includes testing,
the latter just updates the container image. This is based on the
(offline) workflow to push changes to next first and have no commits
in master that haven't been in next before, except for release commits.

The master workflow also triggers on tags in which case it additionally
tags the container images accordingly and pushed that as well.

Travis is removed in the same run to avoid pushing differently built
images to the different registries.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-24 13:13:12 +02:00
Jan Kiszka
c7e15bdbd6 Dockerfile.isar: Pull qemu-user-static from buster-backports
This is needed for running fakeroot from bullseye and sid which triggers
an unimplemented syscall with the current version.

Due to changes in the new qemu-user-static package, the current approach
to activate its binfmt settings no longer works. Instead, switch to an
explicit call of the binfmt-support in init script.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-24 12:56:00 +02:00
Adrian Stratulat
a5c8345ee6 repos: remove trailing slash from layer path
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.

Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-23 09:34:28 +02:00
Jan Kiszka
e4be1a20ff kas-docker, docs: Switch to github container registry
All images are now also available on github's new container registry.
Refer to that as default, both in kas-docker as well as in the user
manual. The latter can benefit from some refreshment at this chance.

Automatic deployment via CI will be switched separately.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-07 13:36:43 +02:00
Jan Kiszka
747790d05d kas-docker: Preserve whitespaces in extra bitbake args
This is a bit tricky because the only posix shell compliant way is to
use the positional parameters list. However, that is already in use for
collecting docker arguments. But we can do the following:

Keep the extra args from the kas-docker command line in the list after
having parsed and removed all other kas-docker args. Save the number of
arguments we keep this way, then append the usual container engine args.
Before calling the engine, rotate the extra args from the front of the
positional parameters list to the back.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-01 08:22:39 +02:00
Jan Kiszka
22b489b33b Release 2.2 2020-08-20 09:17:34 +02:00
Chris Laplante
9518dc4e35 libkas: make get_build_environ more exception safe
Eliminate manual call to os.remove by using tempfile context manager.

The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:33 +02:00
Chris Laplante
22f7c0b03d config: Make common repository check a bit more Pythonic
Shortens code and makes the intent clearer.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:25 +02:00
Nathan Genetzky
0e7c32cc79 repos: GitRepo fetch_cmd shouldn't default to "git fetch --all"
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.

From git help fetch:

> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.

Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-27 08:21:28 +02:00
Chris Laplante
7068767d27 kas: support .yaml extension
.yaml is the recommended extension (https://yaml.org/faq.html)

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-24 18:44:45 +02:00
Florian Bezdeka
6b025e4910 kas-docker: Podman: Fixing isar builds failing with exec format errors
When trying to run kas-docker with --isar on podman the build fails
with some "exec format" errors when the target has a different
architecture than the host. The reason is that /proc/sys/fs/binfmt_misc
is not mounted as expected inside the container.

The way the /proc filesystem is mounted seems to differ between podman
and docker. See [1] for the podman issue tracker. This is a temporary
fix and should be removed when the upstream issue was resolved.

[1] https://github.com/containers/podman/issues/4725

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:41:20 +02:00
Florian Bezdeka
7ac9debaff kas-docker: Podman: Update podman flags to make isar builds working
The --permissive and --cap-add options are conflicting for podman.
--permissive tells podman to enable all capabilities so additional
--cap-add options are not allowed / supported.

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:41:04 +02:00
Florian Bezdeka
1688d60177 kas-docker: Podman: Update podman flags to make non-isar builds working
The following flags were added:
  --userns=keep-id:
    By default podman maps the user that is executing the podman run
    command into the container as root user. That is not what the kas
    entrypoint inside the container expects. Using keep-id tells podman
    to use the expected 1:1 mapping. The entrypoint is able to create
    the build user as expected.

  --security-opt label=disable:
    This is for SELinux enabled systems only. When running on SELinux
    enabled systems the volumes assigned to the container would not be
    accessible at all. The alternative to disable the labeling would
    be passing the "z" or "Z" flag to all volumes (-v) which may be
    even more dangerous.

Signed-off-by: Florian Bezdeka <florian@bezdeka.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-15 13:40:50 +02:00
Henning Schild
4cfbf43f07 tests: add testcase for repo layers
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:57 +02:00
Henning Schild
0672592768 tests: introduce fixture to change back to right dir
Tests changing the working directory could leave us stranded somewhere
where all following tests could fail. Make sure to always go back on
failure.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:57 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Claudius Heine
b2e1074771 checkcode: add shell check
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:31 +02:00
Claudius Heine
f2c68e6862 kas-docker: fix shellcheck warnings
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: tiny indention fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:01 +02:00
Vijai Kumar K
a48ae6a7fa kas-docker: Fix bad substitution error
kas-docker uses sh shell, which doesnot support pattern substitution
like bash. Replace Bash like pattern substitution with sed.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-19 10:38:52 +02:00