Commit Graph

314 Commits

Author SHA1 Message Date
Peter Hoyes
3d35fe2f60 libkas: Update BB_ENV_PASSTHROUGH_ADDITIONS with env_vars
Poky master has renamed BB_ENV_EXTRAWHITE ->
BB_ENV_PASSTHROUGH_ADDITIONS ahead of the kirkstone release
https://git.yoctoproject.org/poky/commit/?id=492214104a55ec56cc483c9b045038e355c6af7b

Update env var logic so that it updates either BB_ENV_EXTRAWHITE (for
honister or earlier) or BB_ENV_PASSTHROUGH_ADDITIONS (for master or
kirkstone), depending on which is defined in the bitbake environment.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-22 19:17:17 +01:00
Tobias Schaffner
1c53b817b5 fix: Raise an error if no refspec is provided
Only local repositories may be configured without refspec.
Raise an error if a repository url is given but refspec is not.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-21 20:05:31 +01:00
Jan Kiszka
75d1a5cce4 Release 2.6.3 2021-11-29 18:19:12 +01:00
Jan Kiszka
e3af001f53 libkas: Do not overwrite existing .ssh/config
kas wrongly assumes so far that setting $HOME will also make ssh use the
kas provided $HOME/.ssh/ folder. But ssh will pick up the homedir from
/etc/passwd instead. This could cause kas to overwrite the users
~/.ssh/config when using SSH_PRIVATE_KEY*. We can try to cure ssh config
isolation, but that may cause surprises for users so far silently
relying on it.

For now, as a stable fix, avoid to cause damage to ~/.ssh/config in
cases where this is likely not desired, namely when there is already
config file. Warn if that file does not contain the generated content
from a previous run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-23 18:26:13 +01:00
Jan Kiszka
3ad03d1590 Release 2.6.2 2021-11-02 13:01:21 +01:00
Jan Kiszka
0b394f0bb7 Release 2.6.1 2021-10-22 20:22:20 +02:00
Jan Kiszka
a81ee2c2b1 Release 2.6 2021-10-22 17:40:17 +02:00
Jan Kiszka
a40127f8ed for_all_repos: Empty KAS_REPO_URL if repo is not under version control
repo.url will point the repo.path in that case, but we want the user to
see an empty KAS_REPO_URL, as also documented.

Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-15 15:42:27 +02:00
Jan Kiszka
bfee47ebbc libcmds: Write more bblayers.conf boilerplate settings
Irrespective of build system (OE / Yocto / Isar), they all have

    BBPATH = "${TOPDIR}"
    BBFILES ?= ""

in their bblayers_conf_header fragments. This could be dropped if kas
wrote those standard assignments already. So let's do that, weakly so
that those could still be overwritten via bblayers_conf_header.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-11 08:20:32 +02:00
Jan Kiszka
5fb9067894 Fall back to .config.yaml if no configuration file given
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence

kas menu -> save & exit
kas build
kas shell
kas ...

and also makes rebuilding the self-configured image simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
eeab1bc9bc config: Pass context to Config constructor
Will be used later on to retrieve kas_work_dir.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
4f23045ce8 Add kconfiglib-based menu plugin
This uses the kconfiglib python module to implement simple interactive
configuration for complex layers with many kas options and even variable
input. See the menu.py documentation header for details on the Kconfig
format and its translation to kas.

Rather than using the limited menuconfig implementation of kconfiglib,
this comes with an own, python-newt based version. The permits smooth
integration in the workflow, e.g. to offer an option that directly
triggers a build after completing a configuration. And it comes with a
nicer layout.

Unfortunately, python-newt can't be fulfilled via pip. But it is
available as proper package via common distros. So warn if the package
is missing.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
4c975a04f9 plugins: Invoke setup_parser_common_args explicitly from setup_parser
This allows to define plugins which do not need have any of common args.

Move setup_parser_common_args into libkas for this as it now becomes a
library function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
abb645f8e9 Bump format revision due to changes to include paths
This justifies a version bump, even though the old behavior is still
supported.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 14:40:07 +02:00
ciarancourtney
460699685e yml: Move json schema from python dict to standalone json file
- Easier to reference as schema for IDEs
- Can be added to schemastore.org

Signed-off-by: ciarancourtney <ciaran.courtney@activeenergy.ie>
[Jan: rebased, merged in schema installation]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-05 10:28:24 +02:00
Jan Kiszka
6630780935 includehandler: Simplify code
Since 2077900b4e, the existence of a repo key is enforced by the
config schema. So we can drop the redundant check in the include
handler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-01 11:16:28 +02:00
Jan Kiszka
50e2d2d0fa Align include documentation to new logic
Commit 2077900b4e was wrong in claiming that the documentation already
demanded repo-relative paths for local includes. It only did that for
dictionary-based include entries. Aligning both to repo-relative paths
remain a value, but we definitely have to update the documentation now.

While at it, also update the code comments accordingly. That also still
talked about the repo key being option, something that was never the
case in practice.

Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-01 11:16:28 +02:00
Jan Kiszka
08631e428f build: Clarify need for separating extra bitbake args
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-09-25 09:27:58 +02:00
Ralf Anton Beier
2c3997ad56 Enable kas to checkout repositories using git credentials
Using ssh to authenticate against repositories to check out is enabled.
When switching to https and authentication via the git credential helper
it is necessary to first setup the home directory before checking out the
repositories.

Signed-off-by: Ralf Anton Beier <ralf_beier@me.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-09-15 07:29:00 +02:00
Paul Barker
c7ee61616a Drop unnecessary shebang lines
When packaging kas for OpenSUSE, rpmlint threw the following warnings:

    RPMLINT report:
    ===============
    kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/__main__.py 644 /usr/bin/env python3
    kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/kas.py 644 /usr/bin/env python3
    This text file contains a shebang or is located in a path dedicated for
    executables, but lacks the executable bits and cannot thus be executed.  If
    the file is meant to be an executable script, add the executable bits,
    otherwise remove the shebang or move the file elsewhere.

    2 packages and 0 specfiles checked; 0 errors, 2 warnings.

The Python files inside the kas module don't need shebang lines as they
are not executed directly so we can just remove them.

Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-30 21:56:49 +02:00
Jasper Orschulko
9732ec16f2 for_all_repos: Exit on command failure
KAS should exit with the appropriate exit code, if the command parsed to the for_all_repos
plugin fails to apply on one of the repositories.

Example use-case: This allows for CI pipelines or wrapper scripts to take appropriate action (e.g. fail),
if an issue occured within the KAS for_all_repos command.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-26 18:56:38 +02:00
Jan Kiszka
2077900b4e includehandler: Fix repo path detection for local includes
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.

Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.

As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:14 +02:00
Jan Kiszka
8a1c27bc92 libkas: Gracefully handle empty repo set
We will fail differently then, but with a nicer error message (missing
init-build-env script) than

ValueError: Set of coroutines/Futures is empty.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 23:17:41 +02:00
Jan Kiszka
4c81f96742 context: Add support for relative KAS_WORK/BUILD/REPO_REF_DIR paths
No one asked for this so far but the result of providing a relative
path, e.g. to KAS_WORK_DIR, is so confusing that it is better to simply
resolve all those vars to absolute paths.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 12:43:09 +02:00
Drew Reed
6b65de3e2f Enable gerrit/gitlab/github refspecs
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree.  This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes.  When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.

Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Jan Kiszka
635b1023a1 Declare proxy_config obsolete
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.

Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-12 08:03:40 +02:00
Jasper Orschulko
15de0142d4 Add environment variable SSH_PRIVATE_KEY_FILE
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-10 11:25:40 +02:00
Jan Kiszka
4d2b16e04d includehandler: Avoid duplicate cloning of repos in command line includes
If command line includes refer to the same repo, missing_repos was
aggregating those multiple times, causing multiple clone procedures to
be triggered. This generally failed. Therefore, build missing_repos
while checking for duplicates.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-03 18:08:12 +02:00
Jan Kiszka
28891711c0 Release 2.5 2021-05-26 07:27:23 +02:00
Claudius Heine
71e7f5523d kas: add GIT_CREDENTIAL_HELPER environment variable
This allows to set a git credential helper in the .gitconfig, in order
to handle git authentications. For instance when bitbake recipes require
to clone from git via https.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-06 14:08:40 +02:00
Konrad Weihmann
b250c341fb repos: strip dot from layer name
when importing nested layer in combination with the top level layer like

meta-security
meta-security/meta-tpm

one can use

layers:
    .:
    meta-tpm:

in the spec file to import the parent layer.
This leads to the effect that

.
meta-tpm             = "HEAD:<hash>"

is shown in the build console and in /etc/build
when using image-buildinfo.bbclass.
To fix that just strip the trailing dot from each layer name.

Add test to avoid future regressions

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
[Jan: use only a single rstrip call, fix style]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-05 15:43:58 +02:00
Peter Hatina
f32290d425 Introduce KAS_BUILD_DIR environment variable
This variable can override default build path `${KAS_WORK_DIR}/build`.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-10 07:53:09 +01:00
Peter Hatina
24c13a491b Apply patches before doing an environment setup
We may need to patch the environment setup, too.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-01 13:58:49 +01:00
Jan Kiszka
8fe75298e4 Release 2.4 2021-02-25 19:09:02 +01:00
Henning Schild
c50b3d21d8 kas: drop bitbakes "-k" from the default args
That gets us in line with the default of bitbake itself. If users still
want to pass it, they will have to do so from now on.

While that continue switch can seem like a sane default for interactive
builds, it can also be pretty wasteful in CI. Where it would not just
eat compute ressources but also peoples time, because it potentially
delays build failures significantly.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-01-19 18:14:59 +01:00
Silvano Cirujano Cuesta
0892b37ad6 repos: refactor repo path calculation
Refactored repo path calculation eliminates duplicated code and aligns
the matching logic (with respect to url and path) with the
documentation.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:12 +01:00
Silvano Cirujano Cuesta
1f1a21992d repos: fix path if no url, but path given
The `repo` documentation says:
"If the url is missing and the path defined, this entry references the
directory the path points to. [...] In case of a relative path name
`kas_work_dir` is prepended."

But if no `url` is given and a relative path is given in `path`, then
the layer paths being added to `bblayer.conf` are missing the
`kas_work_dir` prefix.

This patch fixes it.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:04 +01:00
Jan Kiszka
87af46ff8c Silence "Exception ignored when trying to write to the signal wakeup fd"
Disable warn_on_full_buffer via set_wakeup_fd. This avoids tons of

    Exception ignored when trying to write to the signal wakeup fd:
    BlockingIOError: [Errno 11] Resource temporarily unavailable

that are currently issued after running complex builds. Only works since
Python 3.7, though, so this remains best-effort. Fortunately, the Debian
in our container image is 3.7.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-02 12:40:19 +01:00
Jan Kiszka
368cdffc24 Release 2.3.3 2020-11-28 12:36:46 +01:00
Jan Kiszka
ba439d4200 Release 2.3.2 2020-11-20 12:20:24 +01:00
Jan Kiszka
f50316e1c9 Release 2.3.1 2020-11-20 11:16:18 +01:00
Jan Kiszka
7bbccbfb47 Release 2.3 2020-11-20 08:19:32 +01:00
Jan Kiszka
b36b2dbedd Rename forallrepos plugin to for_all_repos
Comes closer to how the command is spelled. We can't uses dashes as that
is not compatible with the python syntax.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 18:26:12 +01:00
Paul Barker
a89b011b80 for-all-repos: Extend environment to include repo config
The environment for executing the command in each repository is extended
to include the variables KAS_REPO_NAME, KAS_REPO_PATH, KAS_REPO_URL and
KAS_REPO_REFSPEC. This allows more complex actions to be automated via
this command.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:57:46 +01:00
Paul Barker
d42df24afe repos: Keep repo name as read from config file
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:52:00 +01:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Paul Barker
a8b5c79326 checkout: New plugin
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:57 +01:00
Paul Barker
baacd3a8b9 for-all-repos: New plugin
This plugin runs a specified command in all checked out repositories.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:51 +01:00
Paul Barker
bcb748c22a kas: Factor out common setup & cleanup steps
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.

By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:56:52 +01:00
Paul Barker
451a51f94a shell: Handle --keep-config-unchanged with existing skip mechanism
This makes the body of the Shell.run function closely match Build.run so
that we can factor out common code.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:58 +01:00
Paul Barker
c408e89ef1 kas: Eliminate Context.keep_config flag
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:35 +01:00
Jan Kiszka
85e3b1e74f Account for removal of aynchio.Tasks.all_tasks
As we still support 3.5..3.7, we need to catch the case that
asyncio.get_running_loop and asyncio.all_tasks are not yet available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:53:33 +01:00
Jan Kiszka
79641ac382 libkas: Acount for changes to asyncio.wait
Only tasks should be provided to it, no longer coroutines. Python 3.9
starts to complain about the deprecated pattern.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:23:05 +01:00
Jan Kiszka
204019f54e Remove up obsolete compat code for Python < 3.5
We demand 3.5 since e383e16570.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:16:01 +01:00
Paul Barker
c2e62df73f config: Make target argument optional in Config()
The target argument is only used when initializing the build plugin so
we can make it optional. This allows us to simplify the call to Config()
in the shell plugin and in other future plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:47:05 +01:00
Paul Barker
bad24e7468 kas: Load plugins in kas_get_argparser()
This fixes the missing sub-command entries in the User Guide section of
the docs by allowing the commands and arguments introduced by the
plugins to be seen by sphinx when building the docs.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:46:21 +01:00
Paul Barker
4e153a2872 plugins: Improve documentation of existing modules
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.

Also fix a typo in the build command help message.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:45:07 +01:00
Mert Kirpici
e79910aa29 config: avoid duplicates in BBMULTICONFIG
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-02 09:10:07 +01:00
Paul Barker
5e3ca820b3 plugins: Refactor plugin identification and access
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.

The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:17:03 +02:00
Paul Barker
7b9bce8e46 kas: Drop undocumented external plugin support
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:14:00 +02:00
Paul Barker
1dffb986b1 plugins: Move plugins to dedicated submodule
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:13:06 +02:00
Paul Barker
af4389e606 kas: Simplify plugin lookup
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.

The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:11:49 +02:00
Jan Kiszka
e5701a03a6 repos: Make set_remote_url_cmd fail loudly
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.

Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-13 14:06:31 +02:00
Jan Kiszka
b8a6577ece Report invalid repo key
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.

Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-10 16:17:16 +02:00
Adrian Stratulat
a5c8345ee6 repos: remove trailing slash from layer path
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.

Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-23 09:34:28 +02:00
Jan Kiszka
22b489b33b Release 2.2 2020-08-20 09:17:34 +02:00
Chris Laplante
9518dc4e35 libkas: make get_build_environ more exception safe
Eliminate manual call to os.remove by using tempfile context manager.

The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:33 +02:00
Chris Laplante
22f7c0b03d config: Make common repository check a bit more Pythonic
Shortens code and makes the intent clearer.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:25 +02:00
Nathan Genetzky
0e7c32cc79 repos: GitRepo fetch_cmd shouldn't default to "git fetch --all"
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.

From git help fetch:

> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.

Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-27 08:21:28 +02:00
Chris Laplante
7068767d27 kas: support .yaml extension
.yaml is the recommended extension (https://yaml.org/faq.html)

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-24 18:44:45 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Paul Barker
24f2f66cf5 kas: Support --update argument
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:40 +02:00
Paul Barker
1569895001 kas: Support --force-checkout argument
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:32 +02:00
Paul Barker
90598db3c8 context: Store arguments in global context
This makes it much simpler to implement new command line arguments.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:27 +02:00
Paul Barker
f0094af79f kas: De-duplicate common plugin args
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:57:11 +02:00
Paul Barker
a37d72ea7c kas: Simplify setup of plugin argument parsers
By defining the plugin name and help message as attributes of the plugin
class we can move the argument parser creation up into the
kas_get_argparser() function. This will allow us to further reduce
duplication in following commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:49:11 +02:00
Paul Barker
b998dfa722 repos: Make tracking branch resolution more robust
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.

It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:18:43 +02:00
Paul Barker
53668123c7 repos: Fix typo in repo type error message
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:07 +02:00
Jan Kiszka
e41d5af76b repos: Fix cloning of repos without default branch
Handle gracefully the case that a repo has no branch checked out after
cloning.

Reported-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-29 14:33:09 +02:00
Paul Barker
677c469fb5 build: Allow extra bitbake arguments to be passed
This allows the `kas build` command to be used when you need to pass
extra arguments to bitbake.

A `--` separator is needed when passing arguments which begin with a
dash so that argparse doesn't try to parse them as kas further
arguments.

For example, to pass `--skip-setscene` to bitbake:

    kas build config.yml -- --skip-setscene

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:08:00 +02:00
Paul Barker
8fb8b1b1e1 shell: Propagate non-zero exit status
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:37 +02:00
Paul Barker
3ba7091251 build: Propagate non-zero exit status when running in a tty
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:33 +02:00
Paul Barker
6ada99e40d kas: Don't return None from kas()
The return value of kas() is used as the argument to sys.exit() so it
should be 0 to report successful completion.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:02:50 +02:00
Paul Barker
44a87298ba build: Drop unnecessary argument to BuildCommand()
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 11:13:42 +02:00
Jan Kiszka
d02e87a0a7 Release 2.1.1 2020-05-19 08:03:26 +02:00
Jan Kiszka
d58a291513 repos: Silence pycodestyle error
New pycodestyle version, new warning: "l" might be confused with "1".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 07:53:02 +02:00
Jan Kiszka
68809708e8 Release 2.1 2020-05-19 07:30:20 +02:00
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Jan Kiszka
2142fd855e Fix reporting of repo patch IDs
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.

Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-10 18:10:17 +02:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Sam Lewis
b5ba370049 Ensure SSH key ends with newline
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').

This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.

Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:18:54 +01:00
Jan Kiszka
141476203d Release 2.0 2019-11-20 06:53:15 +01:00
Jan Kiszka
230c5a9572 Add support for new multiconfig syntax
Recent bitbake requires to declare multiconfig targets via "mc:".

Use that chance to refactor the function and match on the colon as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-11-08 10:36:08 +01:00
Jan Kiszka
e9ca55a239 Release 1.1 2019-10-01 16:21:40 +02:00
Jan Kiszka
3ef3ac855e Avoid "RuntimeError: Event loop is closed"
Triggered by pytest now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-25 11:00:45 +02:00