Commit Graph

66 Commits

Author SHA1 Message Date
Felix Moessbauer
ecd670e9ae fix: collect exceptions on task errors
When async tasks fail, all exceptions need to be collected to not get
subsequent exceptions about invalid future states. This is achieved by
gathering the task results, instead of just waiting for them. By
gathering the results, also user-requested cancellation (e.g. via
ctrl-c) works without throwing tons of additional exceptions.
Since ac437308 we more likely run into that case, which unvealed the bug.

By properly handling the exception, a TaskResultError is returned
instead of the underlying CommandExecError. This change is reflected in
the corresponding unit test.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:36:31 +02:00
Felix Moessbauer
ac43730889 bail out on error in async command marked as fail
When executing async commands, bail out in case the fail flag is set
and a command returns a non zero exit code. This is required to
ensure that further downstream commands are not executed as well.

Previously, the execution continued till the first error of the command
execution itself happened (e.g. command not found).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-10 09:02:07 +02:00
Felix Moessbauer
222f07de69 refactor: port all sys.exit over to kas exceptions
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:56:32 +02:00
Benedikt Niedermayr
7215e58eab libkas: Fix typo
Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-11 13:36:12 +01:00
Benedikt Niedermayr
c38517df83 add conditional environment variables
Until now all environment variables in the kas config's
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-10 19:03:48 +01:00
Johann Neuhauser
cbdba7e8e1 config: Fix repo-relative include file handling if no config file is given
If we do not specify a configuration file for the "kas build" command,
the repository anchor is set to the current working directory,
which breaks the resolution of repo-relative include file paths
in the IncludeHandler class if repo root != current working directory.

Sets the repository anchor to the root path of the repository with
fallback to KAS_WORK_DIR if we use .config.yaml from KAS_WORK_DIR.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:49 +02:00
Jasper Orschulko
a9cc7d06b3 for-all-repos: Add option to keep current env
Extend support for preserving the current environment to the
for-all-repos plugin with the --preserve-env flag.
This eases the usage of dynamic configuration done via environment
variables within the for-all-repos plugin, e.g. when calling a script.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-22 14:28:00 +02:00
Peter Hoyes
3d35fe2f60 libkas: Update BB_ENV_PASSTHROUGH_ADDITIONS with env_vars
Poky master has renamed BB_ENV_EXTRAWHITE ->
BB_ENV_PASSTHROUGH_ADDITIONS ahead of the kirkstone release
https://git.yoctoproject.org/poky/commit/?id=492214104a55ec56cc483c9b045038e355c6af7b

Update env var logic so that it updates either BB_ENV_EXTRAWHITE (for
honister or earlier) or BB_ENV_PASSTHROUGH_ADDITIONS (for master or
kirkstone), depending on which is defined in the bitbake environment.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-22 19:17:17 +01:00
Jan Kiszka
e3af001f53 libkas: Do not overwrite existing .ssh/config
kas wrongly assumes so far that setting $HOME will also make ssh use the
kas provided $HOME/.ssh/ folder. But ssh will pick up the homedir from
/etc/passwd instead. This could cause kas to overwrite the users
~/.ssh/config when using SSH_PRIVATE_KEY*. We can try to cure ssh config
isolation, but that may cause surprises for users so far silently
relying on it.

For now, as a stable fix, avoid to cause damage to ~/.ssh/config in
cases where this is likely not desired, namely when there is already
config file. Warn if that file does not contain the generated content
from a previous run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-23 18:26:13 +01:00
Jan Kiszka
5fb9067894 Fall back to .config.yaml if no configuration file given
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence

kas menu -> save & exit
kas build
kas shell
kas ...

and also makes rebuilding the self-configured image simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
4c975a04f9 plugins: Invoke setup_parser_common_args explicitly from setup_parser
This allows to define plugins which do not need have any of common args.

Move setup_parser_common_args into libkas for this as it now becomes a
library function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
8a1c27bc92 libkas: Gracefully handle empty repo set
We will fail differently then, but with a nicer error message (missing
init-build-env script) than

ValueError: Set of coroutines/Futures is empty.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 23:17:41 +02:00
Jasper Orschulko
15de0142d4 Add environment variable SSH_PRIVATE_KEY_FILE
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-10 11:25:40 +02:00
Jan Kiszka
87af46ff8c Silence "Exception ignored when trying to write to the signal wakeup fd"
Disable warn_on_full_buffer via set_wakeup_fd. This avoids tons of

    Exception ignored when trying to write to the signal wakeup fd:
    BlockingIOError: [Errno 11] Resource temporarily unavailable

that are currently issued after running complex builds. Only works since
Python 3.7, though, so this remains best-effort. Fortunately, the Debian
in our container image is 3.7.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-02 12:40:19 +01:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Jan Kiszka
79641ac382 libkas: Acount for changes to asyncio.wait
Only tasks should be provided to it, no longer coroutines. Python 3.9
starts to complain about the deprecated pattern.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:23:05 +01:00
Jan Kiszka
204019f54e Remove up obsolete compat code for Python < 3.5
We demand 3.5 since e383e16570.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 08:16:01 +01:00
Paul Barker
1dffb986b1 plugins: Move plugins to dedicated submodule
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:13:06 +02:00
Paul Barker
af4389e606 kas: Simplify plugin lookup
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.

The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:11:49 +02:00
Chris Laplante
9518dc4e35 libkas: make get_build_environ more exception safe
Eliminate manual call to os.remove by using tempfile context manager.

The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:33 +02:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Sam Lewis
b5ba370049 Ensure SSH key ends with newline
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').

This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.

Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:18:54 +01:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Jan Kiszka
7155769f06 libkas: Ensure Python 3.7 compatibility
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.

At this chance, extract the common compat part into a helper function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:51:45 +01:00
Jan Kiszka
ecbda0494b Remove SSH_AGENT_PID forwarding
This variable plays no role in using the agent for authentication, only
in managing its life-cycle.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-17 16:23:46 +02:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Jan Kiszka
bb103365d2 libkas: Remove unused shell mode from run_cmd
No users, and it even seems broken /wrt universal_newlines.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:18:14 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
7937c1963e libkas: Remove parameters from get_build_environ()
All of them can be retrieved from the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:02 +02:00
Jan Kiszka
448107781e libkas: Remove config/context parameters from ssh functions
We can obtain the global context now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:48 +02:00
Jan Kiszka
86dc584284 run_cmd: Default to environment from global context
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:51:34 +02:00
Andreas Reichel
7b18e5ec3b Refactor repo checkout
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.

This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.

This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.

The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).

Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-08-23 16:29:56 +02:00
Daniel Wagner
06f0c20f21 libkas: Disable not-an-itable false positive
pylint has a bug:

https://github.com/PyCQA/pylint/issues/996

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-07-18 16:27:21 +02:00
Claudius Heine
e8851a5fb3 implements patch support for repos
With this commit its now possible to patch 3rd party repos before bitbake is started.

Example:

This is our repo:
  .
  ├── kas.yml
  ├── first-patch.patch
  └── quilt-patches
      ├── second-patch.patch
      ├── third-patch.patch
      └── series

Content of kas.yml:
  header:
    version: 8

  repos:
    my:
    third-party:
      url: "git://example.com/third-party.git"
      refspec: "35adf4...34"
      patches:
        01-first:
          repo: my
          path: "first-patch.patch"
        02-second:
          repo: my
          path: "quilt-patches"

Currently only 'git' repositories can be patched.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:59:07 +01:00
Jan Kiszka
ca6b3c05c9 libkas: Fail run_cmd silently if command is not found
If we only try-execute a command (fail=False), do not bail out with an
exception when starting that command already fails. Catch typical
exceptions and return the corresponding error code along with the
exception string.

This fixes get_root_path in case hg is not installed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-01-25 19:11:43 +01:00
Henning Schild
941fa38a2d repos: move repository functions into repos.py
This commit just moves core around and does not contain any logical
changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
dc8d15c3be git: rename "git_operation_disabled" to "operations_disabled"
Prepare for other version control systems and remove "git" from the
names.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
ce02354627 git: allow an empty refspec even when using version control
If no refspec was given just take the default we get after clone.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 14:26:36 +01:00
Henning Schild
62cf4fb0eb libkas: include another space when extending BB_ENV_EXTRAWHITE
the way the env variable got extended broke the last entry in it and the
first entry in the added string because the two strings where not space
separated

i.e.
  "... no_proxySSTATE_DIR ..."
now you shoud get
  "... no_proxy SSTATE_DIR ..."

Seems that problem was in kas since day one.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Claudius Heine
95015045db implementation of environment variable passthrough
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:

    env:
      VARIABLE_NAME: "default variable value"

Those variables can be overwritten by variables of the environment kas is
started in:

    $ VARIABLE_NAME="overwritten variable value" kas shell ...

Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.

This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-24 10:19:12 +01:00
Daniel Wagner
7aff26d105 libkas: Add /sbin and /usr/sbin to PATH
/sbin and /usr/sbin might contain tools Yocto depens on, e.g. 'ip'.

"""
The following required tools (as specified by HOSTTOOLS) appear to be
unavailable in PATH, please install them in order to proceed:

	ip
"""

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-11-20 14:12:28 +01:00
Henning Schild
d06bdc4781 libkas: do not use absolute paths when executing programs
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2017-09-11 10:19:09 +02:00
Jan Kiszka
2d6b1ce6a0 libkas: Detect and report multiple init scripts
Having more than one them is a sign of serious problems in the
configuration. Better catch that early and fail loudly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:29:54 +02:00
Jan Kiszka
eb284afb8e libkas: Detect error of init script
Do not swallow errors of the init script. Rather leave our wrapper
immediately and let run_cmd report the splash and terminate kas.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:11:54 +02:00
Jan Kiszka
100c7e047f libkas: Pass git proxy related environment variables
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
8e3862112c libkas: Fix false positive of pylint
As suggested by Claudius Heine, reorder the blocks so that we can add
the pylint exception to the if half.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
f58d44f190 libkas: Properly format stderr dump of failing commands
Inject the linefeed that we removed in f0a18868c9 and add an
introduction line.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
264be71ebe libkas: Terminate kas on _repo_fetch_async errors
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.

Drop the return value of repos_fetch - no caller expects any.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:49 +02:00
Jan Kiszka
adb4007482 libkas: Do not sys.exit from a coroutine
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:36 +02:00
Jan Kiszka
f0a18868c9 libkas: Improve further on error/success output
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:28 +02:00