This allows to define plugins which do not need have any of common args.
Move setup_parser_common_args into libkas for this as it now becomes a
library function.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We will fail differently then, but with a nicer error message (missing
init-build-env script) than
ValueError: Set of coroutines/Futures is empty.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.
Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Disable warn_on_full_buffer via set_wakeup_fd. This avoids tons of
Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable
that are currently issued after running complex builds. Only works since
Python 3.7, though, so this remains best-effort. Fortunately, the Debian
in our container image is 3.7.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.
Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Only tasks should be provided to it, no longer coroutines. Python 3.9
starts to complain about the deprecated pattern.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.
The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Eliminate manual call to os.remove by using tempfile context manager.
The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').
This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.
Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.
This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.
At this chance, extract the common compat part into a helper function.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.
This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.
This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.
The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).
Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
With this commit its now possible to patch 3rd party repos before bitbake is started.
Example:
This is our repo:
.
├── kas.yml
├── first-patch.patch
└── quilt-patches
├── second-patch.patch
├── third-patch.patch
└── series
Content of kas.yml:
header:
version: 8
repos:
my:
third-party:
url: "git://example.com/third-party.git"
refspec: "35adf4...34"
patches:
01-first:
repo: my
path: "first-patch.patch"
02-second:
repo: my
path: "quilt-patches"
Currently only 'git' repositories can be patched.
Signed-off-by: Claudius Heine <ch@denx.de>
If we only try-execute a command (fail=False), do not bail out with an
exception when starting that command already fails. Catch typical
exceptions and return the corresponding error code along with the
exception string.
This fixes get_root_path in case hg is not installed.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
the way the env variable got extended broke the last entry in it and the
first entry in the added string because the two strings where not space
separated
i.e.
"... no_proxySSTATE_DIR ..."
now you shoud get
"... no_proxy SSTATE_DIR ..."
Seems that problem was in kas since day one.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:
env:
VARIABLE_NAME: "default variable value"
Those variables can be overwritten by variables of the environment kas is
started in:
$ VARIABLE_NAME="overwritten variable value" kas shell ...
Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.
This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.
Signed-off-by: Claudius Heine <ch@denx.de>
/sbin and /usr/sbin might contain tools Yocto depens on, e.g. 'ip'.
"""
The following required tools (as specified by HOSTTOOLS) appear to be
unavailable in PATH, please install them in order to proceed:
ip
"""
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Having more than one them is a sign of serious problems in the
configuration. Better catch that early and fail loudly.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Do not swallow errors of the init script. Rather leave our wrapper
immediately and let run_cmd report the splash and terminate kas.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As suggested by Claudius Heine, reorder the blocks so that we can add
the pylint exception to the if half.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.
Drop the return value of repos_fetch - no caller expects any.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
With Sphinx it is possible to create the command line documentation
automatically if there is a function that returns just the command line
parser.
Currently the creation of the argument parser is rather entangled with
the rest of kas. This patches seperates this.
Signed-off-by: Claudius Heine <ch@denx.de>
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.
This patch fixes these issues:
************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)
Signed-off-by: Claudius Heine <ch@denx.de>
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.
The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Enhance _stream_subprocess to run_cmd_async, a co-routine variant of
run_cmd that the caller can use to parallelize command execution.
run_cmd becomes a simple wrapper that waits for the async variant to
complete.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Instead of writing "commit", issue a more telling log message if some
repo already contains the required hash.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The two code paths are unfortunate. Factor out the difference and use a
single run_cmd invocation. This also adds the forgotten -q when using
a reference directory.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The current implementation did some more hackish solution to work
around some of kas infrastructure, without changing to much.
This patch cleans this up and therefor remove the more obsure part
of the include mechansim.
Signed-off-by: Claudius Heine <ch@denx.de>
