This allows the `kas build` command to be used when you need to pass
extra arguments to bitbake.
A `--` separator is needed when passing arguments which begin with a
dash so that argparse doesn't try to parse them as kas further
arguments.
For example, to pass `--skip-setscene` to bitbake:
kas build config.yml -- --skip-setscene
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The return value of kas() is used as the argument to sys.exit() so it
should be 0 to report successful completion.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).
That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.
This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.
Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.
Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]
Fixes:
ERROR: OE-core's config sanity checker detected a potential misconfiguration.
Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
Following is the list of potential problems / advisories:
MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.
[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37
Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').
This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.
Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Recent bitbake requires to declare multiconfig targets via "mc:".
Use that chance to refactor the function and match on the colon as well.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If refspec is pointing to an upstream branch name, the checkout for a
second run will not reset a repo back to the upstream commit. That is
because we applied the repo patches to a local branch that carries the
name of the upstream one.
Fix that by switching to local branch prior to applying any patches.
Reported-by: Belisko Marek <marek.belisko@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
patch can not handle file mode headers and can therefore not be used to
create executable files. fixes 8a2a21d799
In addition commit every single patch and treat git patches as
potentially just diffs and not mails.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Adding --cmd to the task argument changed the name of the destination
variable, and that broke the whole subcommand because 'cmd' is already
used by the top-level parser. Fix this by setting dest explicitly.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This aligns us with bitbake. Moreover, -c/--cmd is easier to tell apart
from --target. Keep --task for backward compatibility.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Switch to applying all patches with "patch" and later do a "<vcs> add;
<vcs> commit". Also iterate over quilt series files ourselfs and apply
the patches one by one.
This means we can apply patches on mercurial as well as on git.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The interface cmd prototypes got out of sync so mercurial stopped
working. Align with git again to fix that.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Currently the the remote URL is only set during the initial clone.
This might lead to confusion with the user as changing the URL in the kas.yml
does not get reflected and thus errors about references not found show up.
This change makes sure the URL is always set to the value specified in the
kas.yml file.
For mercurial this currently prints a warning as there is no command to change
the default path.
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
[Jan: fix up style, adjust comment on hg's set_remote_url_cmd]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
There is no good reason to only warn about an incorrect patch entry or
a missing patch file. This just causes CI systems to succeed with a
build - or fail at unexpected places.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
PEP8:
"""
To better support introspection, modules should explicitly declare the
names in their public API using the __all__ attribute.
"""
flake8 reports
./kas/kas.py:51:1: F401 '.build' imported but unused
./kas/kas.py:52:1: F401 '.shell' imported but unused
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
PEP8:
"""
To better support introspection, modules should explicitly declare the
names in their public API using the __all__ attribute.
"""
reported by flake8:
./kas/__init__.py:26:1: F401 '.__version__.__version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__file_version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__compatible_file_version__' imported but unused
./kas/__init__.py:28:1: F401 '.configschema.CONFIGSCHEMA' imported but unused
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.
This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
They were ignored so far is the config file was specifying a target or a
task, respectively. But the proper priority is (in descending order):
1. command line options
2. environment (KAS_TARGET, KAS_TASK)
3. kas config
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.
At this chance, extract the common compat part into a helper function.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allow to use the host-side SSH agent when cloning protected
repositories via kas. We already do this for the build environment.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
Divide the SetupRepos command into three parts: init, step and finish
Build a macro-based loop with the step part of the function.
Rewrite this in build.py and shell.py
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
To keep the macro style of the program flow and explicitely state the
command sequence with Macro.add, we need a mechanism that repeats
several steps of the Macro depending on a given condition. This will be
used by the repo checkout logic.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:
kas build base.yml:board.yml:feature.yml
The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.
In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).
Based on idea by Claudius Heine.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.
Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This helps to avoid circular dependencies. It also allows the Config
constructor to make use of the global context.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This will avoid the increasingly ugly pushing around of a context
instance as parameter to various methods/functions. Everyone can now
simply call get_context() after build or shell called
create_global_context().
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This fixes a regression of 5b85fba0: diff-index is porcelain and seems
to lack some explicit refresh prior to being called. Therefore, it can
incorrectly detect files as being modified although a "git diff" will
not - and will actually also update the status so that the next
diff-index would be fine again.
Fix those false positives while still detecting also cached changes by
using git status -s instead.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The ID_LIKE tag that get_distro_id_base also queries may contain
multiple, space-separated values. Opensuse Leap 15.0 is exploiting that,
e.g. ("suse opensuse").
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.
This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.
This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.
The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).
Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
pylint reports:
kas/includehandler.py:105:0: R0205: Class 'IncludeHandler' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
pycodestyle v2.4.0 checks for W606 (async deprecated), but
we can't disable it on source code level with '# noqa'. Let's disable
the check globaly. We will still catch it with pylint.
By disabling the W606 we need also to list the W503 which was
previously active.
While at it fix following things:
./docs/conf.py:252:6: E121 continuation line under-indented for hanging indent
./docs/conf.py:257:6: E126 continuation line over-indented for hanging indent
./kas/build.py:123:17: W504 line break after binary operator
./kas/configschema.py:48:25: E123 closing bracket does not match indentation of opening bracket's line
./kas/configschema.py:49:21: E123 closing bracket does not match indentation of opening bracket's line
./tests/test_includehandler.py:82:13: E123 closing bracket does not match indentation of opening bracket's line
./tests/test_includehandler.py:121:13: E123 closing bracket does not match indentation of opening bracket's line
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private. The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.
This patch also allows 'get_repos' and 'repo_dict' to be used even in the
'Repo' class. 'get_repo_dict' creates new 'Repo' instances and thus
invalidates the current instance.
This patch is in preparation of the 'patch support'-feature, that
depends on this patch.
Signed-off-by: Claudius Heine <ch@denx.de>
is_dirty_cmd() uses `git diff --shortstat` to see if the repo can be
safely checked out. This only checks for the modified files and does
not take the cached(staged) files into account.
There can be cases where the cached changes could be overwritten by a
checkout. In this case, the git checkout command aborts with an error
message telling the user to commit or stash their changes.
Use `git diff-index HEAD --shortstat` to determine if there are any
changes in the tracked files, whether cached or not.
https://git-scm.com/docs/git-diff-index
Signed-off-by: Abdur Rehman <abdur_rehman@mentor.com>
This reverts commit b9032ec025.
The commit breaks inclusion chains of kas files. In my example a
first kas-file includes a second one. The second one is supposed to
add repos but kas does not fetch those.
Reported-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Copying the .gitconfig failes if the current user doesn't have one,
since the commits that are done within kas, will not be pushed or
merged anywhere, we can just create a minimal .gitconfig in the temp
home directory of kas.
Signed-off-by: Claudius Heine <ch@denx.de>
With this commit its now possible to patch 3rd party repos before bitbake is started.
Example:
This is our repo:
.
├── kas.yml
├── first-patch.patch
└── quilt-patches
├── second-patch.patch
├── third-patch.patch
└── series
Content of kas.yml:
header:
version: 8
repos:
my:
third-party:
url: "git://example.com/third-party.git"
refspec: "35adf4...34"
patches:
01-first:
repo: my
path: "first-patch.patch"
02-second:
repo: my
path: "quilt-patches"
Currently only 'git' repositories can be patched.
Signed-off-by: Claudius Heine <ch@denx.de>
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private. The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.
Signed-off-by: Claudius Heine <ch@denx.de>
Otherwise we cannot overwrite them in mulitconfig files. It's not
expected that anything else in the layer config files sets those
variable, thus shadows our setting in local.conf.
Something changed in latest bitbake to reveal this issue. We first see
it in Isar since bitbake moved to f8126aaf7741.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If we only try-execute a command (fail=False), do not bail out with an
exception when starting that command already fails. Catch typical
exceptions and return the corresponding error code along with the
exception string.
This fixes get_root_path in case hg is not installed.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a repo path is relative, append the kas work dir to it.
Background: The path ends up in bblayers.conf
Without this change the following example will fail:
--------------------------------------------------------
header:
version: 4
machine: qemux86
repos:
poky:
url: "https://git.yoctoproject.org/git/poky"
refspec: "rocko"
path: "thirdparty/poky"
layers:
meta:
meta-poky:
--------------------------------------------------------
Signed-off-by: Georg Lutz <georg@georglutz.de>
This will be used to carry the type of repo we are talking about. At the
moment it defaults to "git".
Signed-off-by: Henning Schild <henning.schild@siemens.com>
The user just calls that factory and gets something that is a Repo, in
fact just a GitRepo for now.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
If we start on a green field (only kas is available) we can't setup
the environment directly. So in this case the order is SetupProxy,
ReposFetch, ReposCheckout and then SetupEnvironment. Only if the
--keep-config-unchanged flag is provided directly go from SetupProxy
to SetupEnvironment.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Forgotten by 49cb9b7a7b: The default is now set in
get_bitbake_targets.
This fixes spurious overwrites of the target defined in the config file.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
pylint reports
R: 52, 4: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
According the Python documentation we should raise AttributeError
if the key is unknown.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
the way the env variable got extended broke the last entry in it and the
first entry in the added string because the two strings where not space
separated
i.e.
"... no_proxySSTATE_DIR ..."
now you shoud get
"... no_proxy SSTATE_DIR ..."
Seems that problem was in kas since day one.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
This could happen if you use kas to checkout repos that do not contain
the directory. i.e. repos that are not actually bb layers
Signed-off-by: Henning Schild <henning.schild@siemens.com>
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:
env:
VARIABLE_NAME: "default variable value"
Those variables can be overwritten by variables of the environment kas is
started in:
$ VARIABLE_NAME="overwritten variable value" kas shell ...
Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.
This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.
Signed-off-by: Claudius Heine <ch@denx.de>
/sbin and /usr/sbin might contain tools Yocto depens on, e.g. 'ip'.
"""
The following required tools (as specified by HOSTTOOLS) appear to be
unavailable in PATH, please install them in order to proceed:
ip
"""
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
This patch adds multiconfig support to kas by defining the
"BBMULTICONFIG" variable in the "local.conf" file.
Signed-off-by: Claudius Heine <ch@denx.de>
Its useful to know the missing repos that are required to
create the full configuration as they are detected.
This might help to find problems within the configuration and kas code
in the future.
Signed-off-by: Claudius Heine <ch@denx.de>
Since its easy to forget to update the documentation when changing the
configuration file format version, this patch adds a small note to
remind the developer about it.
Signed-off-by: Claudius Heine <ch@denx.de>
I was observed that in some more evolved configurations the build
stopped because kas wanted to clone a git repository multiple times
concurrently. This failed.
This patch should prevent these issues by removing all possible
duplicates from the missing_repos list prior to returning it to the
configuration.
Signed-off-by: Claudius Heine <ch@denx.de>
Some bitbake recipes use FTP to fetch objects. This patch adds support
for the ftp_proxy variable to kas. Is now usable in the same manner as
the 'http_proxy' variable
Signed-off-by: Claudius Heine <ch@denx.de>
Using the jsonschema module makes some checks in the includehandler
redundant and can be removed. This patch does this.
Signed-off-by: Claudius Heine <ch@denx.de>
Currently many of the error messages regarding the configuration file
are not really helpful.
To improve reporting this patch adds validation of the configuration
files using the jsonschema module.
Example:
Forgetting ':' at the end of a layer definition, like this:
repos:
meta-iot2000:
layers:
meta-iot2000-example
^
Exception before:
TypeError: string indices must be integers
Exception after:
Validation Error:
'meta-iot2000-example' is not of type 'object'
Failed validating 'type' in schema['properties']['repos']['additionalProperties']['properties']['layers']:
{'additionalProperties': {'oneOf': [{'type': 'null'},
{'type': 'integer'},
{'type': 'boolean'},
{'type': 'string'}]},
'type': 'object'}
On instance['repos']['meta-iot2000']['layers']:
'meta-iot2000-example'
This patch adds 'jsonschema' as an additional dependency.
Signed-off-by: Claudius Heine <ch@denx.de>
With this patch support for building multiple targets in parallel with
bitbake is now supported.
Changes:
- Its now possible to state a list of targets under the target key in
the configuration file.
Example:
target:
- product-image
- product-update-image
- Its now possible to define multiple targets in the kas command line.
Example:
$ kas build --target product-image --target product-update-image \
kas.yml
- Its now possible to define multiple targets via the environment:
Example:
$ export KAS_TARGET="product-image product-update-image"
$ kas build kas.yml
Signed-off-by: Claudius Heine <ch@denx.de>
It might be useful to see the merged configuration in case of debugging.
This patch adds it to the debug output log level.
Signed-off-by: Claudius Heine <ch@denx.de>
Captain Hindsight: We should have never committed 13a87cb8c7 ("Allow
to define task in config and environment") without forwarding the file
version. Without this, no config file writer can tell kas versions
apart that support task control from those that don't.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
9e136e532d ("config: Remove dynamic configuration variant")
removed the posibility to add pre and post hooks. Remove the remaining
bits from libcmds.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Now that we no longer have two different configuration variants, we can
remove the abstract class and make Config the primary interface.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We no longer see an urging use case for it. All field scenarios can
perfectly be modeled with the static format.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
This patch adds a command line argument to the shell plugin, that allows to
enter the shell without changing the bitbake configuration.
Signed-off-by: Claudius Heine <ch@denx.de>
Before this patch, starting the shell in a clean environment, does not
checkout all repository or configures all layers correctly.
This patch adds the missing checkout and configuration steps to the
shell plugin.
Signed-off-by: Claudius Heine <ch@denx.de>
Having more than one them is a sign of serious problems in the
configuration. Better catch that early and fail loudly.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Do not swallow errors of the init script. Rather leave our wrapper
immediately and let run_cmd report the splash and terminate kas.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Instead of flooding the log with misleading error messages, always print
a note which path is used as root for repositories that have no URLs.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If the repo, where the configuration file resides in (in-tree
configuration) is not a git repository, take the directory where the
configuration file is located at as the path of this repo instead of failing.
This patch does not need to raise the configuration file version,
because it does not depend on the configuration file.
Signed-off-by: Claudius Heine <ch@denx.de>
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Sort the header sections by their keys and the layers by their names.
This will re-enable writing deterministic config files, and bitbake can
use its cache again, accelerating rebuild startup times a lot.
While at it, indent the layers in bblayers.conf nicely.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Use a separately incremented integer to track the configuration file
format version. We start with 2 due to the change that 23c3a951f6
introduced. 1 is declared to be equivalent to the original '0.10'.
The separate versioning has the advantage of being able to increment it
already during the development cycle, and using it with config files
that test/exploit the new format.
Using an integer has the advantages of a) differentiating it clearly
from the now independent kas version and b) simplifying the version
parsing. We can now also remove the string type restriction.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When a configuration file depends on a include from a repo, which is
defined in another included file, a 'KeyError' occures.
This fixes this, so that only known repos are tried to be fetched.
Signed-off-by: Claudius Heine <ch@denx.de>
This exception is always thrown with both a message and a file name.
Combine them to a single string, instead of throwing tuples.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
ConfigStatic does not respect the --target overwrite so far. Fix this
by pulling the default definition into ConfigPython (ConfigStatic
already does that via the Config base class) and set the target provided
via the command line in the static config after parsing all files.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
As suggested by Claudius Heine, reorder the blocks so that we can add
the pylint exception to the if half.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The `load_config` function in the includehandler already checks for the
file extension, so checking it before that, is redundant.
Signed-off-by: Claudius Heine <ch@denx.de>
The application should not be stopped from this code, removed `sys.exit`
and added exceptions to improve error handling.
Also added some more checks that were missing.
Signed-off-by: Claudius Heine <ch@denx.de>
In case of ConfigStatic, proxy settings only come from the host
environment. We need them early, prior to fetching the first missing
repo that is pulled via an include.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
For me 'distro.id()' returned 'ubuntu' and thats not equal to 'Ubuntu'.
This patch converts the distro id to lowercase before comparing it.
Signed-off-by: Claudius Heine <ch@denx.de>
This seems to be the cleanest and simplest way of properly terminating
kas while one or more sub-processes are running: just ignore the signal.
The sub-processes will get it as well, and they will terminate
themselves properly. kas will get an error then and shut down. Problem
solved.
Forwarding the signal didn't work reliably anyway because we couldn't
count on kas being a process group leader in all cases.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.
Drop the return value of repos_fetch - no caller expects any.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make the header sections in local.conf and bblayers.conf easier
identifiable by naming them and writing them in chunks, separated by
blank lines. This can help with debugging the resulting configuration.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Another, hopefully better attempt: Ignore the TERM signal in the sending
process so that we can properly clean up all the futures before actually
closing. This also avoids a "Terminated" message even on regular kas
endings.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Documentation for the 'build' and 'shell' command line were missing.
Also fixed kas command line documentation mentioning 'ebs-yocto'.
Signed-off-by: Claudius Heine <ch@denx.de>
With Sphinx it is possible to create the command line documentation
automatically if there is a function that returns just the command line
parser.
Currently the creation of the argument parser is rather entangled with
the rest of kas. This patches seperates this.
Signed-off-by: Claudius Heine <ch@denx.de>
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.
This patch fixes these issues:
************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)
Signed-off-by: Claudius Heine <ch@denx.de>
Changing even simple settings like target or machine requires the
creation of additional configuration files. This does not scale well.
This patch allows the target, machine and distro to be overwritten by
environment variables (called KAS_TARGET, KAS_MACHINE and KAS_DISTRO
respectively).
It also fixes how the environment variables for the proxy settings are
handled. Currently the settings in the config files overwrite the proxy
settings from the environment, but since that would be inconsistent this
patch switches that around. With this patch the environment overwrites
the proxy settings in the configuration file.
Signed-off-by: Claudius Heine <ch@denx.de>
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.
The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Make it more straightforward by removing complete variable and
terminating the loop on the condition - no more missing repos -
directly. Rename variables to clarify what they reference.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Enhance _stream_subprocess to run_cmd_async, a co-routine variant of
run_cmd that the caller can use to parallelize command execution.
run_cmd becomes a simple wrapper that waits for the async variant to
complete.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Instead of writing "commit", issue a more telling log message if some
repo already contains the required hash.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The two code paths are unfortunate. Factor out the difference and use a
single run_cmd invocation. This also adds the forgotten -q when using
a reference directory.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The current implementation did some more hackish solution to work
around some of kas infrastructure, without changing to much.
This patch cleans this up and therefor remove the more obsure part
of the include mechansim.
Signed-off-by: Claudius Heine <ch@denx.de>
Isar allows to build Debian-based images using bitbake and the usual
layer structures. It's very similar to OE, we just need to account for
a different init script name.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If the configuration file structure changes over time, it might
introduce incompatibilities.
This patch introduces versioning of configuration file.
Configuration files needs to add a 'version' entry into their 'header'
like this:
header:
version: '0.9'
After loading this file, the version is checked agains kas version.
Every version 'M.m.p' of kas is allowed to load configuration files with
version 'M.m' and all versions backwards up to and including
the '__compatible_version__' from kas/__version__.py
Signed-off-by: Claudius Heine <ch@denx.de>
In the current configuration file we mix between entries that are file
specific (like includes and later version) and entries that are used by
the rest of kas, that are not file specific (machine, target, repos,
etc.) It might not be obvious to differenciate between those entries.
This patch introduces a header section into the configuration file, to
contain every setting that is specific to the file and not the complete kas
configuration. The 'includes' statement is moved there, because its file
specific.
Signed-off-by: Claudius Heine <ch@denx.de>
Splitting configuration files into multiple files and implementing an
include mechanism allows to handle configurations more flexible and
allowing following scenarios:
- including kas configuration file from external meta layer into
own project
- splitting many different similar configurations into multiple files
with only a couple of common files
To include file its necessary to add a 'includes' entry into the kas
configuration. To include files from the same repo, do it like this:
-----
includes:
- ../base/base_include.yml
-----
The path is relative to the current configuration file. If they start
with a path seperator ("/") they are absolute.
To include files from a different repository, do it like this:
-----
includes:
- file: bsps/my_machine_include.yml
repo: collected_machines
repos:
collected_machines:
url: https://url.to.git.repo/
revspec: master
-----
You have to make sure that the repository definition is available
in your current file, or in any include that is available at this
point.
Yaml ("*.yml") and Json ("*.json") files are supported.
Included in this patch are also some changes to the configuration
file structure. Here is an overview:
The value of the 'repos' key is a dictionary that maps a repo identifier
to a dictionary that contains 5 entries:
- url: The url to the repository. If that is missing, no git operations
are used
- refspec: The git refspec of the repository that is used. If that is
missing the latest commit of the default branch is used.
- name: The is defines under which name the repository is stored. If
that is missing the repository identifier is used
- path: The path where the repository is stored. If no url is given and
a path is missing, the repository is referencing the repository under
which the configuration file is stored. If no url is given and a path is
specified, the repository is referencing a directory where layers might
be stored. If an url is specified, path can be used to overwrite the
default (kas_work_dir + repo.name) checkout directory.
- layers: This contains a dictionary of layers, that should be included
into the bblayers.conf. The keys are paths relative to the repository
and the values can be used to exclude layers if they are one of
"excluded", "disabled", "false", "0", "n", or "no". Also boolean values
are accepted. Any other value, including "None" means that this layer is
included into the bblayers.conf. If "layers" is missing or empty, the
repository itself is included into the bblayers. If this is specified, the
repository itself is not included into the bblayers.conf.
Signed-off-by: Claudius Heine <ch@denx.de>
It was decided that 'sublayers' is to confusing, because this term is
not used in the bitbake/openembedded context.
Signed-off-by: Claudius Heine <ch@denx.de>
Config.has_changed didn't work reliably so far, so the state cache was
only allowing to associate a work directory with configuration file.
However this was rather unintuitive, specifically when moving the work
dir to a different home - nothing told the user about the persistent
association.
We therefore agreed to remove the state cache for now. It can be
reintroduced at any time once a consistent usage model exists.
To establish a relationship to a work directory inside a shell session,
kas now evaluates the KAS_WORK_DIR environment variable.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This shortens the include from "from .__version__ import __version__"
to "from . import __version__" within the module.
Signed-off-by: Claudius Heine <ch@denx.de>
Currently the shell provided via the 'shell' command is always '/bin/sh'
and does not have a TERM variable set. This is a rather spartan shell
experience.
With this patch we add the TERM variable to the whitelist, so that
pagers work better and made the shell binary dependent from the SHELL
environment variable. If no such variable is set, we fall back to using
'/bin/sh'.
Signed-off-by: Claudius Heine <ch@denx.de>
The ssh-agent environment variables of an ssh-agent that is running
outside the kas build system are not transfered to the bitbake
environment. This causes fetching of private git repositories to
fail.
This patch adds those variables to the environment whitelist.
Signed-off-by: Claudius Heine <ch@denx.de>
