When using the kas menu command, we need to track the repository
location outside of the kas container as well. This is required to
prepare the environment for a later kas-container build, as the repo
path inside the container is different from the outside one. For that,
the location of KAS_REPO_DIR is passed via an env-var into the container.
There, this value is picked up by the menu plugin and written to the
.config.yaml file as `_source_host_dir`. When running kas-container build,
we check if the .config.yaml file contains this node and set the
KAS_REPO_DIR accordingly. The schema is extended accordingly.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: rework _source_dir_host extraction to make it usable for menu as well]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a new top-level schema node `_source_dir` which is used
to track the top repo dir on auto-generated kas files. This node is
generated by the kas menu plugin and provides the absolute path to the
top repo dir at time of invoking the plugin. When later calling any other
kas operation that performs a checkout, this node is evaluated and the
top repo dir is set accordingly. This tracking is required when the
build command is executed from another dir than the top repo dir, as the
repo-dir cannot be computed by kas in this case (the .config.yaml file
is in the KAS_WORK_DIR, not in the repo dir).
Proposed-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: aligned format-changelog text, simplified load_config returning]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The common case that an include file cannot be found is now wrapped into
a LoadConfigException. By that, the user gets a more meaningful error
message on the console (instead of a stacktrace).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds the KasUserError exception class to distinguish between
internal kas exceptions and user or configuration errors. Exceptions
previously raised on user errors are ported over by deriving
KasUserError. In case of user errors, only the exception message is
shown, but no stacktrace. This makes it easier for users to locate the
issue as the reason is now stated in the last line of the output.
Kas internal exceptions are not subject to this change to help the
developers to find the root cause more easily.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When checking out repositories, check if a file <filename>.lock.<ext>
exists next to the file specified first on the kas CLI. In case this
file exists and the --update option is not specified, automatically
append this file to the kas CLI before performing any other kas
operations.
When --update is specified, the lockfile is ignored.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Since 2077900b4e, the existence of a repo key is enforced by the
config schema. So we can drop the redundant check in the include
handler.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Commit 2077900b4e was wrong in claiming that the documentation already
demanded repo-relative paths for local includes. It only did that for
dictionary-based include entries. Aligning both to repo-relative paths
remain a value, but we definitely have to update the documentation now.
While at it, also update the code comments accordingly. That also still
talked about the repo key being option, something that was never the
case in practice.
Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.
Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.
As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.
Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If command line includes refer to the same repo, missing_repos was
aggregating those multiple times, causing multiple clone procedures to
be triggered. This generally failed. Therefore, build missing_repos
while checking for duplicates.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
.yaml is the recommended extension (https://yaml.org/faq.html)
Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.
This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:
kas build base.yml:board.yml:feature.yml
The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.
In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).
Based on idea by Claudius Heine.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
pylint reports:
kas/includehandler.py:105:0: R0205: Class 'IncludeHandler' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
I was observed that in some more evolved configurations the build
stopped because kas wanted to clone a git repository multiple times
concurrently. This failed.
This patch should prevent these issues by removing all possible
duplicates from the missing_repos list prior to returning it to the
configuration.
Signed-off-by: Claudius Heine <ch@denx.de>
Using the jsonschema module makes some checks in the includehandler
redundant and can be removed. This patch does this.
Signed-off-by: Claudius Heine <ch@denx.de>
Currently many of the error messages regarding the configuration file
are not really helpful.
To improve reporting this patch adds validation of the configuration
files using the jsonschema module.
Example:
Forgetting ':' at the end of a layer definition, like this:
repos:
meta-iot2000:
layers:
meta-iot2000-example
^
Exception before:
TypeError: string indices must be integers
Exception after:
Validation Error:
'meta-iot2000-example' is not of type 'object'
Failed validating 'type' in schema['properties']['repos']['additionalProperties']['properties']['layers']:
{'additionalProperties': {'oneOf': [{'type': 'null'},
{'type': 'integer'},
{'type': 'boolean'},
{'type': 'string'}]},
'type': 'object'}
On instance['repos']['meta-iot2000']['layers']:
'meta-iot2000-example'
This patch adds 'jsonschema' as an additional dependency.
Signed-off-by: Claudius Heine <ch@denx.de>
Use a separately incremented integer to track the configuration file
format version. We start with 2 due to the change that 23c3a951f6
introduced. 1 is declared to be equivalent to the original '0.10'.
The separate versioning has the advantage of being able to increment it
already during the development cycle, and using it with config files
that test/exploit the new format.
Using an integer has the advantages of a) differentiating it clearly
from the now independent kas version and b) simplifying the version
parsing. We can now also remove the string type restriction.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This exception is always thrown with both a message and a file name.
Combine them to a single string, instead of throwing tuples.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The application should not be stopped from this code, removed `sys.exit`
and added exceptions to improve error handling.
Also added some more checks that were missing.
Signed-off-by: Claudius Heine <ch@denx.de>
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.
This patch fixes these issues:
************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)
Signed-off-by: Claudius Heine <ch@denx.de>
If the configuration file structure changes over time, it might
introduce incompatibilities.
This patch introduces versioning of configuration file.
Configuration files needs to add a 'version' entry into their 'header'
like this:
header:
version: '0.9'
After loading this file, the version is checked agains kas version.
Every version 'M.m.p' of kas is allowed to load configuration files with
version 'M.m' and all versions backwards up to and including
the '__compatible_version__' from kas/__version__.py
Signed-off-by: Claudius Heine <ch@denx.de>
In the current configuration file we mix between entries that are file
specific (like includes and later version) and entries that are used by
the rest of kas, that are not file specific (machine, target, repos,
etc.) It might not be obvious to differenciate between those entries.
This patch introduces a header section into the configuration file, to
contain every setting that is specific to the file and not the complete kas
configuration. The 'includes' statement is moved there, because its file
specific.
Signed-off-by: Claudius Heine <ch@denx.de>
Splitting configuration files into multiple files and implementing an
include mechanism allows to handle configurations more flexible and
allowing following scenarios:
- including kas configuration file from external meta layer into
own project
- splitting many different similar configurations into multiple files
with only a couple of common files
To include file its necessary to add a 'includes' entry into the kas
configuration. To include files from the same repo, do it like this:
-----
includes:
- ../base/base_include.yml
-----
The path is relative to the current configuration file. If they start
with a path seperator ("/") they are absolute.
To include files from a different repository, do it like this:
-----
includes:
- file: bsps/my_machine_include.yml
repo: collected_machines
repos:
collected_machines:
url: https://url.to.git.repo/
revspec: master
-----
You have to make sure that the repository definition is available
in your current file, or in any include that is available at this
point.
Yaml ("*.yml") and Json ("*.json") files are supported.
Included in this patch are also some changes to the configuration
file structure. Here is an overview:
The value of the 'repos' key is a dictionary that maps a repo identifier
to a dictionary that contains 5 entries:
- url: The url to the repository. If that is missing, no git operations
are used
- refspec: The git refspec of the repository that is used. If that is
missing the latest commit of the default branch is used.
- name: The is defines under which name the repository is stored. If
that is missing the repository identifier is used
- path: The path where the repository is stored. If no url is given and
a path is missing, the repository is referencing the repository under
which the configuration file is stored. If no url is given and a path is
specified, the repository is referencing a directory where layers might
be stored. If an url is specified, path can be used to overwrite the
default (kas_work_dir + repo.name) checkout directory.
- layers: This contains a dictionary of layers, that should be included
into the bblayers.conf. The keys are paths relative to the repository
and the values can be used to exclude layers if they are one of
"excluded", "disabled", "false", "0", "n", or "no". Also boolean values
are accepted. Any other value, including "None" means that this layer is
included into the bblayers.conf. If "layers" is missing or empty, the
repository itself is included into the bblayers. If this is specified, the
repository itself is not included into the bblayers.conf.
Signed-off-by: Claudius Heine <ch@denx.de>
