Commit Graph

68 Commits

Author SHA1 Message Date
Felix Moessbauer
3e0dd10416 add support to override refspec of repos
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.

Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:04:51 +02:00
Johann Neuhauser
bce8a916fa config: Symplify .config.yaml file handling
If no configuration file is specified, we try to load a single
.config.yaml from KAS_WORK_DIR and set the top_repo_path to the
repo root of this file with fallback to the containing directory.

This process is identical to loading explicitly specified
configuration files and can therefore be combined into one case.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:55 +02:00
Johann Neuhauser
cbdba7e8e1 config: Fix repo-relative include file handling if no config file is given
If we do not specify a configuration file for the "kas build" command,
the repository anchor is set to the current working directory,
which breaks the resolution of repo-relative include file paths
in the IncludeHandler class if repo root != current working directory.

Sets the repository anchor to the root path of the repository with
fallback to KAS_WORK_DIR if we use .config.yaml from KAS_WORK_DIR.

Signed-off-by: Johann Neuhauser <jneuhauser@dh-electronics.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-08-03 13:00:49 +02:00
Peter Hoyes
958b0ab2f8 config: Public interface amendments
Add a repo_paths argument to find_missing_repos, with the default value
unchanged.

Factor out the contents of the loop in _get_repo_dict in config.py to
get_repo so that Repo instances can be created one at a time. The
behavior of _get_repo_dict is unchanged.

Add get_repos_config so it is possible to check whether a repo name
exists just in the config, as opposed to whether a Repo instance has
been created.

Add get_config to allow the config dict to be read in its entirety for
debugging purposes.

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
[Jan: fix minor style issue]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-07-12 09:20:51 +02:00
Jan Kiszka
5fb9067894 Fall back to .config.yaml if no configuration file given
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence

kas menu -> save & exit
kas build
kas shell
kas ...

and also makes rebuilding the self-configured image simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
eeab1bc9bc config: Pass context to Config constructor
Will be used later on to retrieve kas_work_dir.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
2077900b4e includehandler: Fix repo path detection for local includes
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.

Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.

As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:14 +02:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Paul Barker
c2e62df73f config: Make target argument optional in Config()
The target argument is only used when initializing the build plugin so
we can make it optional. This allows us to simplify the call to Config()
in the shell plugin and in other future plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:47:05 +01:00
Mert Kirpici
e79910aa29 config: avoid duplicates in BBMULTICONFIG
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-02 09:10:07 +01:00
Chris Laplante
22f7c0b03d config: Make common repository check a bit more Pythonic
Shortens code and makes the intent clearer.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:25 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
230c5a9572 Add support for new multiconfig syntax
Recent bitbake requires to declare multiconfig targets via "mc:".

Use that chance to refactor the function and match on the colon as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-11-08 10:36:08 +01:00
Jan Kiszka
b2af1bcfe7 config: Remove dead code
Not used anywhere, dates back to dynamic configs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-30 09:41:46 +02:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Jan Kiszka
fe344f93f1 Fix --target and --task options
They were ignored so far is the config file was specifying a target or a
task, respectively. But the proper priority is (in descending order):

1. command line options
2. environment (KAS_TARGET, KAS_TASK)
3. kas config

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 17:24:36 +01:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Jan Kiszka
d4a615bb0b Command line based configuration file merge
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:

kas build base.yml:board.yml:feature.yml

The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.

In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).

Based on idea by Claudius Heine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:32 +02:00
Jan Kiszka
52fdc4f4ae config: Remove context field
No longer used.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:18 +02:00
Jan Kiszka
4acde53b94 Repo: Rework factory() interface
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.

Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:08 +02:00
Jan Kiszka
3fccd1ad3e Move get_distro_id_base from config to context
This is where it is used, only.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:59 +02:00
Jan Kiszka
5c28272d73 includehandler: Fold GlobalIncludes into IncludeHandler class
No user for this abstraction in sight.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:08 +02:00
Andreas Reichel
7b18e5ec3b Refactor repo checkout
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.

This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.

This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.

The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).

Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-08-23 16:29:56 +02:00
Claudius Heine
6afd92eabc kas/config: Make 'get_repo_dict' private
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private.  The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.

This patch also allows 'get_repos' and 'repo_dict' to be used even in the
'Repo' class. 'get_repo_dict' creates new 'Repo' instances and thus
invalidates the current instance.

This patch is in preparation of the 'patch support'-feature, that
depends on this patch.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-04-04 13:43:37 +02:00
Daniel Wagner
4be43c7bf6 Revert "kas/config: Make 'get_repo_dict' private"
This reverts commit b9032ec025.

The commit breaks inclusion chains of kas files.  In my example a
first kas-file includes a second one. The second one is supposed to
add repos but kas does not fetch those.

Reported-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-03-26 14:49:06 +02:00
Claudius Heine
b9032ec025 kas/config: Make 'get_repo_dict' private
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private.  The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:55:43 +01:00
Henning Schild
6a85296836 repos: move more of the complexity from config to Repo.factory
This commit just moved some code around, to a place where it seems to
fit better.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
0c7db599fd repos: introduce a new repo conf parameter "type"
This will be used to carry the type of repo we are talking about. At the
moment it defaults to "git".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
9da82dba13 config: call Repo.factory only in one place
This patch simplifies the code a bit and does not change any semantics.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
3594ad50a0 repos: introduce a factory to abstract away from Repo implementation
The user just calls that factory and gets something that is a Repo, in
fact just a GitRepo for now.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
b5b2766b24 repos: also move version control call out of config.py
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
941fa38a2d repos: move repository functions into repos.py
This commit just moves core around and does not contain any logical
changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
dc8d15c3be git: rename "git_operation_disabled" to "operations_disabled"
Prepare for other version control systems and remove "git" from the
names.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
19e518cd25 config: support "gentoo" as a known distro
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Claudius Heine
95015045db implementation of environment variable passthrough
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:

    env:
      VARIABLE_NAME: "default variable value"

Those variables can be overwritten by variables of the environment kas is
started in:

    $ VARIABLE_NAME="overwritten variable value" kas shell ...

Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.

This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-24 10:19:12 +01:00
Cedric Hombourger
06b05a35b6 config: support major distro variants
Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
2017-11-09 09:26:43 +01:00
Claudius Heine
fa1575790f multiconfig: implement initial support
This patch adds multiconfig support to kas by defining the
"BBMULTICONFIG" variable in the "local.conf" file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-08 09:42:56 +01:00
Claudius Heine
aa79374f7e config: debug message for list of missing repo names
Its useful to know the missing repos that are required to
create the full configuration as they are detected.

This might help to find problems within the configuration and kas code
in the future.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-19 17:15:26 +02:00
Claudius Heine
845d212f02 Added 'ftp_proxy' environment variable support
Some bitbake recipes use FTP to fetch objects. This patch adds support
for the ftp_proxy variable to kas. Is now usable in the same manner as
the 'http_proxy' variable

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-16 11:03:07 +02:00
Claudius Heine
ab8e4cd375 config: Implemented multi-target support
With this patch support for building multiple targets in parallel with
bitbake is now supported.

Changes:

  - Its now possible to state a list of targets under the target key in
    the configuration file.

    Example:

        target:
          - product-image
          - product-update-image

  - Its now possible to define multiple targets in the kas command line.

    Example:

        $ kas build --target product-image --target product-update-image \
          kas.yml

  - Its now possible to define multiple targets via the environment:

    Example:

        $ export KAS_TARGET="product-image product-update-image"
        $ kas build kas.yml

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:00 +02:00
Claudius Heine
f7d2e5ae20 config: added debug output of complete configuration
It might be useful to see the merged configuration in case of debugging.
This patch adds it to the debug output log level.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:10:47 +02:00
Daniel Wagner
133f1be38a libcmds: Remove unused pre and post hooks
9e136e532d ("config: Remove dynamic configuration variant")
removed the posibility to add pre and post hooks. Remove the remaining
bits from libcmds.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-09-27 10:49:35 +02:00
Jan Kiszka
bfd7770b81 config: Fold ConfigStatic into Config
Now that we no longer have two different configuration variants, we can
remove the abstract class and make Config the primary interface.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-27 10:48:59 +02:00
Jan Kiszka
9e136e532d config: Remove dynamic configuration variant
We no longer see an urging use case for it. All field scenarios can
perfectly be modeled with the static format.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-27 10:48:35 +02:00
Jan Kiszka
13a87cb8c7 Allow to define task in config and environment
Based on original patch by Sascha Weisenberger.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-25 09:33:43 +02:00
Henning Schild
d06bdc4781 libkas: do not use absolute paths when executing programs
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2017-09-11 10:19:09 +02:00
Jan Kiszka
bb853cec02 config: Write proper info message about root of URL-less repos
Instead of flooding the log with misleading error messages, always print
a note which path is used as root for repositories that have no URLs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-21 15:34:20 +02:00
Claudius Heine
96034e2d9a config: Allow in-tree repos not to be in a git repo.
If the repo, where the configuration file resides in (in-tree
configuration) is not a git repository, take the directory where the
configuration file is located at as the path of this repo instead of failing.

This patch does not need to raise the configuration file version,
because it does not depend on the configuration file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-20 16:08:34 +02:00
Jan Kiszka
c9326cc1ed Write deterministic local.conf and bblayers.conf
Sort the header sections by their keys and the layers by their names.
This will re-enable writing deterministic config files, and bitbake can
use its cache again, accelerating rebuild startup times a lot.

While at it, indent the layers in bblayers.conf nicely.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:32:06 +02:00