Commit Graph

48 Commits

Author SHA1 Message Date
Felix Moessbauer
3e0dd10416 add support to override refspec of repos
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.

Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:04:51 +02:00
Jan Kiszka
795ecdbcd6 doc: Fix description of bblayers_conf_header and local_conf_header
For a long time already (c9326cc1ed), the documentation suggests a
different ordering of entries generated by *_conf_header than we
actually create. Finally fix this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-01 08:57:07 +01:00
Felix Moessbauer
5b3e238128 add plugin to dump flattened config
This patch adds a new plugin 'dump' which resolves all kas config
includes and combines them in a single file. The format of the output
file can either be JSON or YAML. In case of YAML, the generated config
fulfills the kas-config schema and can be used as input to kas again.
In addition, the generated files can be used by external tools to
analyse the dependencies and versions of the project.

While the generated configuration is semantically identical to the
input config, we do not guarantee binary compatibility as especially
YAML provides multiple ways to serialize strings and null values.
Not giving this guarantee makes it easier to evolve and maintain the plugin.

The plugin itself extends the checkout plugin, as referenced repos have
to be checked-out first to resolve cross-repo references. This also
requires, that the declared refspec of all referenced repos can actually
be found and checked out.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:38:41 +01:00
Benedikt Niedermayr
c38517df83 add conditional environment variables
Until now all environment variables in the kas config's
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt.niedermayr@siemens.com>
[Jan: fixed overlong line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-11-10 19:03:48 +01:00
Claudius Heine
d7af4dc678 docs/userguide.rst: clarify local file include paths
How relative paths in local file includes are handles is more complex
than this one line makes out to be.

This change clarifies the documentation here.

Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: dropped duplicate blank at beginning of sentence]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-10 12:42:50 +01:00
Quirin Gylstorff
1dea24145e docs/userguide.rst: Add recommendation for repo-id naming
This is added to avoid confusion in the creation of downstream layers.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
[Jan: fixed over-long line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-12-01 19:06:38 +01:00
Tobias Schmidl
6044d85fb8 docs: Extended "layers" section in the user guide.
Adding `.` was not intuitively for me, therefore an explicit comment on
how to add both the repo and a subdir might be useful.

Signed-off-by: Tobias Schmidl <tobias@schmidl.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-24 13:44:17 +01:00
Jan Kiszka
4f23045ce8 Add kconfiglib-based menu plugin
This uses the kconfiglib python module to implement simple interactive
configuration for complex layers with many kas options and even variable
input. See the menu.py documentation header for details on the Kconfig
format and its translation to kas.

Rather than using the limited menuconfig implementation of kconfiglib,
this comes with an own, python-newt based version. The permits smooth
integration in the workflow, e.g. to offer an option that directly
triggers a build after completing a configuration. And it comes with a
nicer layout.

Unfortunately, python-newt can't be fulfilled via pip. But it is
available as proper package via common distros. So warn if the package
is missing.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
50e2d2d0fa Align include documentation to new logic
Commit 2077900b4e was wrong in claiming that the documentation already
demanded repo-relative paths for local includes. It only did that for
dictionary-based include entries. Aligning both to repo-relative paths
remain a value, but we definitely have to update the documentation now.

While at it, also update the code comments accordingly. That also still
talked about the repo key being option, something that was never the
case in practice.

Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-01 11:16:28 +02:00
Jan Kiszka
635b1023a1 Declare proxy_config obsolete
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.

Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-12 08:03:40 +02:00
Jan Kiszka
03da3f9436 docs: Improve link to container images
We were linking version 2.2 here.

Reported-by: Vivien Didelot <vivien.didelot@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-19 16:24:25 +01:00
Peter Hatina
9d1d2adbd5 docs: Fix typo in excluded layers example
Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-12 11:18:24 +01:00
Jan Kiszka
6110863a0c userguide: Fix typo
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-18 19:42:10 +01:00
Jan Kiszka
b36b2dbedd Rename forallrepos plugin to for_all_repos
Comes closer to how the command is spelled. We can't uses dashes as that
is not compatible with the python syntax.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 18:26:12 +01:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Paul Barker
a8b5c79326 checkout: New plugin
This plugin checks out all necessary repositories and sets up the build
directory as specificed in the configuration file.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:57 +01:00
Paul Barker
baacd3a8b9 for-all-repos: New plugin
This plugin runs a specified command in all checked out repositories.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 21:00:51 +01:00
Paul Barker
4e153a2872 plugins: Improve documentation of existing modules
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.

Also fix a typo in the build command help message.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:45:07 +01:00
Paul Barker
6bc2627616 docs: Move command line details to a separate page
This change makes the user guide page more readable and results in a
command line usage page which is easy to navigate as all sub-commands
can be found in the left-hand outline panel.

Also fix the layout of the environment variables table by using a
replacement.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:41:57 +01:00
Jan Kiszka
29bbf059a5 userguide: Switch to kas-container
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-29 08:36:42 +02:00
Jan Kiszka
e4be1a20ff kas-docker, docs: Switch to github container registry
All images are now also available on github's new container registry.
Refer to that as default, both in kas-docker as well as in the user
manual. The latter can benefit from some refreshment at this chance.

Automatic deployment via CI will be switched separately.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-07 13:36:43 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Paul Barker
d81c794f30 docs: Drop reference to Python 2
Python 2 is dead, let's not encourage people to add support for it.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:48 +02:00
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
2f7650bb05 Add KAS_PREMIRRORS support
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:35 +01:00
Maciej Pijanowski
4d09e411ef docs/userguide.rst: fix typos, improve grammar
Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
2019-01-17 15:17:14 +01:00
Jan Kiszka
ecbda0494b Remove SSH_AGENT_PID forwarding
This variable plays no role in using the agent for authentication, only
in managing its life-cycle.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-17 16:23:46 +02:00
Jan Kiszka
d4a615bb0b Command line based configuration file merge
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:

kas build base.yml:board.yml:feature.yml

The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.

In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).

Based on idea by Claudius Heine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:32 +02:00
Jan Kiszka
e1cc66510c docs: Fix typo in KAS_REPO_REF_DIR description
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-23 16:45:22 +02:00
Claudius Heine
e8851a5fb3 implements patch support for repos
With this commit its now possible to patch 3rd party repos before bitbake is started.

Example:

This is our repo:
  .
  ├── kas.yml
  ├── first-patch.patch
  └── quilt-patches
      ├── second-patch.patch
      ├── third-patch.patch
      └── series

Content of kas.yml:
  header:
    version: 8

  repos:
    my:
    third-party:
      url: "git://example.com/third-party.git"
      refspec: "35adf4...34"
      patches:
        01-first:
          repo: my
          path: "first-patch.patch"
        02-second:
          repo: my
          path: "quilt-patches"

Currently only 'git' repositories can be patched.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:59:07 +01:00
Georg Lutz
7fa75d94c0 repos: Fix relative path for repos
If a repo path is relative, append the kas work dir to it.

Background: The path ends up in bblayers.conf

Without this change the following example will fail:

--------------------------------------------------------
header:
     version: 4
machine: qemux86
repos:
     poky:
         url: "https://git.yoctoproject.org/git/poky"
         refspec: "rocko"
         path: "thirdparty/poky"
         layers:
             meta:
             meta-poky:
--------------------------------------------------------

Signed-off-by: Georg Lutz <georg@georglutz.de>
2018-01-25 18:54:55 +01:00
Henning Schild
db25611977 docs: update docs accordning to previous changes
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
ce02354627 git: allow an empty refspec even when using version control
If no refspec was given just take the default we get after clone.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 14:26:36 +01:00
Claudius Heine
95015045db implementation of environment variable passthrough
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:

    env:
      VARIABLE_NAME: "default variable value"

Those variables can be overwritten by variables of the environment kas is
started in:

    $ VARIABLE_NAME="overwritten variable value" kas shell ...

Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.

This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-24 10:19:12 +01:00
Claudius Heine
845d212f02 Added 'ftp_proxy' environment variable support
Some bitbake recipes use FTP to fetch objects. This patch adds support
for the ftp_proxy variable to kas. Is now usable in the same manner as
the 'http_proxy' variable

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-16 11:03:07 +02:00
Claudius Heine
75fa095aa5 config: Added schema validation for configuration files
Currently many of the error messages regarding the configuration file
are not really helpful.

To improve reporting this patch adds validation of the configuration
files using the jsonschema module.

Example:

  Forgetting ':' at the end of a layer definition, like this:

    repos:
      meta-iot2000:
        layers:
          meta-iot2000-example
                              ^

  Exception before:

    TypeError: string indices must be integers

  Exception after:

    Validation Error:
     'meta-iot2000-example' is not of type 'object'

     Failed validating 'type' in schema['properties']['repos']['additionalProperties']['properties']['layers']:
         {'additionalProperties': {'oneOf': [{'type': 'null'},
                                             {'type': 'integer'},
                                             {'type': 'boolean'},
                                             {'type': 'string'}]},
          'type': 'object'}

     On instance['repos']['meta-iot2000']['layers']:
         'meta-iot2000-example'

This patch adds 'jsonschema' as an additional dependency.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:10 +02:00
Claudius Heine
ab8e4cd375 config: Implemented multi-target support
With this patch support for building multiple targets in parallel with
bitbake is now supported.

Changes:

  - Its now possible to state a list of targets under the target key in
    the configuration file.

    Example:

        target:
          - product-image
          - product-update-image

  - Its now possible to define multiple targets in the kas command line.

    Example:

        $ kas build --target product-image --target product-update-image \
          kas.yml

  - Its now possible to define multiple targets via the environment:

    Example:

        $ export KAS_TARGET="product-image product-update-image"
        $ kas build kas.yml

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:00 +02:00
Jan Kiszka
9e136e532d config: Remove dynamic configuration variant
We no longer see an urging use case for it. All field scenarios can
perfectly be modeled with the static format.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-27 10:48:35 +02:00
Jan Kiszka
13a87cb8c7 Allow to define task in config and environment
Based on original patch by Sascha Weisenberger.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-25 09:33:43 +02:00
Jan Kiszka
100c7e047f libkas: Pass git proxy related environment variables
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
2a618686f1 Switch to separate config file version
Use a separately incremented integer to track the configuration file
format version. We start with 2 due to the change that 23c3a951f6
introduced. 1 is declared to be equivalent to the original '0.10'.

The separate versioning has the advantage of being able to increment it
already during the development cycle, and using it with config files
that test/exploit the new format.

Using an integer has the advantages of a) differentiating it clearly
from the now independent kas version and b) simplifying the version
parsing. We can now also remove the string type restriction.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:31:23 +02:00
Jan Kiszka
5478e43f8a docs: Fix and reformat environment table
Sphinx complained:

docs/userguide.rst:66: ERROR: Malformed table.
Column span alignment problem in table line 8.

Avoid this by reformatting to a grid table, also adjusting the content a
bit.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:33 +02:00
Jan Kiszka
2a4a34865a docs: Mention the official docker hub path
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-12 10:22:15 +02:00
Jan Kiszka
eda4129b54 docs: Mark dynamic configuration format experimental
As the note states, we still need to decide about the future of this
API.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Claudius Heine
9cf11bb8cf docs: added description of DL_DIR, SSTATE_DIR, TMPDIR env variables
Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:53:17 +02:00
Claudius Heine
67ec7f3bfc Extended and converted user guide to ReST
Restructured text provide more options to extend the documentation by
including sphinx generated content.

Also added reference documentation for the static configuration format.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00