Commit Graph

30 Commits

Author SHA1 Message Date
Claudius Heine
71e7f5523d kas: add GIT_CREDENTIAL_HELPER environment variable
This allows to set a git credential helper in the .gitconfig, in order
to handle git authentications. For instance when bitbake recipes require
to clone from git via https.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-06 14:08:40 +02:00
Peter Hatina
24c13a491b Apply patches before doing an environment setup
We may need to patch the environment setup, too.

Signed-off-by: Peter Hatina <peter@hatina.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-03-01 13:58:49 +01:00
Jan Kiszka
6a8abf277c Add build_system property to config file
This allows to pre-select the build system, specifically avoiding
confusion when kas-container is accidentally not called with --isar for
an isar config. For that, build_system needs to be defined in the
lop-level config file passed to kas-container.

Theoretically, this also allows to combine layers which have both
oe-init-build-env and isar-init-build-env scripts.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-16 11:53:25 +01:00
Paul Barker
bcb748c22a kas: Factor out common setup & cleanup steps
The list of steps to be executed within a macro is split into setup
commands, main commands and cleanup commands to ensure that new steps
are added to the appropriate place in the execution order by plugins.

By default, the setup and cleanup commands are set to the default list
of steps which the build and shell plugins have in common. It's expected
that most future plugins will also share these common setup and cleanup
steps but we do provide arguments to the Macro initialiser to skip these
if customisation is needed.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:56:52 +01:00
Paul Barker
c408e89ef1 kas: Eliminate Context.keep_config flag
This variable was only set in the shell plugin to disable the normal
behaviour of the FinishSetupRepos step. We can achieve the same result
by not adding the FinishSetupRepos step when we wish to keep the config
in the build environment unchanged.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 20:54:35 +01:00
Jan Kiszka
b8a6577ece Report invalid repo key
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.

Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-10 16:17:16 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
6349f7ab73 macros: Use new macro loop for repo setup
Divide the SetupRepos command into three parts: init, step and finish
Build a macro-based loop with the step part of the function.

Rewrite this in build.py and shell.py

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
fc9ebf1104 macros: Add loop class for loop functionality
To keep the macro style of the program flow and explicitely state the
command sequence with Macro.add, we need a mechanism that repeats
several steps of the Macro depending on a given condition. This will be
used by the repo checkout logic.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:34 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
7937c1963e libkas: Remove parameters from get_build_environ()
All of them can be retrieved from the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:02 +02:00
Jan Kiszka
448107781e libkas: Remove config/context parameters from ssh functions
We can obtain the global context now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:48 +02:00
Andreas Reichel
7b18e5ec3b Refactor repo checkout
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.

This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.

This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.

The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).

Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-08-23 16:29:56 +02:00
Claudius Heine
40e528065f kas/libcmd: Instead of copying the .gitconfig, create a new dummy one
Copying the .gitconfig failes if the current user doesn't have one,
since the commits that are done within kas, will not be pushed or
merged anywhere, we can just create a minimal .gitconfig in the temp
home directory of kas.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-21 17:16:04 +01:00
Claudius Heine
e8851a5fb3 implements patch support for repos
With this commit its now possible to patch 3rd party repos before bitbake is started.

Example:

This is our repo:
  .
  ├── kas.yml
  ├── first-patch.patch
  └── quilt-patches
      ├── second-patch.patch
      ├── third-patch.patch
      └── series

Content of kas.yml:
  header:
    version: 8

  repos:
    my:
    third-party:
      url: "git://example.com/third-party.git"
      refspec: "35adf4...34"
      patches:
        01-first:
          repo: my
          path: "first-patch.patch"
        02-second:
          repo: my
          path: "quilt-patches"

Currently only 'git' repositories can be patched.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:59:07 +01:00
Jan Kiszka
cda778acb5 libcmds: Write MACHINE and DISTRO as weak defaults
Otherwise we cannot overwrite them in mulitconfig files. It's not
expected that anything else in the layer config files sets those
variable, thus shadows our setting in local.conf.

Something changed in latest bitbake to reveal this issue. We first see
it in Isar since bitbake moved to f8126aaf7741.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-02-05 21:07:28 +01:00
Henning Schild
941fa38a2d repos: move repository functions into repos.py
This commit just moves core around and does not contain any logical
changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
4140961b11 libcmds: create directory before open in case it is not there
This could happen if you use kas to checkout repos that do not contain
the directory. i.e. repos that are not actually bb layers

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Claudius Heine
fa1575790f multiconfig: implement initial support
This patch adds multiconfig support to kas by defining the
"BBMULTICONFIG" variable in the "local.conf" file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-08 09:42:56 +01:00
Daniel Wagner
133f1be38a libcmds: Remove unused pre and post hooks
9e136e532d ("config: Remove dynamic configuration variant")
removed the posibility to add pre and post hooks. Remove the remaining
bits from libcmds.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-09-27 10:49:35 +02:00
Jan Kiszka
c9326cc1ed Write deterministic local.conf and bblayers.conf
Sort the header sections by their keys and the layers by their names.
This will re-enable writing deterministic config files, and bitbake can
use its cache again, accelerating rebuild startup times a lot.

While at it, indent the layers in bblayers.conf nicely.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:32:06 +02:00
Jan Kiszka
0bdd7a8d52 libkas: Rework repo_fetch to parallelized repos_fetch
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.

The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Claudius Heine
17691d2756 Refactored fetching and checking out of missing repos in kas.config
The current implementation did some more hackish solution to work
around some of kas infrastructure, without changing to much.

This patch cleans this up and therefor remove the more obsure part
of the include mechansim.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-22 12:33:00 +02:00
Jan Kiszka
013af80599 Add support for Isar build system
Isar allows to build Debian-based images using bitbake and the usual
layer structures. It's very similar to OE, we just need to account for
a different init script name.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-22 10:50:40 +02:00
Claudius Heine
33a21c8d0d run pylint3 and fixed report
This commit adds a pylint configuration and fixed all found issues.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Jan Kiszka
6bc8e08459 Fix pep8 reports
Zero findings after this commit. We should try to keep that level.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-21 17:30:51 +02:00
Daniel Wagner
daf0abab5e Initial public release
This is the first public release of kas.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-06-14 15:21:19 +02:00