Commit Graph

33 Commits

Author SHA1 Message Date
Claudius Heine
e8851a5fb3 implements patch support for repos
With this commit its now possible to patch 3rd party repos before bitbake is started.

Example:

This is our repo:
  .
  ├── kas.yml
  ├── first-patch.patch
  └── quilt-patches
      ├── second-patch.patch
      ├── third-patch.patch
      └── series

Content of kas.yml:
  header:
    version: 8

  repos:
    my:
    third-party:
      url: "git://example.com/third-party.git"
      refspec: "35adf4...34"
      patches:
        01-first:
          repo: my
          path: "first-patch.patch"
        02-second:
          repo: my
          path: "quilt-patches"

Currently only 'git' repositories can be patched.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:59:07 +01:00
Jan Kiszka
ca6b3c05c9 libkas: Fail run_cmd silently if command is not found
If we only try-execute a command (fail=False), do not bail out with an
exception when starting that command already fails. Catch typical
exceptions and return the corresponding error code along with the
exception string.

This fixes get_root_path in case hg is not installed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-01-25 19:11:43 +01:00
Henning Schild
941fa38a2d repos: move repository functions into repos.py
This commit just moves core around and does not contain any logical
changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
dc8d15c3be git: rename "git_operation_disabled" to "operations_disabled"
Prepare for other version control systems and remove "git" from the
names.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
ce02354627 git: allow an empty refspec even when using version control
If no refspec was given just take the default we get after clone.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 14:26:36 +01:00
Henning Schild
62cf4fb0eb libkas: include another space when extending BB_ENV_EXTRAWHITE
the way the env variable got extended broke the last entry in it and the
first entry in the added string because the two strings where not space
separated

i.e.
  "... no_proxySSTATE_DIR ..."
now you shoud get
  "... no_proxy SSTATE_DIR ..."

Seems that problem was in kas since day one.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Claudius Heine
95015045db implementation of environment variable passthrough
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:

    env:
      VARIABLE_NAME: "default variable value"

Those variables can be overwritten by variables of the environment kas is
started in:

    $ VARIABLE_NAME="overwritten variable value" kas shell ...

Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.

This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-24 10:19:12 +01:00
Daniel Wagner
7aff26d105 libkas: Add /sbin and /usr/sbin to PATH
/sbin and /usr/sbin might contain tools Yocto depens on, e.g. 'ip'.

"""
The following required tools (as specified by HOSTTOOLS) appear to be
unavailable in PATH, please install them in order to proceed:

	ip
"""

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-11-20 14:12:28 +01:00
Henning Schild
d06bdc4781 libkas: do not use absolute paths when executing programs
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2017-09-11 10:19:09 +02:00
Jan Kiszka
2d6b1ce6a0 libkas: Detect and report multiple init scripts
Having more than one them is a sign of serious problems in the
configuration. Better catch that early and fail loudly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:29:54 +02:00
Jan Kiszka
eb284afb8e libkas: Detect error of init script
Do not swallow errors of the init script. Rather leave our wrapper
immediately and let run_cmd report the splash and terminate kas.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:11:54 +02:00
Jan Kiszka
100c7e047f libkas: Pass git proxy related environment variables
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
8e3862112c libkas: Fix false positive of pylint
As suggested by Claudius Heine, reorder the blocks so that we can add
the pylint exception to the if half.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
f58d44f190 libkas: Properly format stderr dump of failing commands
Inject the linefeed that we removed in f0a18868c9 and add an
introduction line.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
264be71ebe libkas: Terminate kas on _repo_fetch_async errors
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.

Drop the return value of repos_fetch - no caller expects any.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:49 +02:00
Jan Kiszka
adb4007482 libkas: Do not sys.exit from a coroutine
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:36 +02:00
Jan Kiszka
f0a18868c9 libkas: Improve further on error/success output
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:28 +02:00
Claudius Heine
503fd3d47a Seperated argument parser creation from main kas entry point
With Sphinx it is possible to create the command line documentation
automatically if there is a function that returns just the command line
parser.

Currently the creation of the argument parser is rather entangled with
the rest of kas. This patches seperates this.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
546b51f450 Fixed issues that occured with pylint 1.7.2
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.

This patch fixes these issues:

************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:10:14 +02:00
Jan Kiszka
0bdd7a8d52 libkas: Rework repo_fetch to parallelized repos_fetch
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.

The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
8e4fbd6d3f libkas: Provide asynchronous interface for command execution
Enhance _stream_subprocess to run_cmd_async, a co-routine variant of
run_cmd that the caller can use to parallelize command execution.
run_cmd becomes a simple wrapper that waits for the async variant to
complete.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
dddbdb0112 libkas: Improve reporting of repo_fetch
Instead of writing "commit", issue a more telling log message if some
repo already contains the required hash.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
94555bb892 libkas: Do not close event loop after each command
Rather close the loop when terminating the kas process.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
40ddeafa22 libkas: Remove redundant local var initialization
retc is already initialized via loop.run_until_complete().

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:38 +02:00
Jan Kiszka
8bb7257ecc libkas: Fix some comments
Typo and a missing verb.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:33 +02:00
Jan Kiszka
47e8a0730f libkas: Reorder cloning commands and add silent mode consistently
The two code paths are unfortunate. Factor out the difference and use a
single run_cmd invocation. This also adds the forgotten -q when using
a reference directory.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:30 +02:00
Claudius Heine
17691d2756 Refactored fetching and checking out of missing repos in kas.config
The current implementation did some more hackish solution to work
around some of kas infrastructure, without changing to much.

This patch cleans this up and therefor remove the more obsure part
of the include mechansim.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-22 12:33:00 +02:00
Jan Kiszka
013af80599 Add support for Isar build system
Isar allows to build Debian-based images using bitbake and the usual
layer structures. It's very similar to OE, we just need to account for
a different init script name.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-22 10:50:40 +02:00
Claudius Heine
33a21c8d0d run pylint3 and fixed report
This commit adds a pylint configuration and fixed all found issues.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Jan Kiszka
6bc8e08459 Fix pep8 reports
Zero findings after this commit. We should try to keep that level.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-21 17:30:51 +02:00
Claudius Heine
81a30bc600 Added SHELL and TERM to environment whitelist and start $SHELL if set
Currently the shell provided via the 'shell' command is always '/bin/sh'
and does not have a TERM variable set. This is a rather spartan shell
experience.

With this patch we add the TERM variable to the whitelist, so that
pagers work better and made the shell binary dependent from the SHELL
environment variable. If no such variable is set, we fall back to using
'/bin/sh'.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-19 10:40:01 +02:00
Claudius Heine
bb2e98344a Added 'SSH_AGENT_PID' and 'SSH_AUTH_SOCK' to environment whitelist
The ssh-agent environment variables of an ssh-agent that is running
outside the kas build system are not transfered to the bitbake
environment. This causes fetching of private git repositories to
fail.

This patch adds those variables to the environment whitelist.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-19 10:39:34 +02:00
Daniel Wagner
daf0abab5e Initial public release
This is the first public release of kas.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-06-14 15:21:19 +02:00