Commit Graph

638 Commits

Author SHA1 Message Date
Daniel Wagner
f60a8b4e6d test_includehandler: Remove useless object inheritanc
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-07-18 16:26:47 +02:00
Daniel Wagner
bc7aa5c694 includehandler: Silence too-many-arguments pyling warning
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-07-18 16:26:47 +02:00
Daniel Wagner
1b804f937a includehandler: Remove useless object inheritance
pylint reports:

kas/includehandler.py:105:0: R0205: Class 'IncludeHandler' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-07-18 16:26:47 +02:00
Daniel Wagner
de243b28b8 Release 0.18.0 2018-05-17 13:12:10 +02:00
Daniel Wagner
c1e5c4f379 build: Fix all style reports by pycodestyle v2.4.0
pycodestyle v2.4.0 checks for W606 (async deprecated), but
we can't disable it on source code level with '# noqa'. Let's disable
the check globaly. We will still catch it with pylint.

By disabling the W606 we need also to list the W503 which was
previously active.

While at it fix following things:

./docs/conf.py:252:6: E121 continuation line under-indented for hanging indent
./docs/conf.py:257:6: E126 continuation line over-indented for hanging indent
./kas/build.py:123:17: W504 line break after binary operator
./kas/configschema.py:48:25: E123 closing bracket does not match indentation of opening bracket's line
./kas/configschema.py:49:21: E123 closing bracket does not match indentation of opening bracket's line
./tests/test_includehandler.py:82:13: E123 closing bracket does not match indentation of opening bracket's line
./tests/test_includehandler.py:121:13: E123 closing bracket does not match indentation of opening bracket's line

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-05-17 13:00:26 +02:00
Henning Schild
03ad285919 docker: Take debootstrap from stretch for Isar image
The jessie version has a bug which prevents it from building debian
"buster" and above.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-04-19 12:06:26 +02:00
Daniel Wagner
6dfc870feb travis: Tag the last release with latest-release
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-04-04 14:29:26 +02:00
Claudius Heine
6afd92eabc kas/config: Make 'get_repo_dict' private
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private.  The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.

This patch also allows 'get_repos' and 'repo_dict' to be used even in the
'Repo' class. 'get_repo_dict' creates new 'Repo' instances and thus
invalidates the current instance.

This patch is in preparation of the 'patch support'-feature, that
depends on this patch.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-04-04 13:43:37 +02:00
Abdur Rehman
5b85fba0df repos: use git diff-index to check if repo is dirty
is_dirty_cmd() uses `git diff --shortstat` to see if the repo can be
safely checked out. This only checks for the modified files and does
not take the cached(staged) files into account.

There can be cases where the cached changes could be overwritten by a
checkout. In this case, the git checkout command aborts with an error
message telling the user to commit or stash their changes.

Use `git diff-index HEAD --shortstat` to determine if there are any
changes in the tracked files, whether cached or not.

https://git-scm.com/docs/git-diff-index

Signed-off-by: Abdur Rehman <abdur_rehman@mentor.com>
2018-04-03 11:39:37 +02:00
Daniel Wagner
4be43c7bf6 Revert "kas/config: Make 'get_repo_dict' private"
This reverts commit b9032ec025.

The commit breaks inclusion chains of kas files.  In my example a
first kas-file includes a second one. The second one is supposed to
add repos but kas does not fetch those.

Reported-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-03-26 14:49:06 +02:00
Claudius Heine
40e528065f kas/libcmd: Instead of copying the .gitconfig, create a new dummy one
Copying the .gitconfig failes if the current user doesn't have one,
since the commits that are done within kas, will not be pushed or
merged anywhere, we can just create a minimal .gitconfig in the temp
home directory of kas.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-21 17:16:04 +01:00
Claudius Heine
e8851a5fb3 implements patch support for repos
With this commit its now possible to patch 3rd party repos before bitbake is started.

Example:

This is our repo:
  .
  ├── kas.yml
  ├── first-patch.patch
  └── quilt-patches
      ├── second-patch.patch
      ├── third-patch.patch
      └── series

Content of kas.yml:
  header:
    version: 8

  repos:
    my:
    third-party:
      url: "git://example.com/third-party.git"
      refspec: "35adf4...34"
      patches:
        01-first:
          repo: my
          path: "first-patch.patch"
        02-second:
          repo: my
          path: "quilt-patches"

Currently only 'git' repositories can be patched.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:59:07 +01:00
Claudius Heine
b9032ec025 kas/config: Make 'get_repo_dict' private
Since 'get_repo_dict' does much more than just querying available
repositories, this patch makes it private.  The same data is now
available via the 'repo_dict' property. This also means that
'get_repos' is now faster, because it accesses cached data, instead of
calling 'get_repo_dict' that generates this data.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-03-09 09:55:43 +01:00
Jan Kiszka
57f0ed85b9 docker: Add debootstrap to Isar image
Needed due to upcoming switch from multistrap to debootstrap. The former
can eventually be removed in a later release.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-03-07 09:22:27 +01:00
Jan Kiszka
b2180899b8 docker: Take qemu-user-static from stretch for Isar image
The jessie version easily crashes, e.g. during kernel build over
dpkg-deb.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-03-07 09:22:27 +01:00
Daniel Wagner
22d196a058 scripts: Remove the double Subject line from releases script
The subject line is written by mutt.

While at it also start emacs in text mode for editing the changelog.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-03-07 09:22:27 +01:00
Daniel Wagner
1e73112a9a Release 0.17.0 2018-02-08 15:49:56 +01:00
Jan Kiszka
cda778acb5 libcmds: Write MACHINE and DISTRO as weak defaults
Otherwise we cannot overwrite them in mulitconfig files. It's not
expected that anything else in the layer config files sets those
variable, thus shadows our setting in local.conf.

Something changed in latest bitbake to reveal this issue. We first see
it in Isar since bitbake moved to f8126aaf7741.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-02-05 21:07:28 +01:00
Daniel Sangorrin
9bd33ac221 Dockerfile: add iproute2
runqemu requires /sbin/ip

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
2018-01-29 10:07:24 +01:00
Jan Kiszka
ca6b3c05c9 libkas: Fail run_cmd silently if command is not found
If we only try-execute a command (fail=False), do not bail out with an
exception when starting that command already fails. Catch typical
exceptions and return the corresponding error code along with the
exception string.

This fixes get_root_path in case hg is not installed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-01-25 19:11:43 +01:00
Georg Lutz
7fa75d94c0 repos: Fix relative path for repos
If a repo path is relative, append the kas work dir to it.

Background: The path ends up in bblayers.conf

Without this change the following example will fail:

--------------------------------------------------------
header:
     version: 4
machine: qemux86
repos:
     poky:
         url: "https://git.yoctoproject.org/git/poky"
         refspec: "rocko"
         path: "thirdparty/poky"
         layers:
             meta:
             meta-poky:
--------------------------------------------------------

Signed-off-by: Georg Lutz <georg@georglutz.de>
2018-01-25 18:54:55 +01:00
Jan Kiszka
37cd2f42d5 Docker: Install xz-utils from stretch
The stretch version supports multi-thread compression, allowing to
accelerate e.g. artifact compression tasks. The newer version of xz also
seem to resolve mysterious compression failures that created too short
output images.

As we install the stretch source in the base image now, we can remove
that port from the isar image and align both regarding the setup.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-01-24 10:44:50 +01:00
Daniel Wagner
41dc0bef59 Release 0.16.1 2018-01-10 13:06:15 +01:00
Daniel Wagner
afbcd0018b scripts: Add release script
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-10 13:04:18 +01:00
Daniel Wagner
c6ce77a4c9 Release 0.16.0 2018-01-10 11:29:20 +01:00
Daniel Wagner
03caf33162 shell: Add missing imports for CleanupSSHAgent and SetupSSHAgent
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-09 14:26:47 +01:00
Henning Schild
db25611977 docs: update docs accordning to previous changes
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
ec69bcb0c2 docker: add mercurial to our toolbox
With kas now supporting mercuial as well, add that tool to the docker
containers.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
013722fd28 repos: add support for Mercurial Repos
This commit adds support for cloning mercurial repos using the type
string 'hg'.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
c47b511d34 repos: factor git-cmds out of a common Repo implementation
This creates an interface that could also be used to support other
version control systems.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
6a85296836 repos: move more of the complexity from config to Repo.factory
This commit just moved some code around, to a place where it seems to
fit better.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
0c7db599fd repos: introduce a new repo conf parameter "type"
This will be used to carry the type of repo we are talking about. At the
moment it defaults to "git".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
9da82dba13 config: call Repo.factory only in one place
This patch simplifies the code a bit and does not change any semantics.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
3594ad50a0 repos: introduce a factory to abstract away from Repo implementation
The user just calls that factory and gets something that is a Repo, in
fact just a GitRepo for now.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
b5b2766b24 repos: also move version control call out of config.py
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
941fa38a2d repos: move repository functions into repos.py
This commit just moves core around and does not contain any logical
changes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Henning Schild
dc8d15c3be git: rename "git_operation_disabled" to "operations_disabled"
Prepare for other version control systems and remove "git" from the
names.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-08 17:34:59 +01:00
Daniel Wagner
9210fe57b4 shell: Defer SetupEnviron step
If we start on a green field (only kas is available) we can't setup
the environment directly. So in this case the order is SetupProxy,
ReposFetch, ReposCheckout and then SetupEnvironment. Only if the
--keep-config-unchanged flag is provided directly go from SetupProxy
to SetupEnvironment.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-08 16:55:44 +01:00
Daniel Wagner
415e0bf114 shell: Honor SSH_PRIVATE_KEY environment variable
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-08 16:55:44 +01:00
Daniel Wagner
bef5aa80b3 docs: Install kas as user
pip warns us with:

"Running pip install with root privileges is generally not a good
idea. Try `pip3 install --user` instead."

So update the documentation to install kas as user.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-08 16:55:44 +01:00
Jan Kiszka
b42aafb97d shell: Drop default from --target
Forgotten by 49cb9b7a7b: The default is now set in
get_bitbake_targets.

This fixes spurious overwrites of the target defined in the config file.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-01-02 14:30:29 +01:00
Daniel Wagner
8a87f09c70 repos: Raise exception if attribute is unknown in __getattr__
pylint reports

R: 52, 4: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)

According the Python documentation we should raise AttributeError
if the key is unknown.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-02 14:26:46 +01:00
Daniel Wagner
90315d47bf includehandler: Fix string formating issue
pylint is reporting

W: 76, 8: Exception arguments suggest string formatting might be intended (raising-format-tuple)

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-01-02 14:26:46 +01:00
Henning Schild
ce02354627 git: allow an empty refspec even when using version control
If no refspec was given just take the default we get after clone.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 14:26:36 +01:00
Henning Schild
62cf4fb0eb libkas: include another space when extending BB_ENV_EXTRAWHITE
the way the env variable got extended broke the last entry in it and the
first entry in the added string because the two strings where not space
separated

i.e.
  "... no_proxySSTATE_DIR ..."
now you shoud get
  "... no_proxy SSTATE_DIR ..."

Seems that problem was in kas since day one.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Henning Schild
4140961b11 libcmds: create directory before open in case it is not there
This could happen if you use kas to checkout repos that do not contain
the directory. i.e. repos that are not actually bb layers

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Henning Schild
19e518cd25 config: support "gentoo" as a known distro
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-01-02 12:59:30 +01:00
Daniel Wagner
3c960140fd Release 0.15.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-12-06 16:47:28 +01:00
Claudius Heine
95015045db implementation of environment variable passthrough
With this its possible to create custom environment variables within the kas
configuration file, that are passed through to the build environment:

    env:
      VARIABLE_NAME: "default variable value"

Those variables can be overwritten by variables of the environment kas is
started in:

    $ VARIABLE_NAME="overwritten variable value" kas shell ...

Those variables are also added to the 'BB_ENV_EXTRAWHITE' variable list to make
them available to bitbake.

This is useful for instance storing sensitive information, that should not be
part of the configuration files or bitbake recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-24 10:19:12 +01:00
Jan Kiszka
cd076101f4 docker: Avoid error messages on container restarts
When the user restarts an already existing kas container, errors are
thrown because of existing folders or users. One example is the usage
of gitlab-ci runners.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-11-22 12:38:07 +01:00