Commit Graph

346 Commits

Author SHA1 Message Date
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Claudius Heine
b2e1074771 checkcode: add shell check
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:31 +02:00
Claudius Heine
f2c68e6862 kas-docker: fix shellcheck warnings
Signed-off-by: Claudius Heine <ch@denx.de>
[Jan: tiny indention fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 16:31:01 +02:00
Vijai Kumar K
a48ae6a7fa kas-docker: Fix bad substitution error
kas-docker uses sh shell, which doesnot support pattern substitution
like bash. Replace Bash like pattern substitution with sed.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-19 10:38:52 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Paul Barker
f8067a449b checkcode.sh: Abort if no path is provided
This ensures we don't try to check all python files in '/' if the
argument to the checkcode.sh script is missed by accident.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:00:04 +02:00
Paul Barker
04e5a2487c Add flake8 config
When running the checkcode.sh script, if no flake8 config file is
present then all error codes will be emitted. This may include error
codes from flake8 plugins installed locally which are not relevant for
this project (such as the flake8-black plugin). To prevent this issue we
add a flake8 config file which enables the built-in error codes C, E, F
and W.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:59:51 +02:00
Paul Barker
c2bbfee8f3 tests: Check refspec switch updates git branch/commit correctly
Mercurial has no concept of a local tracking branch so we only need to
test git repositories here.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:44 +02:00
Paul Barker
24f2f66cf5 kas: Support --update argument
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:40 +02:00
Paul Barker
1569895001 kas: Support --force-checkout argument
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:32 +02:00
Paul Barker
90598db3c8 context: Store arguments in global context
This makes it much simpler to implement new command line arguments.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:27 +02:00
Paul Barker
f0094af79f kas: De-duplicate common plugin args
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:57:11 +02:00
Paul Barker
a37d72ea7c kas: Simplify setup of plugin argument parsers
By defining the plugin name and help message as attributes of the plugin
class we can move the argument parser creation up into the
kas_get_argparser() function. This will allow us to further reduce
duplication in following commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:49:11 +02:00
Paul Barker
23277836c4 tests: Check patches can still be applied after switching refspec
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:53:59 +02:00
Paul Barker
79bfe5a04d tests: Return to kas directory when test_patch finishes
This prevents subseqent tests from starting in the wrong directory.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:53:54 +02:00
Paul Barker
b998dfa722 repos: Make tracking branch resolution more robust
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.

It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:18:43 +02:00
Harald Seiler
a5dc5f8663 kas-docker: Add support for podman
On many modern Linux systems, docker no longer works because they only
have cgroups v2 support.  To use kas in a container on these systems,
one can resort to using podman.  Add support for this alternative docker
engine to kas-docker.

Now, kas-docker searches for the first available engine in "docker,
podman" or, if the KAS_DOCKER_ENGINE environment variable is set, will
force the specified engine.

Signed-off-by: Harald Seiler <hws@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-03 12:00:11 +02:00
Paul Barker
99af709434 requirements_rtd: Add jsonschema module
This module is needed so that we can import kas during the readthedocs
build to extract command line usage information.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 10:36:54 +02:00
Paul Barker
d81c794f30 docs: Drop reference to Python 2
Python 2 is dead, let's not encourage people to add support for it.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:48 +02:00
Paul Barker
f03f79c814 kas-docker: Allow extra bitbake arguments to be passed
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:37 +02:00
Paul Barker
4cae209f83 docs: Drop list of supported Yocto versions
This informaition is out-of-sync with the upstream supported Yocto
Project versions and gives the impression that kas is unmaintained.

kas should work with any recent Yocto version.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:29 +02:00
Paul Barker
53668123c7 repos: Fix typo in repo type error message
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:07 +02:00
Jan Kiszka
e41d5af76b repos: Fix cloning of repos without default branch
Handle gracefully the case that a repo has no branch checked out after
cloning.

Reported-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-29 14:33:09 +02:00
Claudius Heine
116853cbb4 kas-docker: switch to /bin/bash as SHELL per default
In case SHELL is `/usr/bin/bash`, `/usr/bin/zsh` or something else that
is not available in the kas docker container, fall back to `/bin/bash`.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-29 14:22:40 +02:00
Paul Barker
677c469fb5 build: Allow extra bitbake arguments to be passed
This allows the `kas build` command to be used when you need to pass
extra arguments to bitbake.

A `--` separator is needed when passing arguments which begin with a
dash so that argparse doesn't try to parse them as kas further
arguments.

For example, to pass `--skip-setscene` to bitbake:

    kas build config.yml -- --skip-setscene

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:08:00 +02:00
Paul Barker
8fb8b1b1e1 shell: Propagate non-zero exit status
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:37 +02:00
Paul Barker
3ba7091251 build: Propagate non-zero exit status when running in a tty
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:33 +02:00
Paul Barker
6ada99e40d kas: Don't return None from kas()
The return value of kas() is used as the argument to sys.exit() so it
should be 0 to report successful completion.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:02:50 +02:00
Paul Barker
44a87298ba build: Drop unnecessary argument to BuildCommand()
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 11:13:42 +02:00
Jan Kiszka
d02e87a0a7 Release 2.1.1 2020-05-19 08:03:26 +02:00
Jan Kiszka
d58a291513 repos: Silence pycodestyle error
New pycodestyle version, new warning: "l" might be confused with "1".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 07:53:02 +02:00
Jan Kiszka
68809708e8 Release 2.1 2020-05-19 07:30:20 +02:00
Jan Kiszka
378457b2c8 kas-docker: Make loop device passing optional
This used to be required in older Isar versions (prio to commit
8b8a339cfaa4) for one configuration. It may still be required by
downstream layers, though it can be avoided. To overcome this
potential need to enter the host's root password without breaking those
cases, keep the feature but make it explicit (--with-loop-dev option).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-15 18:37:18 +02:00
Jan Kiszka
385e42ff11 kas-docker: Warn when running as root
While this works from kas perspective, some build recipes will no like
to be run with ID 0 and may either fail loudly or subtly. Better warn
when a user applies sudo on kas-docker or otherwise runs it from a root
shell.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-15 18:37:13 +02:00
Claudius Heine
96df30a80f kas-docker: add aws-dir command parameter
The `--aws-dir` parameter for the kas-docker script allows to specify a
AWScli config directory which will then be mounted to `/etc/skel/.aws`
and copied by kas into the internal home directory using the `AWS_*`
environment variables.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:31 +02:00
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Claudius Heine
e61dfb7650 Dockefile: add awscli for s3 fetcher
bitbake provides a fetcher for Amazon S3 storage using the awscli
command. This patch adds the awscli package to the container.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-29 14:54:25 +02:00
Jan Kiszka
2142fd855e Fix reporting of repo patch IDs
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.

Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-10 18:10:17 +02:00
Jan Kiszka
1c1e3e592a kas-docker: Fix error message
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-18 17:02:46 +01:00
Jan Kiszka
789d3aa604 kas-docker: Fix a shellcheck warning
Resolves "SC2145: Argument mixes string and array". Harmless here, but
the result looks nicer anyway.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-18 17:02:46 +01:00
Jan Kiszka
a1d9ae15e0 kas-docker: Add support for spaces in paths
This requires consistent quoting of related variables. Furthermore, the
eval in trace() has to be removed, and along with it the additional
single-tick quotes of some parameters.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-18 17:01:44 +01:00
Jan Kiszka
5932dc77ff kas-docker: Eliminate intermediate DOCKER_ARGS variable
This will allow passing of all docker arguments in a properly quoted
way. More of them will need that in order to support paths with spaces.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-17 15:07:04 +01:00
Jan Kiszka
d96a26f1c8 kas-docker: Simplify SHELL variable passing
The SHELL variable we pass can neither contain spaces nor newlines - we
check for supported values. Simplify the related assignment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-17 15:03:38 +01:00
Jan Kiszka
1ba260deeb kas-docker: Fix clean of non-isar builddirs
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-03-17 09:46:18 +01:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Jan Kiszka
e383e16570 Drop Python 3.4 compatibility, include 3.8
PyYAML started to require at least 3.5. There is possibly a way to pick
an older version of it to fulfill its needs, but it's also not worth to
invest effort in this old version anymore.

Use that chance to include 3.8 in the support scope.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:35:40 +01:00
Sam Lewis
b5ba370049 Ensure SSH key ends with newline
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').

This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.

Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:18:54 +01:00