Commit Graph

208 Commits

Author SHA1 Message Date
Paul Barker
4e153a2872 plugins: Improve documentation of existing modules
The docstrings for the build and shell plugins are extended so that they
clearly describe each plugin and give brief examples of their use.

Also fix a typo in the build command help message.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-13 07:45:07 +01:00
Mert Kirpici
e79910aa29 config: avoid duplicates in BBMULTICONFIG
Collecting all declared multiconfigs in a set rather than a list
avoids duplicates in the resulting string when multiple targets
with the same multiconfig are declared.

Signed-off-by: Mert Kirpici <mert.kirpici@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-02 09:10:07 +01:00
Paul Barker
5e3ca820b3 plugins: Refactor plugin identification and access
This change moves all plugin handling code into the kas.plugins module.
New accessor functions `plugins.get(name)` and `plugins.all()` are
provided to wrap the plugins dictionary so that the kas main function
doesn't need to worry about how this is accessed. Plugins are loaded at
runtime rather than at parse time by calling `plugins.load()` which
gives us an improved ability to handle errors.

The `@kasplugin` decorator is removed as it modified and attribute on
the kasplugin function itself when a plugin module was loaded. Importing
a module should not result in changes to a variable in a different
module as it leads to an initialization code flow which is difficult to
reason about. Instead, plugin modules should now list the plugins which
they introduce in a `__KAS_PLUGINS__` list which will be walked at
runtime by `plugins.load()`.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:17:03 +02:00
Paul Barker
7b9bce8e46 kas: Drop undocumented external plugin support
This support for external plugins is undocumented and results in
differences between how internal and external plugins are loaded. It
will shortly be replaced by more integrated support for dynamically
finding and loading plugins.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:14:00 +02:00
Paul Barker
1dffb986b1 plugins: Move plugins to dedicated submodule
This change groups the plugin modules together under kas.plugins and
moves the @kasplugin decorator to this submodule. This will ensure the
codebase stays organised as more plugins are added.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:13:06 +02:00
Paul Barker
af4389e606 kas: Simplify plugin lookup
By storing plugins in a dictionary indexed by plugin name rather than a
list, we can simplify plugin lookup and remove the undocumented need for
a plugin's run() method to return True when it has matched the given
command.

The command will be rejected by the argument parser if it does not match
one of the plugin names so we do not need to handle failure to lookup
the plugin in the dictionary.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-21 17:11:49 +02:00
Jan Kiszka
e5701a03a6 repos: Make set_remote_url_cmd fail loudly
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.

Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-13 14:06:31 +02:00
Jan Kiszka
b8a6577ece Report invalid repo key
If an include references a repo key that is undefined, kas bailed out
with a ValueError because ctx.missing_repos was empty. Catch that and
report it properly.

Reported-by: Nathan Genetzky <ngenetzky@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-10 16:17:16 +02:00
Adrian Stratulat
a5c8345ee6 repos: remove trailing slash from layer path
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.

Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-23 09:34:28 +02:00
Jan Kiszka
22b489b33b Release 2.2 2020-08-20 09:17:34 +02:00
Chris Laplante
9518dc4e35 libkas: make get_build_environ more exception safe
Eliminate manual call to os.remove by using tempfile context manager.

The 'get bb env' script has to be created in a directory (rather than
using a NamedTemporaryFile directly, for instance), since otherwise
you'll get a "Text file busy" error when trying to execute the script.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:33 +02:00
Chris Laplante
22f7c0b03d config: Make common repository check a bit more Pythonic
Shortens code and makes the intent clearer.

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-31 11:32:25 +02:00
Nathan Genetzky
0e7c32cc79 repos: GitRepo fetch_cmd shouldn't default to "git fetch --all"
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.

From git help fetch:

> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.

Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-27 08:21:28 +02:00
Chris Laplante
7068767d27 kas: support .yaml extension
.yaml is the recommended extension (https://yaml.org/faq.html)

Signed-off-by: Chris Laplante <mostthingsweb@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-24 18:44:45 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Paul Barker
24f2f66cf5 kas: Support --update argument
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:40 +02:00
Paul Barker
1569895001 kas: Support --force-checkout argument
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:32 +02:00
Paul Barker
90598db3c8 context: Store arguments in global context
This makes it much simpler to implement new command line arguments.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:27 +02:00
Paul Barker
f0094af79f kas: De-duplicate common plugin args
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:57:11 +02:00
Paul Barker
a37d72ea7c kas: Simplify setup of plugin argument parsers
By defining the plugin name and help message as attributes of the plugin
class we can move the argument parser creation up into the
kas_get_argparser() function. This will allow us to further reduce
duplication in following commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:49:11 +02:00
Paul Barker
b998dfa722 repos: Make tracking branch resolution more robust
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.

It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:18:43 +02:00
Paul Barker
53668123c7 repos: Fix typo in repo type error message
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:07 +02:00
Jan Kiszka
e41d5af76b repos: Fix cloning of repos without default branch
Handle gracefully the case that a repo has no branch checked out after
cloning.

Reported-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-29 14:33:09 +02:00
Paul Barker
677c469fb5 build: Allow extra bitbake arguments to be passed
This allows the `kas build` command to be used when you need to pass
extra arguments to bitbake.

A `--` separator is needed when passing arguments which begin with a
dash so that argparse doesn't try to parse them as kas further
arguments.

For example, to pass `--skip-setscene` to bitbake:

    kas build config.yml -- --skip-setscene

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:08:00 +02:00
Paul Barker
8fb8b1b1e1 shell: Propagate non-zero exit status
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:37 +02:00
Paul Barker
3ba7091251 build: Propagate non-zero exit status when running in a tty
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:05:33 +02:00
Paul Barker
6ada99e40d kas: Don't return None from kas()
The return value of kas() is used as the argument to sys.exit() so it
should be 0 to report successful completion.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 12:02:50 +02:00
Paul Barker
44a87298ba build: Drop unnecessary argument to BuildCommand()
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 11:13:42 +02:00
Jan Kiszka
d02e87a0a7 Release 2.1.1 2020-05-19 08:03:26 +02:00
Jan Kiszka
d58a291513 repos: Silence pycodestyle error
New pycodestyle version, new warning: "l" might be confused with "1".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 07:53:02 +02:00
Jan Kiszka
68809708e8 Release 2.1 2020-05-19 07:30:20 +02:00
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Jan Kiszka
2142fd855e Fix reporting of repo patch IDs
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.

Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-10 18:10:17 +02:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Sam Lewis
b5ba370049 Ensure SSH key ends with newline
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').

This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.

Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:18:54 +01:00
Jan Kiszka
141476203d Release 2.0 2019-11-20 06:53:15 +01:00
Jan Kiszka
230c5a9572 Add support for new multiconfig syntax
Recent bitbake requires to declare multiconfig targets via "mc:".

Use that chance to refactor the function and match on the colon as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-11-08 10:36:08 +01:00
Jan Kiszka
e9ca55a239 Release 1.1 2019-10-01 16:21:40 +02:00
Jan Kiszka
3ef3ac855e Avoid "RuntimeError: Event loop is closed"
Triggered by pytest now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-25 11:00:45 +02:00
Jan Kiszka
26a1a0b03b Create from local branch before applying patches with git
If refspec is pointing to an upstream branch name, the checkout for a
second run will not reset a repo back to the upstream commit. That is
because we applied the repo patches to a local branch that carries the
name of the upstream one.

Fix that by switching to local branch prior to applying any patches.

Reported-by: Belisko Marek <marek.belisko@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-23 13:08:51 +02:00
Henning Schild
81c5beedaa repos: apply patches with version control system and not patch
patch can not handle file mode headers and can therefore not be used to
create executable files. fixes 8a2a21d799

In addition commit every single patch and treat git patches as
potentially just diffs and not mails.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-21 12:49:47 +02:00
Jan Kiszka
419e196cbb build: Fix name of argument variable
Adding --cmd to the task argument changed the name of the destination
variable, and that broke the whole subcommand because 'cmd' is already
used by the top-level parser. Fix this by setting dest explicitly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-03 10:42:19 +02:00
Jan Kiszka
8558b566d9 build: Add -c and --cmd as aliases for --task
This aligns us with bitbake. Moreover, -c/--cmd is easier to tell apart
from --target. Keep --task for backward compatibility.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-31 12:30:53 +02:00
Jan Kiszka
b2af1bcfe7 config: Remove dead code
Not used anywhere, dates back to dynamic configs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-30 09:41:46 +02:00
Henning Schild
8a2a21d799 repos: change the way we apply patches
Switch to applying all patches with "patch" and later do a "<vcs> add;
<vcs> commit". Also iterate over quilt series files ourselfs and apply
the patches one by one.

This means we can apply patches on mercurial as well as on git.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-20 12:53:54 +02:00