Commit Graph

173 Commits

Author SHA1 Message Date
Claudius Heine
267a53e78d kas/libcmds: add creation of $HOME/.aws
Bitbake provides a S3 fetcher (`lib/bb/fetch2/s3.py`), however the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` variables are not
transfered to the environment of the fetcher command (`aws` in this
case) in the `runfetchcmd` function (`lib/bb/fetch2/__init__.py`).

That means the location of these files need to be the default one, so
copying it in KAS to the new HOME directory is necessary.

This patch implements the copying of those files if the
`AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` are set.

Per default there variables are not available in the bitbake
environment. To have them available there a `env` entry in the kas
configuration will be necessary.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-05 14:34:28 +02:00
Jan Kiszka
2142fd855e Fix reporting of repo patch IDs
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.

Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-10 18:10:17 +02:00
Pierre-Jean Texier
71b9c43472 config: use 'qemux86-64' instead of 'qemu' as default for KAS_MACHINE
'qemu' is not a valid machine. So, let's use the default one defined in
poky -> 'qemux86-64' [1]

Fixes:

ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    MACHINE=qemu is invalid. Please set a valid MACHINE in your local.conf, environment or other configuration file.

[1] - https://git.yoctoproject.org/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample#n37

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-02-18 13:05:13 +01:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Sam Lewis
b5ba370049 Ensure SSH key ends with newline
In at least some versions of the 'ssh-add' command, adding a SSH key
requires that SSH key is newline terminated. If it is not, ssh-add
doesn't add the key and, instead, unhelpfully prompts for a
passphrase for the key ('Enter passphrase for (stdin):').

This change makes sure the key is terminated with a newline character to
hopefully avoid this issue occurring as often.

Signed-off-by: Sam Lewis <sam.vr.lewis@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:18:54 +01:00
Jan Kiszka
141476203d Release 2.0 2019-11-20 06:53:15 +01:00
Jan Kiszka
230c5a9572 Add support for new multiconfig syntax
Recent bitbake requires to declare multiconfig targets via "mc:".

Use that chance to refactor the function and match on the colon as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-11-08 10:36:08 +01:00
Jan Kiszka
e9ca55a239 Release 1.1 2019-10-01 16:21:40 +02:00
Jan Kiszka
3ef3ac855e Avoid "RuntimeError: Event loop is closed"
Triggered by pytest now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-25 11:00:45 +02:00
Jan Kiszka
26a1a0b03b Create from local branch before applying patches with git
If refspec is pointing to an upstream branch name, the checkout for a
second run will not reset a repo back to the upstream commit. That is
because we applied the repo patches to a local branch that carries the
name of the upstream one.

Fix that by switching to local branch prior to applying any patches.

Reported-by: Belisko Marek <marek.belisko@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-23 13:08:51 +02:00
Henning Schild
81c5beedaa repos: apply patches with version control system and not patch
patch can not handle file mode headers and can therefore not be used to
create executable files. fixes 8a2a21d799

In addition commit every single patch and treat git patches as
potentially just diffs and not mails.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-21 12:49:47 +02:00
Jan Kiszka
419e196cbb build: Fix name of argument variable
Adding --cmd to the task argument changed the name of the destination
variable, and that broke the whole subcommand because 'cmd' is already
used by the top-level parser. Fix this by setting dest explicitly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-03 10:42:19 +02:00
Jan Kiszka
8558b566d9 build: Add -c and --cmd as aliases for --task
This aligns us with bitbake. Moreover, -c/--cmd is easier to tell apart
from --target. Keep --task for backward compatibility.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-31 12:30:53 +02:00
Jan Kiszka
b2af1bcfe7 config: Remove dead code
Not used anywhere, dates back to dynamic configs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-30 09:41:46 +02:00
Henning Schild
8a2a21d799 repos: change the way we apply patches
Switch to applying all patches with "patch" and later do a "<vcs> add;
<vcs> commit". Also iterate over quilt series files ourselfs and apply
the patches one by one.

This means we can apply patches on mercurial as well as on git.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-20 12:53:54 +02:00
Henning Schild
df8d7b929d repos: Fix up the basic mercurial support
The interface cmd prototypes got out of sync so mercurial stopped
working. Align with git again to fix that.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-14 12:10:08 +02:00
Pascal Bach
66bd078e7f repos: make sure the remote url is matching the kas file
Currently the the remote URL is only set during the initial clone.
This might lead to confusion with the user as changing the URL in the kas.yml
does not get reflected and thus errors about references not found show up.

This change makes sure the URL is always set to the value specified in the
kas.yml file.

For mercurial this currently prints a warning as there is no command to change
the default path.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
[Jan: fix up style, adjust comment on hg's set_remote_url_cmd]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-13 20:38:03 +02:00
Daniel Wagner
907816a5c4 Release 1.0 2019-03-21 13:07:32 +01:00
Daniel Wagner
5b5694daf7 Release 0.20.1 2019-02-18 15:03:10 +01:00
Daniel Wagner
3ce0a1f941 Release 0.20.0 2019-02-18 11:15:00 +01:00
Jan Kiszka
2f24bdeac2 repos: Fail on unsuccessful patching
There is no good reason to only warn about an incorrect patch entry or
a missing patch file. This just causes CI systems to succeed with a
build - or fail at unexpected places.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:22:55 +01:00
Jan Kiszka
2f7650bb05 Add KAS_PREMIRRORS support
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:35 +01:00
Daniel Wagner
46c7d6e38c kas: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
    names in their public API using the __all__ attribute.
"""

flake8 reports

./kas/kas.py:51:1: F401 '.build' imported but unused
./kas/kas.py:52:1: F401 '.shell' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
294d4df286 __init__: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
names in their public API using the __all__ attribute.
"""

reported by flake8:

./kas/__init__.py:26:1: F401 '.__version__.__version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__file_version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__compatible_file_version__' imported but unused
./kas/__init__.py:28:1: F401 '.configschema.CONFIGSCHEMA' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Jan Kiszka
fe344f93f1 Fix --target and --task options
They were ignored so far is the config file was specifying a target or a
task, respectively. But the proper priority is (in descending order):

1. command line options
2. environment (KAS_TARGET, KAS_TASK)
3. kas config

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 17:24:36 +01:00
Jan Kiszka
7155769f06 libkas: Ensure Python 3.7 compatibility
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.

At this chance, extract the common compat part into a helper function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:51:45 +01:00
Jan Kiszka
5025f89c01 shell: Drop pointless --target parameter
The target plays no role when invoking the shell because bitbake is not
called.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:17:14 +01:00
Jan Kiszka
ecbda0494b Remove SSH_AGENT_PID forwarding
This variable plays no role in using the agent for authentication, only
in managing its life-cycle.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-17 16:23:46 +02:00
Jan Kiszka
82b0034bac context: Add SSH_AGENT_PID and SSH_AUTH_SOCK to initial environment
This allow to use the host-side SSH agent when cloning protected
repositories via kas. We already do this for the build environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
2018-09-13 11:06:54 +02:00
Daniel Wagner
e1ac18d1ba Release 0.19.0 2018-09-10 12:31:20 +02:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
6349f7ab73 macros: Use new macro loop for repo setup
Divide the SetupRepos command into three parts: init, step and finish
Build a macro-based loop with the step part of the function.

Rewrite this in build.py and shell.py

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
fc9ebf1104 macros: Add loop class for loop functionality
To keep the macro style of the program flow and explicitely state the
command sequence with Macro.add, we need a mechanism that repeats
several steps of the Macro depending on a given condition. This will be
used by the repo checkout logic.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:34 +02:00
Jan Kiszka
c7cc57c42f build: Run bitbake in caller shell if available
When kas is invoked in a shell, pass this property to bitbake, enabling
its fancy UI mode.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:34:44 +02:00
Jan Kiszka
bb103365d2 libkas: Remove unused shell mode from run_cmd
No users, and it even seems broken /wrt universal_newlines.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:18:14 +02:00
Jan Kiszka
d4a615bb0b Command line based configuration file merge
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:

kas build base.yml:board.yml:feature.yml

The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.

In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).

Based on idea by Claudius Heine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:32 +02:00
Jan Kiszka
4a1fba912e repos: Add fallback parameter to get_root_path()
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:25 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
7937c1963e libkas: Remove parameters from get_build_environ()
All of them can be retrieved from the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:02 +02:00
Jan Kiszka
448107781e libkas: Remove config/context parameters from ssh functions
We can obtain the global context now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:48 +02:00
Jan Kiszka
52fdc4f4ae config: Remove context field
No longer used.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:18 +02:00
Jan Kiszka
4acde53b94 Repo: Rework factory() interface
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.

Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:08 +02:00
Jan Kiszka
0c66b24a12 repos: Remove environ parameter from get_root_path()
This is just the default, namely the environment from the global
context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:52:07 +02:00
Jan Kiszka
86dc584284 run_cmd: Default to environment from global context
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:51:34 +02:00
Jan Kiszka
f5cea27269 Move Config creation out of Context constructor
This helps to avoid circular dependencies. It also allows the Config
constructor to make use of the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:50:51 +02:00
Jan Kiszka
594fa97692 context: Provide global instance
This will avoid the increasingly ugly pushing around of a context
instance as parameter to various methods/functions. Everyone can now
simply call get_context() after build or shell called
create_global_context().

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:49:42 +02:00
Jan Kiszka
3fccd1ad3e Move get_distro_id_base from config to context
This is where it is used, only.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:59 +02:00
Jan Kiszka
b10b59e330 context: Stop keeping config_filename, bitbake_target/task
No one uses those properties, and we have them in the config anyway.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:40 +02:00
Jan Kiszka
efbac41590 shell: Unify import pattern
We do not spell out kas. elsewhere, only here so far. Unify that.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:15 +02:00