Commit Graph

309 Commits

Author SHA1 Message Date
Vijai Kumar K
6ed21ad62c Dockerfile: Add gnupg package
- gnupg is currently installed into kas-docker due to reprepro's
requirement. Hence, available only in isar version of kas-docker.
- Add gnupg explicitly in Dockerfile. It is needed for both isar and
yocto builds.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-08 14:39:21 +02:00
Daniel Wagner
907816a5c4 Release 1.0 2019-03-21 13:07:32 +01:00
Jan Kiszka
2fc711a459 Dockerfile.isar: Adjust binfmt setup for qemu-user-static
It turned out [1] that our current way of setting up binfmt_misc during
container start does not reliably work, namely does not overwrite
incompatible host settings. Running dpkg-reconfigure solves this.

To avoid confusing warnings of the kind

update-binfmts: warning: qemu-aarch64 already enabled in kernel.

we filter them out. That will continue to flood the console with
warnings in case the container is started without privileges.

[1] https://groups.google.com/forum/#!msg/isar-users/2G-Diz0BR1Y/Y0X6eHggCgAJ

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-03-20 09:49:34 +01:00
Henning Schild
6fdbe299a4 docker: Take qemu-user-static from buster for Isar image
Doing that will allow building arm64 buster images with Isar. The new
qemu works just fine for all older suites like stretch.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-26 10:24:38 +01:00
Daniel Wagner
5b5694daf7 Release 0.20.1 2019-02-18 15:03:10 +01:00
Jan Kiszka
3eab427cc0 kas-docker: Restore KAS_PREMIRRORS support
Got broken by the merge in 0cb5d17e33.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-18 14:54:46 +01:00
Daniel Wagner
3ce0a1f941 Release 0.20.0 2019-02-18 11:15:00 +01:00
Hosgor, Tolga (CT RDA DS EU TR MTS)
0cb5d17e33 kas-docker: add --no-proxy-from-env option
Allow kas-docker to use proxy configuration of docker client instead of
inheriting proxy configuration from the current environment.

Signed-off-by: Hosgor, Tolga (CT RDA DS EU TR MTS) <tolga.hosgor@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:31:44 +01:00
Jan Kiszka
2f24bdeac2 repos: Fail on unsuccessful patching
There is no good reason to only warn about an incorrect patch entry or
a missing patch file. This just causes CI systems to succeed with a
build - or fail at unexpected places.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:22:55 +01:00
Jan Kiszka
2f7650bb05 Add KAS_PREMIRRORS support
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:35 +01:00
Jan Kiszka
dce5c0029a kas-docker: Enable multi-line variables
We need to double-quote the content of the variables in order to
preserve newlines.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:34 +01:00
Jan Kiszka
7bd738a43c Dockerfile: Patch oe-git-proxy to truly disable proxying by default
Upstream oe-git-proxy has a bug [1] and currently does proxying
unconditionally, just being stopped in the absence of other proxy
settings. Patch in the fix until upstream picks it up.

[1] http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278496.html

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:32 +01:00
Jan Kiszka
1b6e3f2fb9 kas-docker: Pass in NO_PROXY
Required to control oe-git-proxy exceptions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:29 +01:00
Maciej Pijanowski
4d09e411ef docs/userguide.rst: fix typos, improve grammar
Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
2019-01-17 15:17:14 +01:00
Jan Kiszka
dabda7617f kas-docker: Translate all kas files into containter paths
Ensure that all file elements of the kas file parameter are properly
canonicalized and mapped onto the container volume. This is important
when kas-docker is called using relative paths while PWD is not the
repository root.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-01-02 16:56:04 +01:00
Henning Schild
70c568f05f CONTRIBUTING: add subscription links for list to document
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-12-17 10:58:33 +01:00
Henning Schild
ffbfedaa2a kas-docker: fix hg argument for working directory
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
280a695ab3 kas-docker: Pass build and shell options into container
This allows invocations like "kas-docker build --task clean kas.yml".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
5a68998ba7 kas-docker: Allow passing multi-word variables into the container
This is complex, not just because we are passing the arguments to the
trace function, but already because they need to be protected in their
structure when unrolling them at the top level. The trick seems to be
using the argument list via set and "$@" to preserve the individual
words. Due to the trace indirection, we additionally need the single
quotes around the variable values.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
ccc58c7010 kas-docker: Fix SSH_DIR check
We were missing a closing brace, and the line became rather long as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Daniel Wagner
fd58905753 kas-docker: Add missing } to variable derefencing
Reported-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-17 10:50:55 +01:00
Cirujano Cuesta, Silvano
49a6bbfc82 kas-docker: enable passing SSH configs
This commit enables passing the Docker container a directory containing
SSH configuration and optionally SSH identities and list of known SSH
hosts. Basically what you might expect to find in the ~/.ssh directory
of the container.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Suggested-by: Henning Schild <henning.schild@siemens.com>
2018-12-14 14:33:45 +01:00
Daniel Wagner
46c7d6e38c kas: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
    names in their public API using the __all__ attribute.
"""

flake8 reports

./kas/kas.py:51:1: F401 '.build' imported but unused
./kas/kas.py:52:1: F401 '.shell' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
294d4df286 __init__: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
names in their public API using the __all__ attribute.
"""

reported by flake8:

./kas/__init__.py:26:1: F401 '.__version__.__version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__file_version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__compatible_file_version__' imported but unused
./kas/__init__.py:28:1: F401 '.configschema.CONFIGSCHEMA' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
e24f8cff7c scripts: Add flake8 checker
Add code and style checker flake8.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:08 +01:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Claudius Heine
98b4d888bc kas-docker: made loop back device setup more idiot proof
I understood the message:

    Setting up loop device requires root privileges

as suggestion that I should start this script as root and hit
^C before `sudo` could prompt for my password.

This patch removes that line and adds a sudo prompt to elaborate on why
root permissions are needed and what is done with them. It also adds a
check if this was unsuccessful and suggests to the user to do it
themselves. This will also be shown if sudo is not installed.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-11-30 11:17:02 +01:00
Jan Kiszka
90ae592ff1 docker: Align also caller's group ID with container
This ensures that both UID and GID of the builder user inside the
container is aligned with the caller of kas-docker - or that of "docker
run" when "-e GROUP_ID=..." is specified.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-30 11:15:47 +01:00
Jan Kiszka
fe344f93f1 Fix --target and --task options
They were ignored so far is the config file was specifying a target or a
task, respectively. But the proper priority is (in descending order):

1. command line options
2. environment (KAS_TARGET, KAS_TASK)
3. kas config

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 17:24:36 +01:00
Cirujano Cuesta, Silvano
0eb2f5edeb kas-docker: make usage printing POSIX compatible
The usage of 'echo -e' for printing the usage message is not POSIX
conform and is not being supported in the default shell of Debian
(dash).

As the 'Rationale' of the POSIX 'printf' documentation states, it was
created "due to irreconcilable differences in the various versions of
echo extant" and it should provide better compatibility throughout
different shells.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
2018-11-07 14:54:45 +01:00
Jan Kiszka
7155769f06 libkas: Ensure Python 3.7 compatibility
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.

At this chance, extract the common compat part into a helper function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:51:45 +01:00
Jan Kiszka
5025f89c01 shell: Drop pointless --target parameter
The target plays no role when invoking the shell because bitbake is not
called.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:17:14 +01:00
Jan Kiszka
a84e2c2222 kas-docker: Pass ftp_proxy variable
Forgotten because it's rarely used by now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-10-02 10:38:58 +02:00
Jan Kiszka
ecbda0494b Remove SSH_AGENT_PID forwarding
This variable plays no role in using the agent for authentication, only
in managing its life-cycle.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-17 16:23:46 +02:00
Jan Kiszka
82b0034bac context: Add SSH_AGENT_PID and SSH_AUTH_SOCK to initial environment
This allow to use the host-side SSH agent when cloning protected
repositories via kas. We already do this for the build environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
2018-09-13 11:06:54 +02:00
Daniel Wagner
e1ac18d1ba Release 0.19.0 2018-09-10 12:31:20 +02:00
Jan Kiszka
6aa0336d36 Add kas-in-docker invocation script
This shall help to standardize the way how to run kas inside a docker
container on a kas file that is locally available. The pattern is

- checkout repo with kas file(s)
- go to directory where the build output should go into
- call "kas-docker build /path/to/kas.yml"

As building Isar images both require a specific docker image (that
could be changed, though) as well as additional privileges (that needs
to be changed in Isar one day), the option "--isar" selects that mode.

And because the output of an Isar build generally contains root-owned
files, the clean command is added which use docker privileges to clean
the build folder, avoiding a "sudo".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-10 12:14:31 +02:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
249a05267e pylint: disable duplicate code
build.py and shell.py have a long list of explicit macro commands,
which are code duplicates. Since this is wanted coding pattern,
disable the pylint warning about this.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
6349f7ab73 macros: Use new macro loop for repo setup
Divide the SetupRepos command into three parts: init, step and finish
Build a macro-based loop with the step part of the function.

Rewrite this in build.py and shell.py

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
fc9ebf1104 macros: Add loop class for loop functionality
To keep the macro style of the program flow and explicitely state the
command sequence with Macro.add, we need a mechanism that repeats
several steps of the Macro depending on a given condition. This will be
used by the repo checkout logic.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:34 +02:00
Jan Kiszka
c7cc57c42f build: Run bitbake in caller shell if available
When kas is invoked in a shell, pass this property to bitbake, enabling
its fancy UI mode.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:34:44 +02:00
Jan Kiszka
bb103365d2 libkas: Remove unused shell mode from run_cmd
No users, and it even seems broken /wrt universal_newlines.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:18:14 +02:00
Jan Kiszka
d4a615bb0b Command line based configuration file merge
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:

kas build base.yml:board.yml:feature.yml

The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.

In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).

Based on idea by Claudius Heine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:32 +02:00
Jan Kiszka
4a1fba912e repos: Add fallback parameter to get_root_path()
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:25 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
7937c1963e libkas: Remove parameters from get_build_environ()
All of them can be retrieved from the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:02 +02:00
Jan Kiszka
448107781e libkas: Remove config/context parameters from ssh functions
We can obtain the global context now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:48 +02:00
Jan Kiszka
52fdc4f4ae config: Remove context field
No longer used.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:18 +02:00
Jan Kiszka
4acde53b94 Repo: Rework factory() interface
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.

Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:08 +02:00