Commit Graph

346 Commits

Author SHA1 Message Date
Maciej Pijanowski
4d09e411ef docs/userguide.rst: fix typos, improve grammar
Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
2019-01-17 15:17:14 +01:00
Jan Kiszka
dabda7617f kas-docker: Translate all kas files into containter paths
Ensure that all file elements of the kas file parameter are properly
canonicalized and mapped onto the container volume. This is important
when kas-docker is called using relative paths while PWD is not the
repository root.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-01-02 16:56:04 +01:00
Henning Schild
70c568f05f CONTRIBUTING: add subscription links for list to document
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-12-17 10:58:33 +01:00
Henning Schild
ffbfedaa2a kas-docker: fix hg argument for working directory
Signed-off-by: Henning Schild <henning.schild@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
280a695ab3 kas-docker: Pass build and shell options into container
This allows invocations like "kas-docker build --task clean kas.yml".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
5a68998ba7 kas-docker: Allow passing multi-word variables into the container
This is complex, not just because we are passing the arguments to the
trace function, but already because they need to be protected in their
structure when unrolling them at the top level. The trick seems to be
using the argument list via set and "$@" to preserve the individual
words. Due to the trace indirection, we additionally need the single
quotes around the variable values.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Jan Kiszka
ccc58c7010 kas-docker: Fix SSH_DIR check
We were missing a closing brace, and the line became rather long as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-12-17 10:53:20 +01:00
Daniel Wagner
fd58905753 kas-docker: Add missing } to variable derefencing
Reported-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-17 10:50:55 +01:00
Cirujano Cuesta, Silvano
49a6bbfc82 kas-docker: enable passing SSH configs
This commit enables passing the Docker container a directory containing
SSH configuration and optionally SSH identities and list of known SSH
hosts. Basically what you might expect to find in the ~/.ssh directory
of the container.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Suggested-by: Henning Schild <henning.schild@siemens.com>
2018-12-14 14:33:45 +01:00
Daniel Wagner
46c7d6e38c kas: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
    names in their public API using the __all__ attribute.
"""

flake8 reports

./kas/kas.py:51:1: F401 '.build' imported but unused
./kas/kas.py:52:1: F401 '.shell' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
294d4df286 __init__: Better support introspection
PEP8:
"""
To better support introspection, modules should explicitly declare the
names in their public API using the __all__ attribute.
"""

reported by flake8:

./kas/__init__.py:26:1: F401 '.__version__.__version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__file_version__' imported but unused
./kas/__init__.py:27:1: F401 '.__version__.__compatible_file_version__' imported but unused
./kas/__init__.py:28:1: F401 '.configschema.CONFIGSCHEMA' imported but unused

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:09 +01:00
Daniel Wagner
e24f8cff7c scripts: Add flake8 checker
Add code and style checker flake8.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:56:08 +01:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Claudius Heine
98b4d888bc kas-docker: made loop back device setup more idiot proof
I understood the message:

    Setting up loop device requires root privileges

as suggestion that I should start this script as root and hit
^C before `sudo` could prompt for my password.

This patch removes that line and adds a sudo prompt to elaborate on why
root permissions are needed and what is done with them. It also adds a
check if this was unsuccessful and suggests to the user to do it
themselves. This will also be shown if sudo is not installed.

Signed-off-by: Claudius Heine <ch@denx.de>
2018-11-30 11:17:02 +01:00
Jan Kiszka
90ae592ff1 docker: Align also caller's group ID with container
This ensures that both UID and GID of the builder user inside the
container is aligned with the caller of kas-docker - or that of "docker
run" when "-e GROUP_ID=..." is specified.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-30 11:15:47 +01:00
Jan Kiszka
fe344f93f1 Fix --target and --task options
They were ignored so far is the config file was specifying a target or a
task, respectively. But the proper priority is (in descending order):

1. command line options
2. environment (KAS_TARGET, KAS_TASK)
3. kas config

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 17:24:36 +01:00
Cirujano Cuesta, Silvano
0eb2f5edeb kas-docker: make usage printing POSIX compatible
The usage of 'echo -e' for printing the usage message is not POSIX
conform and is not being supported in the default shell of Debian
(dash).

As the 'Rationale' of the POSIX 'printf' documentation states, it was
created "due to irreconcilable differences in the various versions of
echo extant" and it should provide better compatibility throughout
different shells.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
2018-11-07 14:54:45 +01:00
Jan Kiszka
7155769f06 libkas: Ensure Python 3.7 compatibility
In 3.7, async became a reserved word. So we can no longer use it
directly for pre-3.5 compatibility.

At this chance, extract the common compat part into a helper function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:51:45 +01:00
Jan Kiszka
5025f89c01 shell: Drop pointless --target parameter
The target plays no role when invoking the shell because bitbake is not
called.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-11-07 14:17:14 +01:00
Jan Kiszka
a84e2c2222 kas-docker: Pass ftp_proxy variable
Forgotten because it's rarely used by now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-10-02 10:38:58 +02:00
Jan Kiszka
ecbda0494b Remove SSH_AGENT_PID forwarding
This variable plays no role in using the agent for authentication, only
in managing its life-cycle.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-17 16:23:46 +02:00
Jan Kiszka
82b0034bac context: Add SSH_AGENT_PID and SSH_AUTH_SOCK to initial environment
This allow to use the host-side SSH agent when cloning protected
repositories via kas. We already do this for the build environment.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Reviewed-by: Lukasz Majewski <lukma@denx.de>
2018-09-13 11:06:54 +02:00
Daniel Wagner
e1ac18d1ba Release 0.19.0 2018-09-10 12:31:20 +02:00
Jan Kiszka
6aa0336d36 Add kas-in-docker invocation script
This shall help to standardize the way how to run kas inside a docker
container on a kas file that is locally available. The pattern is

- checkout repo with kas file(s)
- go to directory where the build output should go into
- call "kas-docker build /path/to/kas.yml"

As building Isar images both require a specific docker image (that
could be changed, though) as well as additional privileges (that needs
to be changed in Isar one day), the option "--isar" selects that mode.

And because the output of an Isar build generally contains root-owned
files, the clean command is added which use docker privileges to clean
the build folder, avoiding a "sudo".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-10 12:14:31 +02:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
249a05267e pylint: disable duplicate code
build.py and shell.py have a long list of explicit macro commands,
which are code duplicates. Since this is wanted coding pattern,
disable the pylint warning about this.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
6349f7ab73 macros: Use new macro loop for repo setup
Divide the SetupRepos command into three parts: init, step and finish
Build a macro-based loop with the step part of the function.

Rewrite this in build.py and shell.py

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Andreas Reichel
fc9ebf1104 macros: Add loop class for loop functionality
To keep the macro style of the program flow and explicitely state the
command sequence with Macro.add, we need a mechanism that repeats
several steps of the Macro depending on a given condition. This will be
used by the repo checkout logic.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:34 +02:00
Jan Kiszka
c7cc57c42f build: Run bitbake in caller shell if available
When kas is invoked in a shell, pass this property to bitbake, enabling
its fancy UI mode.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:34:44 +02:00
Jan Kiszka
bb103365d2 libkas: Remove unused shell mode from run_cmd
No users, and it even seems broken /wrt universal_newlines.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-09-03 10:18:14 +02:00
Jan Kiszka
d4a615bb0b Command line based configuration file merge
This extends the command line syntax for specifying configuration files.
You can now combine files by concatenating them, separated by colons:

kas build base.yml:board.yml:feature.yml

The motivation for this feature is to avoid having to write tons of
configuration files that perform this combinations statically via
includes.

In order to avoid complications and prevent that users shoot themselves
too easily into their feet, we deny the case of distributing the
configuration files over multiple repositories. Either all files
specified on the command line come from the same repo, or they are all
local (without versioning control).

Based on idea by Claudius Heine.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:32 +02:00
Jan Kiszka
4a1fba912e repos: Add fallback parameter to get_root_path()
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:25 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
7937c1963e libkas: Remove parameters from get_build_environ()
All of them can be retrieved from the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:02 +02:00
Jan Kiszka
448107781e libkas: Remove config/context parameters from ssh functions
We can obtain the global context now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:48 +02:00
Jan Kiszka
52fdc4f4ae config: Remove context field
No longer used.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:18 +02:00
Jan Kiszka
4acde53b94 Repo: Rework factory() interface
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.

Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:08 +02:00
Jan Kiszka
0c66b24a12 repos: Remove environ parameter from get_root_path()
This is just the default, namely the environment from the global
context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:52:07 +02:00
Jan Kiszka
86dc584284 run_cmd: Default to environment from global context
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:51:34 +02:00
Jan Kiszka
f5cea27269 Move Config creation out of Context constructor
This helps to avoid circular dependencies. It also allows the Config
constructor to make use of the global context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:50:51 +02:00
Jan Kiszka
594fa97692 context: Provide global instance
This will avoid the increasingly ugly pushing around of a context
instance as parameter to various methods/functions. Everyone can now
simply call get_context() after build or shell called
create_global_context().

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:49:42 +02:00
Jan Kiszka
3fccd1ad3e Move get_distro_id_base from config to context
This is where it is used, only.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:59 +02:00
Jan Kiszka
b10b59e330 context: Stop keeping config_filename, bitbake_target/task
No one uses those properties, and we have them in the config anyway.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:40 +02:00
Jan Kiszka
efbac41590 shell: Unify import pattern
We do not spell out kas. elsewhere, only here so far. Unify that.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:15 +02:00
Jan Kiszka
5c28272d73 includehandler: Fold GlobalIncludes into IncludeHandler class
No user for this abstraction in sight.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:48:08 +02:00
Jan Kiszka
76e7339fdb repos: Fix false positives when detecting dirty repos
This fixes a regression of 5b85fba0: diff-index is porcelain and seems
to lack some explicit refresh prior to being called. Therefore, it can
incorrectly detect files as being modified although a "git diff" will
not - and will actually also update the status so that the next
diff-index would be fine again.

Fix those false positives while still detecting also cached changes by
using git status -s instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:45:15 +02:00
Jan Kiszka
3744ff329a config: Account for multiple ID_LIKE words in os-release
The ID_LIKE tag that get_distro_id_base also queries may contain
multiple, space-separated values. Opensuse Leap 15.0 is exploiting that,
e.g. ("suse opensuse").

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-23 16:53:25 +02:00
Jan Kiszka
e1cc66510c docs: Fix typo in KAS_REPO_REF_DIR description
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-23 16:45:22 +02:00
Andreas Reichel
7b18e5ec3b Refactor repo checkout
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.

This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.

This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.

The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).

Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-08-23 16:29:56 +02:00
Jan Kiszka
3bada55d30 Docker: Update to Debian stretch
Time to move forward, buster is almost in sight. We may break some older
Yocto versions this way, but they can always use older images.

Along this update, this migrates the locale setup towards the
recommendation on https://docs.docker.com/samples/library/debian/#locales.
Furthermore, DEBIAN_FRONTEND=noninteractive is moved to the top-level
Dockerfile - it's generic.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-07-19 09:25:34 +02:00