Commit Graph

376 Commits

Author SHA1 Message Date
Jan Kiszka
f58d44f190 libkas: Properly format stderr dump of failing commands
Inject the linefeed that we removed in f0a18868c9 and add an
introduction line.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
eda4129b54 docs: Mark dynamic configuration format experimental
As the note states, we still need to decide about the future of this
API.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Claudius Heine
e637b2c725 travis: added execution of pytest
Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Claudius Heine
a482a86eb0 tests/includehandler: added initial tests for includehandler
Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Claudius Heine
7ef96e732a includehandler: removed sys.exit(), expanded error checks
The application should not be stopped from this code, removed `sys.exit`
and added exceptions to improve error handling.

Also added some more checks that were missing.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Jan Kiszka
2be4007113 config: Initialize proxy environment prior to fetching includes
In case of ConfigStatic, proxy settings only come from the host
environment. We need them early, prior to fetching the first missing
repo that is pulled via an include.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:10:11 +02:00
Claudius Heine
6f37db72d4 Compare the distro_id in lowercase
For me 'distro.id()' returned 'ubuntu' and thats not equal to 'Ubuntu'.
This patch converts the distro id to lowercase before comparing it.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-06 09:10:09 +02:00
Jan Kiszka
dd04836896 Ignore SIGINT/TERM at kas level, let sub-processes handle it
This seems to be the cleanest and simplest way of properly terminating
kas while one or more sub-processes are running: just ignore the signal.
The sub-processes will get it as well, and they will terminate
themselves properly. kas will get an error then and shut down. Problem
solved.

Forwarding the signal didn't work reliably anyway because we couldn't
count on kas being a process group leader in all cases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:52 +02:00
Jan Kiszka
264be71ebe libkas: Terminate kas on _repo_fetch_async errors
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.

Drop the return value of repos_fetch - no caller expects any.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:49 +02:00
Jan Kiszka
9f4af22ac6 config: Write out config headers in sections
Make the header sections in local.conf and bblayers.conf easier
identifiable by naming them and writing them in chunks, separated by
blank lines. This can help with debugging the resulting configuration.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:04 +02:00
Jan Kiszka
7248d5e1bb Fix termination steps
Another, hopefully better attempt: Ignore the TERM signal in the sending
process so that we can properly clean up all the futures before actually
closing. This also avoids a "Terminated" message even on regular kas
endings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:50 +02:00
Jan Kiszka
adb4007482 libkas: Do not sys.exit from a coroutine
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:36 +02:00
Jan Kiszka
f0a18868c9 libkas: Improve further on error/success output
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:28 +02:00
Jan Kiszka
1e405f155d Add Dockerfile for Isar image builder
Isar requires a number of extra packages compared to a Yocto build. It
also needs a newer e2fsprogs version which can be taken from backports.
Last but not least, its keystone multistrap contains a nasty bug [1]
under in jessie so that we need to pull the updated version from
stretch.

The value of having this image still based on jessie is that it can be
used for both purposes: Yocto (2.1/2.2) and Isar builds. We push it
along the kas core image as kas-isar during the CI run.

Isar still requires some raised privileges. Therefore, the image has to
be started with the additional switches "--cap-add=SYS_ADMIN
--cap-add=MKNOD --device $(/sbin/losetup -f)".

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774698

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:08 +02:00
Mustafa Yücel
b1f343a3fd add yocto 2.1 in comment
Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Mustafa Yücel
cb6a432929 set LANG env for poky builds
Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Mustafa Yücel
8084437140 fix locale warning during docker build
/usr/share/locale/locale.alias is a symlink to /etc/locale.alias and
needed by locale-gen.  /usr/share/locale/locale.alias gets normally
installed by locale package but is filtered in jessie-slim, so re-add
the symlink.

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Claudius Heine
9cf11bb8cf docs: added description of DL_DIR, SSTATE_DIR, TMPDIR env variables
Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:53:17 +02:00
Claudius Heine
afc411f8cd Added link to the documentation in the documentation
Since recursion is recursion and we always have to go deeper, putting a
link to the documentation in the documentation is essential.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:49:03 +02:00
Claudius Heine
30379f7bd7 README.md: Replaced documentation links to a readthedocs link
While the guides are available and renderd on github, there are many
things, that gar generated by sphinx, missing.

This patch replaces those links to one that points to the readthedocs
site.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:49:03 +02:00
Jan Kiszka
e63d64e77d Fix pylint warning about unused paramater
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 15:14:45 +02:00
Claudius Heine
730733fcc9 Made project buildable with rtd
- removed markdown support for documentation, because of import
difficulties.
- converted `intro.md` to restructured text `intro.rst`
- added requirements_rtd.txt that lists the required packages for rtd to
build the documentation.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
7ae7fcdb79 Extended and converted developer guide to ReST
In order to include sphinx generated documentation, restructured text
has to be used. Added some sphinx generated content.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
67ec7f3bfc Extended and converted user guide to ReST
Restructured text provide more options to extend the documentation by
including sphinx generated content.

Also added reference documentation for the static configuration format.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
16f74a55a6 Moved documentation from README.md to sphinx documentation system
Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
32b9b4bb76 Fixed wrapping of cell content in readthedocs style
The readthedocs style for sphinx has an issue where it does not auto
wrap text in table cells. This results in very long cells with a
vertical scrollbar.

This adds a css style sheet to fix this wrapping issue.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
e02bee0eec Added initial sphinx documentation
This commit adds initial files for the Sphinx documentation system and
expanded checkcode.sh to include `doc8` as documentation checker. Also
added this checker to the dependencies for travis CI.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
75c247216e Expanded and documented .gitignore
The current .gitignore is very specific to the project in the current
form and developing process and not commented.

This patch uses a more generic and commented .gitignore for python base
projects that should fit all future needs.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
115625c8fb Added and fixed documentation for kas command line
Documentation for the 'build' and 'shell' command line were missing.
Also fixed kas command line documentation mentioning 'ebs-yocto'.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
c1f2902b21 travis: added codecheck.sh
Added automatic execution of the checkcode.sh script by travis.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
1ef50b559a Added code checker script
This script just calls code checkers pep8 and pylint.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
503fd3d47a Seperated argument parser creation from main kas entry point
With Sphinx it is possible to create the command line documentation
automatically if there is a function that returns just the command line
parser.

Currently the creation of the argument parser is rather entangled with
the rest of kas. This patches seperates this.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
546b51f450 Fixed issues that occured with pylint 1.7.2
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.

This patch fixes these issues:

************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:10:14 +02:00
Claudius Heine
8ba1ce52e8 Allow environment variables to overwrite proxy, target, machine, distro
Changing even simple settings like target or machine requires the
creation of additional configuration files. This does not scale well.

This patch allows the target, machine and distro to be overwritten by
environment variables (called KAS_TARGET, KAS_MACHINE and KAS_DISTRO
respectively).

It also fixes how the environment variables for the proxy settings are
handled. Currently the settings in the config files overwrite the proxy
settings from the environment, but since that would be inconsistent this
patch switches that around. With this patch the environment overwrites
the proxy settings in the configuration file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:10:08 +02:00
Jan Kiszka
0bdd7a8d52 libkas: Rework repo_fetch to parallelized repos_fetch
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.

The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
19ee6edcb1 config: Clean up ConfigStatic constructor
Make it more straightforward by removing complete variable and
terminating the loop on the condition - no more missing repos -
directly. Rename variables to clarify what they reference.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
8e4fbd6d3f libkas: Provide asynchronous interface for command execution
Enhance _stream_subprocess to run_cmd_async, a co-routine variant of
run_cmd that the caller can use to parallelize command execution.
run_cmd becomes a simple wrapper that waits for the async variant to
complete.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
dddbdb0112 libkas: Improve reporting of repo_fetch
Instead of writing "commit", issue a more telling log message if some
repo already contains the required hash.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
e9feb07567 Install SIGINT/TERM handler for backtrace-free CTRL-C
This just avoids dumping a trace in case of CTRL-C.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
94555bb892 libkas: Do not close event loop after each command
Rather close the loop when terminating the kas process.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
34cf6900ca Terminate process group when terminating kas
This addresses the case that someone calls sys.exit() while some
processes are still running.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Daniel Mohr
90f6bafe16 docker: Re-add ssh-client package
This has been removed from the image
by introducing --no-install-recommends.
It's dependency of libkas.py.
2017-06-28 12:40:43 +02:00
Jan Kiszka
40ddeafa22 libkas: Remove redundant local var initialization
retc is already initialized via loop.run_until_complete().

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:38 +02:00
Jan Kiszka
9a300046e6 config: Detect OpenSUSE via distro.id()
distro.id() reports OpenSUSE differently than platform.get_distro_id().
Account for this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:35 +02:00
Jan Kiszka
8bb7257ecc libkas: Fix some comments
Typo and a missing verb.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:33 +02:00
Jan Kiszka
47e8a0730f libkas: Reorder cloning commands and add silent mode consistently
The two code paths are unfortunate. Factor out the difference and use a
single run_cmd invocation. This also adds the forgotten -q when using
a reference directory.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:30 +02:00
Mustafa Yücel
b911942883 dont install (unneeded) recommended/suggested packages
add python3-setuptools for pip3 installation

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Mustafa Yücel
d39978265f use existing entrypoint file
avoid an unnecessary copy to the docker image

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Mustafa Yücel
6b15e5958b add dockerignore
some files or directories (like .gitignore or .git) makes no sense to copy to the docker image

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Jan Kiszka
69a295fa56 docker: Make it slimmer
This safes another 111 MB on the unpacked image.

Suggested-by: Silvano Cirujano-Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-23 08:42:00 +02:00