Commit Graph

349 Commits

Author SHA1 Message Date
Jan Kiszka
171440c0a1 docker: Start bash if no command is provided
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
bd61564b58 docker: Set proxy for pip explicitly
Some versions/configurations of pip do not seem to pick up the proxy
settings from the environment. Help out via the command line. It's a nop
when https_proxy is unset.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
047a8b6878 docker: Add oe-git-proxy to base image
Useful to bypass nasty firewalls via http proxies while pulling from
git repos using the native protocol. In order to use oe-git-proxy,
define or clear NO_PROXY according to local needs in the CI script or
via --env on docker run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
100c7e047f libkas: Pass git proxy related environment variables
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Daniel Wagner
a4c424c253 Add missing URLs and maintainer information to setup.py
With these changes it is possible to upload the package to PyPI live.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-07-20 16:08:28 +02:00
Jan Kiszka
c9326cc1ed Write deterministic local.conf and bblayers.conf
Sort the header sections by their keys and the layers by their names.
This will re-enable writing deterministic config files, and bitbake can
use its cache again, accelerating rebuild startup times a lot.

While at it, indent the layers in bblayers.conf nicely.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:32:06 +02:00
Jan Kiszka
ef0ef7c60c kas: Also print config format version on --version
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:31:34 +02:00
Jan Kiszka
2a618686f1 Switch to separate config file version
Use a separately incremented integer to track the configuration file
format version. We start with 2 due to the change that 23c3a951f6
introduced. 1 is declared to be equivalent to the original '0.10'.

The separate versioning has the advantage of being able to increment it
already during the development cycle, and using it with config files
that test/exploit the new format.

Using an integer has the advantages of a) differentiating it clearly
from the now independent kas version and b) simplifying the version
parsing. We can now also remove the string type restriction.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:31:23 +02:00
Jan Kiszka
5478e43f8a docs: Fix and reformat environment table
Sphinx complained:

docs/userguide.rst:66: ERROR: Malformed table.
Column span alignment problem in table line 8.

Avoid this by reformatting to a grid table, also adjusting the content a
bit.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:33 +02:00
Jan Kiszka
72d2eff6bc docs: Remove unused githubpages extension
No need to require its availability.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:21 +02:00
Jan Kiszka
4f5cd3ca5b README: Convert to structured text
This is a preferred format of Python READMEs and also properly rendered
by github.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:12 +02:00
Jan Kiszka
a96197716b README: Remove reference to supported Yocto versions
Easy to get out of sync with duplicate info in docs/instro.rst.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:28:36 +02:00
Claudius Heine
23c3a951f6 config: ConfigStatic should only try to fetch missing known repos
When a configuration file depends on a include from a repo, which is
defined in another included file, a 'KeyError' occures.

This fixes this, so that only known repos are tried to be fetched.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-12 10:22:15 +02:00
Jan Kiszka
2a4a34865a docs: Mention the official docker hub path
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-12 10:22:15 +02:00
Jan Kiszka
f3e8d9ffd8 includehandler: Format LoadConfigException message nicely
This exception is always thrown with both a message and a file name.
Combine them to a single string, instead of throwing tuples.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-12 10:22:15 +02:00
Jan Kiszka
b30a8ebc3b travis: Push next builds under separate tag
This prevents overwriting next builds with master output, and vice
versa. Users can now decide if they want more or less bleeding edge.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-12 10:22:15 +02:00
Jan Kiszka
1e11d314e8 travis: Fix upload of releases
Yeah, that was "too simple to possibly fail". Consequently, the release
revealed two issues: The script only looked for annotated tags but they
are not used so far. Moreover, the TRAVIS_BRANCH name of a tag is that
tag and neither "master" or "next".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-12 10:22:15 +02:00
Daniel Wagner
6bc7715f15 Release 0.10.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-07-10 12:27:26 +02:00
Jan Kiszka
c329e850d4 travis: Model via stages
This avoids building the docker images for each Python version test.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-08 22:08:50 +02:00
Jan Kiszka
49cb9b7a7b Fix overwriting of target via command line with static configs
ConfigStatic does not respect the --target overwrite so far. Fix this
by pulling the default definition into ConfigPython (ConfigStatic
already does that via the Config base class) and set the target provided
via the command line in the static config after parsing all files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-08 22:08:47 +02:00
Jan Kiszka
7f1ccba5ea docker: Warn about broken host setup based on aufs
There is a nasty problem with legacy aufs: wic tries to find out the
block size of the filesystem that holds the partition images, but aufs
does not seem to implement this properly, returning 0 at least on Debian
Jessie. That makes wic become upset and through a division-by-zero
exception soon after.

Catch this case by warning the user about the inappropriate docker
setup during container start.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-08 16:21:11 +02:00
Jan Kiszka
8e3862112c libkas: Fix false positive of pylint
As suggested by Claudius Heine, reorder the blocks so that we can add
the pylint exception to the if half.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Claudius Heine
f10718043e config: Removes file extension check for ConfigStatic
The `load_config` function in the includehandler already checks for the
file extension, so checking it before that, is redundant.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:57:21 +02:00
Jan Kiszka
f58d44f190 libkas: Properly format stderr dump of failing commands
Inject the linefeed that we removed in f0a18868c9 and add an
introduction line.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Jan Kiszka
eda4129b54 docs: Mark dynamic configuration format experimental
As the note states, we still need to decide about the future of this
API.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-07 16:57:21 +02:00
Claudius Heine
e637b2c725 travis: added execution of pytest
Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Claudius Heine
a482a86eb0 tests/includehandler: added initial tests for includehandler
Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Claudius Heine
7ef96e732a includehandler: removed sys.exit(), expanded error checks
The application should not be stopped from this code, removed `sys.exit`
and added exceptions to improve error handling.

Also added some more checks that were missing.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-07 16:56:49 +02:00
Jan Kiszka
2be4007113 config: Initialize proxy environment prior to fetching includes
In case of ConfigStatic, proxy settings only come from the host
environment. We need them early, prior to fetching the first missing
repo that is pulled via an include.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:10:11 +02:00
Claudius Heine
6f37db72d4 Compare the distro_id in lowercase
For me 'distro.id()' returned 'ubuntu' and thats not equal to 'Ubuntu'.
This patch converts the distro id to lowercase before comparing it.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-06 09:10:09 +02:00
Jan Kiszka
dd04836896 Ignore SIGINT/TERM at kas level, let sub-processes handle it
This seems to be the cleanest and simplest way of properly terminating
kas while one or more sub-processes are running: just ignore the signal.
The sub-processes will get it as well, and they will terminate
themselves properly. kas will get an error then and shut down. Problem
solved.

Forwarding the signal didn't work reliably anyway because we couldn't
count on kas being a process group leader in all cases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:52 +02:00
Jan Kiszka
264be71ebe libkas: Terminate kas on _repo_fetch_async errors
This behavior got lost through 0bdd7a8d52. Specifically, errors during
git clone no longer shut down kas. Fix this by reporting back an error
code from _repo_fetch_async and terminate in repos_fetch if it's
non-zero.

Drop the return value of repos_fetch - no caller expects any.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:49 +02:00
Jan Kiszka
9f4af22ac6 config: Write out config headers in sections
Make the header sections in local.conf and bblayers.conf easier
identifiable by naming them and writing them in chunks, separated by
blank lines. This can help with debugging the resulting configuration.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-06 09:05:04 +02:00
Jan Kiszka
7248d5e1bb Fix termination steps
Another, hopefully better attempt: Ignore the TERM signal in the sending
process so that we can properly clean up all the futures before actually
closing. This also avoids a "Terminated" message even on regular kas
endings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:50 +02:00
Jan Kiszka
adb4007482 libkas: Do not sys.exit from a coroutine
That only creates exceptions and complaints if more routines are running
or we are already in the exiting process. Rather leave the coroutine and
catch the error on the waiting end.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:36 +02:00
Jan Kiszka
f0a18868c9 libkas: Improve further on error/success output
Don't add a newline to stderr dumps on failing commands, the logger does
that already. And don't print success messages if the fetch command
actually failed.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:28 +02:00
Jan Kiszka
1e405f155d Add Dockerfile for Isar image builder
Isar requires a number of extra packages compared to a Yocto build. It
also needs a newer e2fsprogs version which can be taken from backports.
Last but not least, its keystone multistrap contains a nasty bug [1]
under in jessie so that we need to pull the updated version from
stretch.

The value of having this image still based on jessie is that it can be
used for both purposes: Yocto (2.1/2.2) and Isar builds. We push it
along the kas core image as kas-isar during the CI run.

Isar still requires some raised privileges. Therefore, the image has to
be started with the additional switches "--cap-add=SYS_ADMIN
--cap-add=MKNOD --device $(/sbin/losetup -f)".

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774698

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-29 09:09:08 +02:00
Mustafa Yücel
b1f343a3fd add yocto 2.1 in comment
Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Mustafa Yücel
cb6a432929 set LANG env for poky builds
Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Mustafa Yücel
8084437140 fix locale warning during docker build
/usr/share/locale/locale.alias is a symlink to /etc/locale.alias and
needed by locale-gen.  /usr/share/locale/locale.alias gets normally
installed by locale package but is filtered in jessie-slim, so re-add
the symlink.

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-29 09:05:02 +02:00
Claudius Heine
9cf11bb8cf docs: added description of DL_DIR, SSTATE_DIR, TMPDIR env variables
Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:53:17 +02:00
Claudius Heine
afc411f8cd Added link to the documentation in the documentation
Since recursion is recursion and we always have to go deeper, putting a
link to the documentation in the documentation is essential.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:49:03 +02:00
Claudius Heine
30379f7bd7 README.md: Replaced documentation links to a readthedocs link
While the guides are available and renderd on github, there are many
things, that gar generated by sphinx, missing.

This patch replaces those links to one that points to the readthedocs
site.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-29 08:49:03 +02:00
Jan Kiszka
e63d64e77d Fix pylint warning about unused paramater
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 15:14:45 +02:00
Claudius Heine
730733fcc9 Made project buildable with rtd
- removed markdown support for documentation, because of import
difficulties.
- converted `intro.md` to restructured text `intro.rst`
- added requirements_rtd.txt that lists the required packages for rtd to
build the documentation.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
7ae7fcdb79 Extended and converted developer guide to ReST
In order to include sphinx generated documentation, restructured text
has to be used. Added some sphinx generated content.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
67ec7f3bfc Extended and converted user guide to ReST
Restructured text provide more options to extend the documentation by
including sphinx generated content.

Also added reference documentation for the static configuration format.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
16f74a55a6 Moved documentation from README.md to sphinx documentation system
Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
32b9b4bb76 Fixed wrapping of cell content in readthedocs style
The readthedocs style for sphinx has an issue where it does not auto
wrap text in table cells. This results in very long cells with a
vertical scrollbar.

This adds a css style sheet to fix this wrapping issue.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
e02bee0eec Added initial sphinx documentation
This commit adds initial files for the Sphinx documentation system and
expanded checkcode.sh to include `doc8` as documentation checker. Also
added this checker to the dependencies for travis CI.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00