Commit Graph

139 Commits

Author SHA1 Message Date
Jan Kiszka
cd076101f4 docker: Avoid error messages on container restarts
When the user restarts an already existing kas container, errors are
thrown because of existing folders or users. One example is the usage
of gitlab-ci runners.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-11-22 12:38:07 +01:00
Daniel Wagner
7aff26d105 libkas: Add /sbin and /usr/sbin to PATH
/sbin and /usr/sbin might contain tools Yocto depens on, e.g. 'ip'.

"""
The following required tools (as specified by HOSTTOOLS) appear to be
unavailable in PATH, please install them in order to proceed:

	ip
"""

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-11-20 14:12:28 +01:00
Claudius Heine
5fac90611f setup: added python 3.6 line
Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-10 09:21:42 +01:00
Claudius Heine
6d6e7ff2e7 setup: added python version requirement
Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-10 09:21:37 +01:00
Cedric Hombourger
06b05a35b6 config: support major distro variants
Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
2017-11-09 09:26:43 +01:00
Claudius Heine
8119dff5f0 scripts: pep8 is renamed to pycodestyle
Fix it to use the new name.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-08 09:46:00 +01:00
Claudius Heine
fa1575790f multiconfig: implement initial support
This patch adds multiconfig support to kas by defining the
"BBMULTICONFIG" variable in the "local.conf" file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-11-08 09:42:56 +01:00
[ext] Henning Schild
9370bd6267 docker: isar: configure sudo to keep proxy env variables
Keep proxy related env variables when running commands with sudo.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2017-11-02 16:16:34 +01:00
Daniel Wagner
f24b341745 Release 0.14.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-11-02 16:02:33 +01:00
Claudius Heine
aa79374f7e config: debug message for list of missing repo names
Its useful to know the missing repos that are required to
create the full configuration as they are detected.

This might help to find problems within the configuration and kas code
in the future.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-19 17:15:26 +02:00
Claudius Heine
3eff938a54 kas: added version output when starting kas
The used version of kas is useful to have in resulting build logs.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-19 17:15:26 +02:00
Claudius Heine
5d7fae7094 configschema: Added copyright and license variables
Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-18 13:49:09 +02:00
Claudius Heine
41fc9c8261 __version__: Added note about updating docs/format-changelog.rst
Since its easy to forget to update the documentation when changing the
configuration file format version, this patch adds a small note to
remind the developer about it.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-18 13:49:06 +02:00
Claudius Heine
4940d4d580 includehandler: remove possible duplicates from the missing_repos list
I was observed that in some more evolved configurations the build
stopped because kas wanted to clone a git repository multiple times
concurrently. This failed.

This patch should prevent these issues by removing all possible
duplicates from the missing_repos list prior to returning it to the
configuration.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-18 10:04:06 +02:00
Claudius Heine
9753b561d2 docs/format-changelog.rst: Updated to include the recent changes
This patch updated the 'docs/format-changelog.rst' file to include
information about the configuration file version 3 and 4.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-16 11:05:16 +02:00
Claudius Heine
c1bfeefa04 docs/devguide: Added the 'no_proxy' parameter to the description
In environments where proxies are used, there are usually some addresses
that should be reached without proxy. So it makes sense to document
that.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-16 11:03:20 +02:00
Claudius Heine
845d212f02 Added 'ftp_proxy' environment variable support
Some bitbake recipes use FTP to fetch objects. This patch adds support
for the ftp_proxy variable to kas. Is now usable in the same manner as
the 'http_proxy' variable

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-16 11:03:07 +02:00
Claudius Heine
f140d7905c config: includehandler: remove other checks that are no longer needed
Using the jsonschema module makes some checks in the includehandler
redundant and can be removed. This patch does this.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:19 +02:00
Claudius Heine
75fa095aa5 config: Added schema validation for configuration files
Currently many of the error messages regarding the configuration file
are not really helpful.

To improve reporting this patch adds validation of the configuration
files using the jsonschema module.

Example:

  Forgetting ':' at the end of a layer definition, like this:

    repos:
      meta-iot2000:
        layers:
          meta-iot2000-example
                              ^

  Exception before:

    TypeError: string indices must be integers

  Exception after:

    Validation Error:
     'meta-iot2000-example' is not of type 'object'

     Failed validating 'type' in schema['properties']['repos']['additionalProperties']['properties']['layers']:
         {'additionalProperties': {'oneOf': [{'type': 'null'},
                                             {'type': 'integer'},
                                             {'type': 'boolean'},
                                             {'type': 'string'}]},
          'type': 'object'}

     On instance['repos']['meta-iot2000']['layers']:
         'meta-iot2000-example'

This patch adds 'jsonschema' as an additional dependency.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:10 +02:00
Claudius Heine
ab8e4cd375 config: Implemented multi-target support
With this patch support for building multiple targets in parallel with
bitbake is now supported.

Changes:

  - Its now possible to state a list of targets under the target key in
    the configuration file.

    Example:

        target:
          - product-image
          - product-update-image

  - Its now possible to define multiple targets in the kas command line.

    Example:

        $ kas build --target product-image --target product-update-image \
          kas.yml

  - Its now possible to define multiple targets via the environment:

    Example:

        $ export KAS_TARGET="product-image product-update-image"
        $ kas build kas.yml

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:11:00 +02:00
Claudius Heine
f7d2e5ae20 config: added debug output of complete configuration
It might be useful to see the merged configuration in case of debugging.
This patch adds it to the debug output log level.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-10-13 14:10:47 +02:00
Daniel Wagner
15e6fb0a81 Release 0.13.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-10-10 09:13:50 +02:00
Jan Kiszka
4f76e47090 config: Belatedly increment file version to account for task field
Captain Hindsight: We should have never committed 13a87cb8c7 ("Allow
to define task in config and environment") without forwarding the file
version. Without this, no config file writer can tell kas versions
apart that support task control from those that don't.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-10-10 09:12:20 +02:00
Daniel Wagner
d1968a7f84 Release 0.12.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-10-09 17:53:40 +02:00
Daniel Wagner
133f1be38a libcmds: Remove unused pre and post hooks
9e136e532d ("config: Remove dynamic configuration variant")
removed the posibility to add pre and post hooks. Remove the remaining
bits from libcmds.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-09-27 10:49:35 +02:00
Jan Kiszka
bfd7770b81 config: Fold ConfigStatic into Config
Now that we no longer have two different configuration variants, we can
remove the abstract class and make Config the primary interface.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-27 10:48:59 +02:00
Jan Kiszka
9e136e532d config: Remove dynamic configuration variant
We no longer see an urging use case for it. All field scenarios can
perfectly be modeled with the static format.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-27 10:48:35 +02:00
Jan Kiszka
13a87cb8c7 Allow to define task in config and environment
Based on original patch by Sascha Weisenberger.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-25 09:33:43 +02:00
Henning Schild
d06bdc4781 libkas: do not use absolute paths when executing programs
Using absolute paths breaks on systems that place the binaries in
another directory and it defeats local overlays with PATH.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
2017-09-11 10:19:09 +02:00
Jan Kiszka
aa3d109f0b docker: Add binfmt-support for Isar cross-building
Additionally requires to run the container with --privileged - not nice but
unavoidable right now, see also
https://groups.google.com/forum/#!topic/isar-users/Un75m49InjY

Also note that running containers with different binfmt settings
concurrently will cause troubles. Same for the case that the host
requires deviating settings. Use carefully.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-09-06 17:25:41 +02:00
Jan Kiszka
affd1184e7 entry: Consistently call python3
This may not be needed in practice, but it is cleaner to align all
shebangs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-08-31 22:31:41 +02:00
Claudius Heine
8745aabb62 kas/shell: add 'keep-config-unchanged' switch
This patch adds a command line argument to the shell plugin, that allows to
enter the shell without changing the bitbake configuration.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-08-04 09:53:50 +02:00
Claudius Heine
11f047a408 kas/shell: Prepare complete bitbake configuration before entering shell
Before this patch, starting the shell in a clean environment, does not
checkout all repository or configures all layers correctly.

This patch adds the missing checkout and configuration steps to the
shell plugin.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-08-01 09:35:34 +02:00
Claudius Heine
bb36b907f0 Dockerfile: added less and vim
If you are using kas interactivly vim and less are very convenient
tools to have.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-08-01 09:35:17 +02:00
Jan Kiszka
2d6b1ce6a0 libkas: Detect and report multiple init scripts
Having more than one them is a sign of serious problems in the
configuration. Better catch that early and fail loudly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:29:54 +02:00
Jan Kiszka
eb284afb8e libkas: Detect error of init script
Do not swallow errors of the init script. Rather leave our wrapper
immediately and let run_cmd report the splash and terminate kas.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-28 17:11:54 +02:00
Daniel Wagner
39dba6fe75 Release 0.11.0
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-07-24 17:17:36 +02:00
Jan Kiszka
8d5ce95347 docker: Allow to call with USER_ID=0
When we run as root on the host and want to allow the builder to do the
same, e.g. to access root-owned volumes, accept USER_ID=0 to express
this.

This allows to tell the user to call "docker run -e USER_ID=$(id -u)",
and it will always reflect the calling context's permissions into the
container.

Reported-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-21 15:35:05 +02:00
Jan Kiszka
bb853cec02 config: Write proper info message about root of URL-less repos
Instead of flooding the log with misleading error messages, always print
a note which path is used as root for repositories that have no URLs.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-21 15:34:20 +02:00
Claudius Heine
96034e2d9a config: Allow in-tree repos not to be in a git repo.
If the repo, where the configuration file resides in (in-tree
configuration) is not a git repository, take the directory where the
configuration file is located at as the path of this repo instead of failing.

This patch does not need to raise the configuration file version,
because it does not depend on the configuration file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-07-20 16:08:34 +02:00
Jan Kiszka
171440c0a1 docker: Start bash if no command is provided
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
bd61564b58 docker: Set proxy for pip explicitly
Some versions/configurations of pip do not seem to pick up the proxy
settings from the environment. Help out via the command line. It's a nop
when https_proxy is unset.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
047a8b6878 docker: Add oe-git-proxy to base image
Useful to bypass nasty firewalls via http proxies while pulling from
git repos using the native protocol. In order to use oe-git-proxy,
define or clear NO_PROXY according to local needs in the CI script or
via --env on docker run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Jan Kiszka
100c7e047f libkas: Pass git proxy related environment variables
This allows to define GIT_PROXY_COMMAND="oe-git-proxy" outside of kas
and pass this to the build. Now we can pull from native git repos when
there is only a http proxy available.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 16:08:34 +02:00
Daniel Wagner
a4c424c253 Add missing URLs and maintainer information to setup.py
With these changes it is possible to upload the package to PyPI live.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-07-20 16:08:28 +02:00
Jan Kiszka
c9326cc1ed Write deterministic local.conf and bblayers.conf
Sort the header sections by their keys and the layers by their names.
This will re-enable writing deterministic config files, and bitbake can
use its cache again, accelerating rebuild startup times a lot.

While at it, indent the layers in bblayers.conf nicely.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:32:06 +02:00
Jan Kiszka
ef0ef7c60c kas: Also print config format version on --version
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:31:34 +02:00
Jan Kiszka
2a618686f1 Switch to separate config file version
Use a separately incremented integer to track the configuration file
format version. We start with 2 due to the change that 23c3a951f6
introduced. 1 is declared to be equivalent to the original '0.10'.

The separate versioning has the advantage of being able to increment it
already during the development cycle, and using it with config files
that test/exploit the new format.

Using an integer has the advantages of a) differentiating it clearly
from the now independent kas version and b) simplifying the version
parsing. We can now also remove the string type restriction.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:31:23 +02:00
Jan Kiszka
5478e43f8a docs: Fix and reformat environment table
Sphinx complained:

docs/userguide.rst:66: ERROR: Malformed table.
Column span alignment problem in table line 8.

Avoid this by reformatting to a grid table, also adjusting the content a
bit.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:33 +02:00
Jan Kiszka
72d2eff6bc docs: Remove unused githubpages extension
No need to require its availability.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-07-20 15:29:21 +02:00