Commit Graph

346 Commits

Author SHA1 Message Date
Claudius Heine
1ef50b559a Added code checker script
This script just calls code checkers pep8 and pylint.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
503fd3d47a Seperated argument parser creation from main kas entry point
With Sphinx it is possible to create the command line documentation
automatically if there is a function that returns just the command line
parser.

Currently the creation of the argument parser is rather entangled with
the rest of kas. This patches seperates this.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:12:15 +02:00
Claudius Heine
546b51f450 Fixed issues that occured with pylint 1.7.2
When updating the previously used version of pylint 1.6.5 to 1.7.2
some new issues where found.

This patch fixes these issues:

************* Module kas.config
E:402, 0: Bad option value 'redefined-variable-type' (bad-option-value)
************* Module kas.includehandler
E: 33, 0: No name 'version' in module 'distutils' (no-name-in-module)
E: 33, 0: Unable to import 'distutils.version' (import-error)
R:239,12: Unnecessary "else" after "return" (no-else-return)
************* Module kas.libkas
C:214, 7: Do not use `len(SEQUENCE)` as condition value
(len-as-condition)
************* Module kas.repos
R: 54,12: Unnecessary "else" after "return" (no-else-return)

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:10:14 +02:00
Claudius Heine
8ba1ce52e8 Allow environment variables to overwrite proxy, target, machine, distro
Changing even simple settings like target or machine requires the
creation of additional configuration files. This does not scale well.

This patch allows the target, machine and distro to be overwritten by
environment variables (called KAS_TARGET, KAS_MACHINE and KAS_DISTRO
respectively).

It also fixes how the environment variables for the proxy settings are
handled. Currently the settings in the config files overwrite the proxy
settings from the environment, but since that would be inconsistent this
patch switches that around. With this patch the environment overwrites
the proxy settings in the configuration file.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-28 15:10:08 +02:00
Jan Kiszka
0bdd7a8d52 libkas: Rework repo_fetch to parallelized repos_fetch
Building on top of run_cmd_async, this reworks repo_fetch to a
repository list fetcher repos_fetch that runs those operations in
parallel.

The two users, ReposFetch and ConfigStatic, are converted to exploit
this parallelization.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
19ee6edcb1 config: Clean up ConfigStatic constructor
Make it more straightforward by removing complete variable and
terminating the loop on the condition - no more missing repos -
directly. Rename variables to clarify what they reference.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
8e4fbd6d3f libkas: Provide asynchronous interface for command execution
Enhance _stream_subprocess to run_cmd_async, a co-routine variant of
run_cmd that the caller can use to parallelize command execution.
run_cmd becomes a simple wrapper that waits for the async variant to
complete.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
dddbdb0112 libkas: Improve reporting of repo_fetch
Instead of writing "commit", issue a more telling log message if some
repo already contains the required hash.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
e9feb07567 Install SIGINT/TERM handler for backtrace-free CTRL-C
This just avoids dumping a trace in case of CTRL-C.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
94555bb892 libkas: Do not close event loop after each command
Rather close the loop when terminating the kas process.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Jan Kiszka
34cf6900ca Terminate process group when terminating kas
This addresses the case that someone calls sys.exit() while some
processes are still running.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 14:51:29 +02:00
Daniel Mohr
90f6bafe16 docker: Re-add ssh-client package
This has been removed from the image
by introducing --no-install-recommends.
It's dependency of libkas.py.
2017-06-28 12:40:43 +02:00
Jan Kiszka
40ddeafa22 libkas: Remove redundant local var initialization
retc is already initialized via loop.run_until_complete().

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:38 +02:00
Jan Kiszka
9a300046e6 config: Detect OpenSUSE via distro.id()
distro.id() reports OpenSUSE differently than platform.get_distro_id().
Account for this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:35 +02:00
Jan Kiszka
8bb7257ecc libkas: Fix some comments
Typo and a missing verb.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:33 +02:00
Jan Kiszka
47e8a0730f libkas: Reorder cloning commands and add silent mode consistently
The two code paths are unfortunate. Factor out the difference and use a
single run_cmd invocation. This also adds the forgotten -q when using
a reference directory.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-28 11:06:30 +02:00
Mustafa Yücel
b911942883 dont install (unneeded) recommended/suggested packages
add python3-setuptools for pip3 installation

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Mustafa Yücel
d39978265f use existing entrypoint file
avoid an unnecessary copy to the docker image

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Mustafa Yücel
6b15e5958b add dockerignore
some files or directories (like .gitignore or .git) makes no sense to copy to the docker image

Signed-off-by: Mustafa Yücel <mustafa.yuecel@siemens.com>
2017-06-26 09:29:07 +02:00
Jan Kiszka
69a295fa56 docker: Make it slimmer
This safes another 111 MB on the unpacked image.

Suggested-by: Silvano Cirujano-Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-23 08:42:00 +02:00
Claudius Heine
58b5014345 Add install_requires in setup.py
The necessary dependencies should also be defined in the setup.py file,
so that they are automatically installed by `pip`.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-23 08:36:48 +02:00
Claudius Heine
a618ea95bd Fixes install instructions in README.md
Without period pip complains.

Also rather use `pip3` instead of `pip` because some distributions still
default to python 2 and pip for python 2.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-23 08:36:28 +02:00
Claudius Heine
17691d2756 Refactored fetching and checking out of missing repos in kas.config
The current implementation did some more hackish solution to work
around some of kas infrastructure, without changing to much.

This patch cleans this up and therefor remove the more obsure part
of the include mechansim.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-22 12:33:00 +02:00
Jan Kiszka
013af80599 Add support for Isar build system
Isar allows to build Debian-based images using bitbake and the usual
layer structures. It's very similar to OE, we just need to account for
a different init script name.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-22 10:50:40 +02:00
Daniel Mohr
f3cee3b296 ci: Add Travis build status batches to README
The batches also provide a link to the build
status on travis-ci.org

Signed-off-by: Daniel Mohr <daniel.mohr@evosoft.com>
2017-06-22 10:24:51 +02:00
Claudius Heine
7451b35f46 Updated Readme.md to include documentation about include mechanism
This patch updates the Readme.md to describe the include mechanism
and to use YAML instead of JSON.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Claudius Heine
abc80a8c59 Implement file version and a check when using it
If the configuration file structure changes over time, it might
introduce incompatibilities.

This patch introduces versioning of configuration file.

Configuration files needs to add a 'version' entry into their 'header'
like this:

header:
  version: '0.9'

After loading this file, the version is checked agains kas version.

Every version 'M.m.p' of kas is allowed to load configuration files with
version 'M.m' and all versions backwards up to and including
the '__compatible_version__' from kas/__version__.py

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Claudius Heine
a65a85b373 Added file header section and put 'includes' under it.
In the current configuration file we mix between entries that are file
specific (like includes and later version) and entries that are used by
the rest of kas, that are not file specific (machine, target, repos,
etc.) It might not be obvious to differenciate between those entries.

This patch introduces a header section into the configuration file, to
contain every setting that is specific to the file and not the complete kas
configuration. The 'includes' statement is moved there, because its file
specific.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Claudius Heine
34dd07e76e added initial implementation of a include handler
Splitting configuration files into multiple files and implementing an
include mechanism allows to handle configurations more flexible and
allowing following scenarios:

 - including kas configuration file from external meta layer into
   own project
 - splitting many different similar configurations into multiple files
   with only a couple of common files

To include file its necessary to add a 'includes' entry into the kas
configuration. To include files from the same repo, do it like this:

-----
includes:
  - ../base/base_include.yml
-----

The path is relative to the current configuration file. If they start
with a path seperator ("/") they are absolute.

To include files from a different repository, do it like this:

-----
includes:
  - file: bsps/my_machine_include.yml
    repo: collected_machines

repos:
  collected_machines:
    url: https://url.to.git.repo/
    revspec: master
-----

You have to make sure that the repository definition is available
in your current file, or in any include that is available at this
point.

Yaml ("*.yml") and Json ("*.json") files are supported.

Included in this patch are also some changes to the configuration
file structure. Here is an overview:

The value of the 'repos' key is a dictionary that maps a repo identifier
to a dictionary that contains 5 entries:

- url: The url to the repository. If that is missing, no git operations
are used
- refspec: The git refspec of the repository that is used. If that is
missing the latest commit of the default branch is used.
- name: The is defines under which name the repository is stored. If
that is missing the repository identifier is used
- path: The path where the repository is stored. If no url is given and
a path is missing, the repository is referencing the repository under
which the configuration file is stored. If no url is given and a path is
specified, the repository is referencing a directory where layers might
be stored. If an url is specified, path can be used to overwrite the
default (kas_work_dir + repo.name) checkout directory.
- layers: This contains a dictionary of layers, that should be included
into the bblayers.conf. The keys are paths relative to the repository
and the values can be used to exclude layers if they are one of
"excluded", "disabled", "false", "0", "n", or "no". Also boolean values
are accepted. Any other value, including "None" means that this layer is
included into the bblayers.conf. If "layers" is missing or empty, the
repository itself is included into the bblayers. If this is specified, the
repository itself is not included into the bblayers.conf.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Claudius Heine
c950de46f9 Renamed sublayers back to layers
It was decided that 'sublayers' is to confusing, because this term is
not used in the bitbake/openembedded context.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Claudius Heine
33a21c8d0d run pylint3 and fixed report
This commit adds a pylint configuration and fixed all found issues.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-21 17:30:51 +02:00
Jan Kiszka
6bc8e08459 Fix pep8 reports
Zero findings after this commit. We should try to keep that level.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-21 17:30:51 +02:00
Jan Kiszka
062e919295 docker: Clean apt caches after installation
Combining both installation steps and cleaning caches afterwards saves
about 12 MB.

At this chances reformat to have only one command per line.

Suggested-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-20 15:56:31 +02:00
Jan Kiszka
1aba8f80ef docker: Remove libsdl and xterm
These packages date back from the (pre-public) times when the list was
also used for developer host setups.

Suggested-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-20 15:56:29 +02:00
Jan Kiszka
b355b42136 docker: Switch to Debian 8
Friendlier than Ubuntu and roughly the same size.

We just need to add lsb-release to the dependencies so that yocto finds
a known distro and install gosu manually as only Debian 9 started to
package it.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-20 15:56:26 +02:00
Jan Kiszka
c65657cac6 Remove state tracking, use KAS_WORK_DIR env variable
Config.has_changed didn't work reliably so far, so the state cache was
only allowing to associate a work directory with configuration file.
However this was rather unintuitive, specifically when moving the work
dir to a different home - nothing told the user about the persistent
association.

We therefore agreed to remove the state cache for now. It can be
reintroduced at any time once a consistent usage model exists.

To establish a relationship to a work directory inside a shell session,
kas now evaluates the KAS_WORK_DIR environment variable.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-19 12:54:24 +02:00
Jan Kiszka
fe39f9f370 Massage README
* reorder to have most relevant information first
* adjust docker description
* improve consistency of examples
* add community resources

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-19 12:31:12 +02:00
Claudius Heine
4df59537ba Make __version__ available in the whole module
This shortens the include from "from .__version__ import __version__"
to "from . import __version__" within the module.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-19 12:26:08 +02:00
Jan Kiszka
c9d961f3ee docker: Rename USER_ID variable
Also cosmetic: use a consistent name on the interface as well as
internally.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-19 10:47:02 +02:00
Jan Kiszka
85bec05dd6 docker: Rename user and home dir inside containter
Cosmetic cleanup: call the user and its home dir "builder" as it is an
active entity. This will leave yocto build results under
/builder/build/tmp/deploy/...

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-19 10:46:45 +02:00
Claudius Heine
81a30bc600 Added SHELL and TERM to environment whitelist and start $SHELL if set
Currently the shell provided via the 'shell' command is always '/bin/sh'
and does not have a TERM variable set. This is a rather spartan shell
experience.

With this patch we add the TERM variable to the whitelist, so that
pagers work better and made the shell binary dependent from the SHELL
environment variable. If no such variable is set, we fall back to using
'/bin/sh'.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-19 10:40:01 +02:00
Claudius Heine
bb2e98344a Added 'SSH_AGENT_PID' and 'SSH_AUTH_SOCK' to environment whitelist
The ssh-agent environment variables of an ssh-agent that is running
outside the kas build system are not transfered to the bitbake
environment. This causes fetching of private git repositories to
fail.

This patch adds those variables to the environment whitelist.

Signed-off-by: Claudius Heine <ch@denx.de>
2017-06-19 10:39:34 +02:00
Jan Kiszka
7b83fd40b8 Print help if no arguments are provided
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-19 10:04:22 +02:00
Jan Kiszka
d91e2ce0e6 Add docker image build via Travis CI
Build the docker image in Travis. Push the result as latest version if
run from the master or next branch. Only push tagged commits from the
master branch as releases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-17 23:46:22 +02:00
Jan Kiszka
676ab2e541 Update PyYAML dependencies
We now need this extra package to run kas.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2017-06-15 10:07:13 +02:00
Daniel Wagner
daf0abab5e Initial public release
This is the first public release of kas.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2017-06-14 15:21:19 +02:00