Commit Graph

508 Commits

Author SHA1 Message Date
Quirin Gylstorff
1dea24145e docs/userguide.rst: Add recommendation for repo-id naming
This is added to avoid confusion in the creation of downstream layers.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
[Jan: fixed over-long line]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-12-01 19:06:38 +01:00
Adriaan Schmidt
41dbef918a Dockerfile: base containers on bullseye
Currently the containers are based on buster, but use buster-backports,
and (in case if the Isar container) even plain bullseye with pinning as
additional deb soure.
This changes that to use bullseye only.

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-29 18:32:02 +01:00
Jan Kiszka
75d1a5cce4 Release 2.6.3 2021-11-29 18:19:12 +01:00
Jan Kiszka
412241b32c Provide a pyproject.toml
Permits installing via wheel, even if this dependency is not yet
installed.

Reported-by: Marius Kriegerowski <marius.kriegerowski@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-25 09:14:21 +01:00
Tobias Schmidl
6044d85fb8 docs: Extended "layers" section in the user guide.
Adding `.` was not intuitively for me, therefore an explicit comment on
how to add both the repo and a subdir might be useful.

Signed-off-by: Tobias Schmidl <tobias@schmidl.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-24 13:44:17 +01:00
Jan Kiszka
e3af001f53 libkas: Do not overwrite existing .ssh/config
kas wrongly assumes so far that setting $HOME will also make ssh use the
kas provided $HOME/.ssh/ folder. But ssh will pick up the homedir from
/etc/passwd instead. This could cause kas to overwrite the users
~/.ssh/config when using SSH_PRIVATE_KEY*. We can try to cure ssh config
isolation, but that may cause surprises for users so far silently
relying on it.

For now, as a stable fix, avoid to cause damage to ~/.ssh/config in
cases where this is likely not desired, namely when there is already
config file. Warn if that file does not contain the generated content
from a previous run.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-23 18:26:13 +01:00
Jan Kiszka
122f7749a3 readthedocs: Add configuration file
Use the more powerful configuration file .readthedocs.yaml to ensure
that all requirements are installed. This now includes also a distro
package, python-newt, something that cannot be installed via a
requirements.txt.

Fixes the doc build for 2.6.x which become broken with the addition of
the menu plugin.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-22 19:20:14 +01:00
Quirin Gylstorff
2c0486d846 kas-container: Add check for KAS_REPO_REF_DIR
Check if KAS_REPO_REF_DIR exists to avoid an error during
the execution of readlink.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-06 10:57:15 +01:00
Quirin Gylstorff
9b9bb00c56 kas-container: create KAS_WORK_DIR if it not exists
If KAS_WORK_DIR is set in the executing shell
and does not exists kas-container fails silently
during readlink -f on KAS_BUILD_DIR if the default
"${KAS_WORK_DIR}/build" is used.

Create KAS_WORK_DIR to ensure the subsequent
code execution.

This can be tested by setting KAS_WORK_DIR, e.g.

```
KAS_WORK_DIR="$(pwd)/kas_work" kas-container ...
```

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-11-06 10:57:15 +01:00
Jan Kiszka
3ad03d1590 Release 2.6.2 2021-11-02 13:01:21 +01:00
Jan Kiszka
a753a14316 Dockerfile: Drop world-write permission from /kas folder
The github CI environment causes the copied /kas folder to become
world-writable. This is at least undesired, so fix up the permissions.
This is done the classic way, i.e. via a separate chmod, rather than
using the new and not yet widely available "COPY --chmod".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-29 12:51:10 +02:00
Jan Kiszka
f38f287178 Dockerfile: Relocate and rework oe-git-proxy installation
Moving this file to /usr/local/bin broke user configurations that had
/usr/bin/ hard-coded, and it also broke inside the kas sandbox which
does not have /usr/local/bin in its PATH. So move things back to the
original location.

Do that by setting a link to the /kas/contrib folder, rather than
copying the file once more.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-29 12:25:25 +02:00
Jan Kiszka
0b394f0bb7 Release 2.6.1 2021-10-22 20:22:20 +02:00
Jan Kiszka
257e5bdd5e setup: Drop include_package_data
This prevents adding package_data to the source distribution, which we
have to.

Reported by: Ralf Beier <rbeier57@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-22 19:58:59 +02:00
Jan Kiszka
a81ee2c2b1 Release 2.6 2021-10-22 17:40:17 +02:00
Daniel Wagenknecht
358e5924c4 for_all_repos: adapt test
The `this` repo is skipped in the test for the `for-all-repos` command.
Adapt to checking for KAS_REPO_URL being empty instead of comparing the
KAS_REPO_NAME. This way the test will fail if the implementation
diverges from the documentation:
> KAS_REPO_URL: The URL from which this repository was cloned, or an
> empty string if no remote URL was given in the config file.

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-15 15:42:27 +02:00
Jan Kiszka
a40127f8ed for_all_repos: Empty KAS_REPO_URL if repo is not under version control
repo.url will point the repo.path in that case, but we want the user to
see an empty KAS_REPO_URL, as also documented.

Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-15 15:42:27 +02:00
Jan Kiszka
bfee47ebbc libcmds: Write more bblayers.conf boilerplate settings
Irrespective of build system (OE / Yocto / Isar), they all have

    BBPATH = "${TOPDIR}"
    BBFILES ?= ""

in their bblayers_conf_header fragments. This could be dropped if kas
wrote those standard assignments already. So let's do that, weakly so
that those could still be overwritten via bblayers_conf_header.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-11 08:20:32 +02:00
Jan Kiszka
95df3720e1 tests: Add basic tests for menu plugin
This checks basic features of kas menu, some option selection, yaml
including, rebuilding and target selection.

The UI is operated by mocking smack's GridFormHelp.runOnce and some
result evaluation functions. Injecting keys from the keyboard was not
easily possibly (if at all), thus this approach. The menu rendering is
not shown due to pytest.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:02:22 +02:00
Jan Kiszka
da62e0bfbd kas-container: Add support for menu plugin
For this script, this is almost straightforward - except that we need to
extract the static KAS_BUILD_SYSTEM config setting from the selected
Kconfig file so that the correct container image and mode is chosen.

Two new dependencies need to be added to the container image. While
python3-newt can come from Debian, kconfiglib only exists as Python
package. To make sure we are not pulling any other packages via pip,
install kconfiglib upfront. It has no own dependencies, thus can use
--no-deps as well.

Finally, the container-entrypoint needs to be updated to make it aware
of the new plugin.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
5fb9067894 Fall back to .config.yaml if no configuration file given
Make the configuration file on the command line optional and fall back
to trying to open the generated .config.yaml instead. This allows the
sequence

kas menu -> save & exit
kas build
kas shell
kas ...

and also makes rebuilding the self-configured image simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
1271320de3 kas-container: Enter with /repo as current dir
Set KAS_WORK_DIR instead. This will be needed for running build/shell
with the implicit kas configuration file .config.yaml.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
eeab1bc9bc config: Pass context to Config constructor
Will be used later on to retrieve kas_work_dir.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
4f23045ce8 Add kconfiglib-based menu plugin
This uses the kconfiglib python module to implement simple interactive
configuration for complex layers with many kas options and even variable
input. See the menu.py documentation header for details on the Kconfig
format and its translation to kas.

Rather than using the limited menuconfig implementation of kconfiglib,
this comes with an own, python-newt based version. The permits smooth
integration in the workflow, e.g. to offer an option that directly
triggers a build after completing a configuration. And it comes with a
nicer layout.

Unfortunately, python-newt can't be fulfilled via pip. But it is
available as proper package via common distros. So warn if the package
is missing.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
4c975a04f9 plugins: Invoke setup_parser_common_args explicitly from setup_parser
This allows to define plugins which do not need have any of common args.

Move setup_parser_common_args into libkas for this as it now becomes a
library function.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:05 +02:00
Jan Kiszka
6dd27e7c62 ci: Validate shellcheck archive
This avoids pulling in anything unexpected and then running it as part
of the test and later on image build process.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 15:00:04 +02:00
Jan Kiszka
abb645f8e9 Bump format revision due to changes to include paths
This justifies a version bump, even though the old behavior is still
supported.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-09 14:40:07 +02:00
Jan Kiszka
ee2600a51b tests: test_patch: Restore current directory on exit
Avoids that succeeding tests may stumble over this. So far not an issue.

Drop a forgotten debug print at this chance.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-08 17:58:20 +02:00
Vivien Didelot
3484f1c208 Dockerfile: install g++-multilib
Some recipes, especially meta-qt5 packages, requires g++-multilib in
the host as stated in a few projects issues [1] [2].

[1] https://github.com/siemens/meta-iot2000/issues/18#issuecomment-293417336
[2] https://github.com/meta-qt5/meta-qt5/issues/348#issuecomment-677154661

Install the package in the kas container.

Signed-off-by: Vivien Didelot <vdidelot@pbsc.com>
[Jan: fix arm64 build]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-06 15:03:51 +02:00
ciarancourtney
460699685e yml: Move json schema from python dict to standalone json file
- Easier to reference as schema for IDEs
- Can be added to schemastore.org

Signed-off-by: ciarancourtney <ciaran.courtney@activeenergy.ie>
[Jan: rebased, merged in schema installation]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-05 10:28:24 +02:00
Jan Kiszka
6630780935 includehandler: Simplify code
Since 2077900b4e, the existence of a repo key is enforced by the
config schema. So we can drop the redundant check in the include
handler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-01 11:16:28 +02:00
Jan Kiszka
50e2d2d0fa Align include documentation to new logic
Commit 2077900b4e was wrong in claiming that the documentation already
demanded repo-relative paths for local includes. It only did that for
dictionary-based include entries. Aligning both to repo-relative paths
remain a value, but we definitely have to update the documentation now.

While at it, also update the code comments accordingly. That also still
talked about the repo key being option, something that was never the
case in practice.

Reported-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-10-01 11:16:28 +02:00
Jan Kiszka
08631e428f build: Clarify need for separating extra bitbake args
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-09-25 09:27:58 +02:00
Ralf Anton Beier
2c3997ad56 Enable kas to checkout repositories using git credentials
Using ssh to authenticate against repositories to check out is enabled.
When switching to https and authentication via the git credential helper
it is necessary to first setup the home directory before checking out the
repositories.

Signed-off-by: Ralf Anton Beier <ralf_beier@me.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-09-15 07:29:00 +02:00
Paul Barker
c7ee61616a Drop unnecessary shebang lines
When packaging kas for OpenSUSE, rpmlint threw the following warnings:

    RPMLINT report:
    ===============
    kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/__main__.py 644 /usr/bin/env python3
    kas.noarch: W: non-executable-script /usr/lib/python3.6/site-packages/kas/kas.py 644 /usr/bin/env python3
    This text file contains a shebang or is located in a path dedicated for
    executables, but lacks the executable bits and cannot thus be executed.  If
    the file is meant to be an executable script, add the executable bits,
    otherwise remove the shebang or move the file elsewhere.

    2 packages and 0 specfiles checked; 0 errors, 2 warnings.

The Python files inside the kas module don't need shebang lines as they
are not executed directly so we can just remove them.

Signed-off-by: Paul Barker <paul@pbarker.dev>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-30 21:56:49 +02:00
Jan Kiszka
6a97879ad0 Dockerfile: Carry oe-git-proxy locally
This imports revision aa9b9dc9a9 from
https://git.yoctoproject.org/git/poky to avoid fetching it - and having
to add the missing content validation to prevent supply-chain attacks.

Reported-by: Raphael Lisicki <raphael.lisicki@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-30 21:56:04 +02:00
Jasper Orschulko
9732ec16f2 for_all_repos: Exit on command failure
KAS should exit with the appropriate exit code, if the command parsed to the for_all_repos
plugin fails to apply on one of the repositories.

Example use-case: This allows for CI pipelines or wrapper scripts to take appropriate action (e.g. fail),
if an issue occured within the KAS for_all_repos command.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-26 18:56:38 +02:00
Jasper Orschulko
fca78e5601 for_all_repos: Fix test case
An upcoming change will introduce returning non-zero status on the KAS
command if the for_all_repos command fails for one of the repos defined
in the kas configuration. This caused an issue with the non-git "this"
repo defined in the test.yml.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: adjusted commit log to reflect different commit ordering]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-26 18:55:52 +02:00
Jan Kiszka
6eda3ba48a kas-container: Simplify translation of file argument list
Since dabda7617f, all elements of KAS_FILES are already absolute.
Therefore, a single replacement rule is sufficient to translate them to
paths for the container.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-24 19:19:36 +02:00
Jan Kiszka
c7bd8393aa tests: Add include test case for multiple files in sub-dirs
This stresses the previously broken case of the incorrectly detected
repository root while including files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:26 +02:00
Jan Kiszka
2077900b4e includehandler: Fix repo path detection for local includes
So far, repository paths for local includes were derived from the path
name of the config file containing the include, rather than using the
actual repository root as specified in the documentation. No one
complained so far, some layers simply adjusted their includes to this
inconsistency which was only discovered during refactorings.

Fix this issue by passing also the repository path along with the config
filename down the recursive _internal_include_handler calls. The
top-level repo path now needs to be retrieved before the creation of
IncludeHandler and passed to it then. This has the side effect of
enabling deviating top-level repo paths, a feature that will be used by
the upcoming changes for a .config.yaml file in KAS_WORK_DIR.

As there are existing users of the old behavior out there, fall back to
it if a local include cannot be found under the correct path and warn if
this succeeds. This allows smooth migration of layers to the right
pattern as they update their kas version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-19 10:10:14 +02:00
Jan Kiszka
8a1c27bc92 libkas: Gracefully handle empty repo set
We will fail differently then, but with a nicer error message (missing
init-build-env script) than

ValueError: Set of coroutines/Futures is empty.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 23:17:41 +02:00
Jan Kiszka
4c81f96742 context: Add support for relative KAS_WORK/BUILD/REPO_REF_DIR paths
No one asked for this so far but the result of providing a relative
path, e.g. to KAS_WORK_DIR, is so confusing that it is better to simply
resolve all those vars to absolute paths.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 12:43:09 +02:00
Ross Burton
ff99bf2226 Dockerfile: install lz4
OpenEmbedded Core is about to depend on lz4c, pzstd, and zstd being on
the host.  The Dockerfile was already installing zstd which provides
zstd/pzstd, so also install lz4 for lz4c.

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-18 12:41:21 +02:00
Steffen Hieber
41d9f68ec3 Dockerfile: install git-lfs from buster-backports.
Debian Buster comes with version 2.7.1 of git-lfs which does not support
git+ssh yet, therefore we need a newer version of git-lfs for cloning a
repository which uses LFS.

Signed-off-by: Steffen Hieber <steffen.hieber@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-06 12:56:10 +02:00
Jan Kiszka
bb134014f2 Dockerfile: Replace unneeded backports pinning with explicit release selection
Backport repos have a priority of 100 by default, thus can just be added
with extra pinning. Use explicit release selection on the installation
of qemu-user-static instead, that's simpler.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-08-06 11:54:43 +02:00
Jan Kiszka
ed19ad0059 kas-container: Fix indention
No functional changes.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Drew Reed
6b65de3e2f Enable gerrit/gitlab/github refspecs
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree.  This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes.  When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.

Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Jan Kiszka
635b1023a1 Declare proxy_config obsolete
Since 7b18e5ec3b, which was released with 0.19.0, proxy_config has no
effect anymore. No one seems to have missed it, likely because setting
the proxies via the config file was a mistake anyway. The environment
variables are the proper way.

Declare these config entries officially obsolete and warn if they still
hang around in some legacy config. We will drop them from the schema in
one of the next releases.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-12 08:03:40 +02:00
Jasper Orschulko
15de0142d4 Add environment variable SSH_PRIVATE_KEY_FILE
Additionally to the variable SSH_PRIVATE_KEY, which can be used to
reference a private key saved as a string within a variable (the previous
documentation falsely stated, that it references a path and has been
adjusted), the new variable SSH_PRIVATE_KEY_FILE can now be used to reference
the path to a file containing the private key.

Signed-off-by: Jasper Orschulko <jasper@fancydomain.eu>
[Jan: Fix logic for detecting set env vars]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-10 11:25:40 +02:00