Commit Graph

629 Commits

Author SHA1 Message Date
Felix Moessbauer
6fa40363aa kas-main: do not use return type of void function
The 'kas' function does not return any value, hence we must not use the
return value.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-04 09:03:36 +02:00
Felix Moessbauer
bb2148016b do not shadow variables in command tests
As part of the tests of the dump plugins a variable was shadowed. By
that, one branch of the test was not executed.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-04 09:03:22 +02:00
Felix Moessbauer
4015a6b384 kas-container: add support to set log-level
This patch forwards the log-level parameter from kas-container to kas
inside the container. For backward compatibility, the -d parameter is
deprecated but kept and also forwarded as-is to kas. By that, a more recent
kas-container script can still be used to run an older kas container.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 17:25:47 +02:00
Hugo Villeneuve
522a2422c4 kas: add support for --log-level argument
KAS by default will output a lot of information (INFO) messages for
all operations, which makes it difficult to spot warnings thru all
that 'noise'.

Add a command line argument so that the default log level can be
modified.

For backward compatibility, the --debug parameter is still supported
but marked as deprecated in the help message.

Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fixes]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 17:25:36 +02:00
Hugo Villeneuve
246f23cbca kas: define variable to store default log level
To make it easier to display (and modify) the default log level,
especially with the introduction of the new (future commit) argument
--log-level.

Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 17:23:42 +02:00
Felix Moessbauer
88964d63b5 improve documentation of lockfile support
This patch improves the documentation regarding how lockfiles work, where
kas searches for them and how to create/update them. A dedicated section
about the locking mechanism is added to the userguide. The documentation
of the kas internal logic is improved by making the wordings more precise
(e.g. lockspec vs. lockfile).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fixed overlong lines]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 09:05:55 +02:00
Felix Moessbauer
a206f05616 kas-container: mount repos rw when creating lockfile
This patch makes the creation and update of a lockfile more convenient.
When running the dump plugin in inplace mode, a lockfile is created next
to the first file on the kas cmdline. By that, the repo directory also
needs to be mounted rw. Otherwise the kas inside the container cannot
create the file.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-03 08:59:52 +02:00
Felix Moessbauer
471f5e58d2 dump plugin: add newline to file in inplace mode
The auto-generated lockfiles should be terminated with a newline to be
POSIX compliant. Previously, this only worked for output via stdout but
not for inplace operations. This is fixed by appending the newline to
the active output target.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-01 14:19:48 +02:00
Felix Moessbauer
e2bf6d74df add test for lockfile support
This commit adds a test that check the creation, effectiveness and
update of a lockfile. Testing this functionality via the dump plugin is
sufficient, as the plugin directly uses the checkout workflow.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fix over-long lines and removed assert brackets]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:05:04 +02:00
Felix Moessbauer
7c0953c08e dump plugin: add option to create lockfile
This patch adds the --lock option to the dump plugin. When enabled, the
output only contains the resolved refspecs of each repo (as valid kas
format). By that, floating branches can be used in the projects kas files
and these can be pinned to fixed revisions, when required.

When using --lock in combination with --inplace, a lockfile named
<filename>.lock.<ext> is created next to the <filename>.<ext>. In case
multiple files are added to the kas CLI, the lockfile is only created
for the first file (by considering the merged information from all files).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: fold in Python 3.6 support]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:05:04 +02:00
Felix Moessbauer
fe4031ce01 add support for lockfiles on checkout
When checking out repositories, check if a file <filename>.lock.<ext>
exists next to the file specified first on the kas CLI. In case this
file exists and the --update option is not specified, automatically
append this file to the kas CLI before performing any other kas
operations.

When --update is specified, the lockfile is ignored.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:05:04 +02:00
Felix Moessbauer
3e0dd10416 add support to override refspec of repos
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.

Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:04:51 +02:00
Felix Moessbauer
6131038c00 drop support for Python 3.5
As all currently supported distros provide at least python 3.6, we
drop the support for 3.5. At time of this commit, the python versions
of the supported distros are as following:

- Debian: 3.7 (buster)
- Ubuntu: 3.7 (18.04)
- Fedora: 3.11 (Fedora 36)
- RHEL: 2.x (RHEL 7), 3.6 (RHEL 8)
- OpenSUSE: 3.6 (Leap / 15.4)

While updating the lower bound version, we also unify the upper bound in
the setup.py script with the versions tested in the CI.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 11:57:30 +02:00
Felix Moessbauer
93cd91bc7c add meta-schema identifiers in kas json schema
This patch adds the required meta-schema identifiers to allow automatic
validation against a fixed version of json-schema.

Reported-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Freundl, Christoph' via kas-devel
8f61dcfeb6 kas: add GIT_CREDENTIAL_USEHTTPPATH environment variable
This allows the configuration of the git option 'credential.usehttppath' if
the used credential helper requires this.

Signed-off-by: Christoph Freundl <Christoph.Freundl@ifm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Richard Weinberger
10e6df438f kas-container: Report errors from readlink
Currently kas-container fails silently if readlink is unable to resolve
a path.
Add -v to each readlink command to get errors reported.

Before:
$ KAS_BUILD_DIR=/scratch/rwtypo/leia kas-container shell leia.yml
$ echo $?
1

After:
$ KAS_BUILD_DIR=/scratch/rwtypo/leia kas-container shell leia.yml
readlink: /scratch/rwtypo/leia: No such file or directory

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Richard Weinberger
6de4b7bae4 kas-container: Shop colorlog python package
Without this package kas will not use colorful logging.

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:53:15 +02:00
Richard Weinberger
6df962506a container-entrypoint: Switch to POSIX shell
kas-container is carefully written to be POSIX shell compliant.
Let's do the same with container-entrypoint to be consistent.
While we're here, remove the only bashism used in container-entrypoint.

Signed-off-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-20 22:01:31 +02:00
Felix Moessbauer
ea4099c2c8 improve style of dump plugin documentation
This patch puts the code examples of the dump plugin into code-blocks in
the documentation. By that, the documentation is easier to read.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-03-19 19:45:28 +01:00
Stefan Müller-Klieser
1243612a27 kas-container: fix isar setup logic
If we run kas-container with --isar flag and did not set build_system,
we end up calling both enable_isar_mode and enable_oe_mode. This can
trigger:
  Error: keep-id is only supported in rootless mode

Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-03-13 12:49:23 +01:00
Jan Kiszka
b9abbd1e63 Release 3.2.3 2023-03-05 10:36:58 +01:00
Jan Kiszka
258acbead6 Dockerfile: Use original UID/GID when run without kas-container
Since 492b2c56, we create user and group upfront, now using 1000:1000 as
IDs. This can cause unexpected glitches when using the container without
kas-container in environments where older version already created files
with the previously used IDs. In order to stay compatible, switch the
default IDs back to 30000:30000.

This will not affect any user of kas-container.

Reported-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-27 17:02:17 +01:00
Felix Moessbauer
06fad464b5 kas-container: mount host paths below /var/kas
As we no longer create the builder user at runtime, placing data into
/etc/skel at runtime is semantically not correct anymore. Instead, we
bind mount host paths below /var/kas/userdata. By that, we now place
the data into a directory which is fully handled by us.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-27 16:29:50 +01:00
Felix Moessbauer
f971cd363b kas-container: copy host ssh config into home
This patch fixes a regression introduced in 492b2c5. As the builder user
is no longer created in the entrypoint, the data from the home skeleton
is also not copied anymore. This breaks the ssh config (including
known_hosts) when using kas-container with --ssh-dir, as the ssh dir is
mounted into the skeleton, but not copied to the builders home.

With this patch, we now explicitly copy the .ssh folder into the builder
users home, in case it is mounted from the host.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-27 16:29:45 +01:00
Felix Moessbauer
b8576ee99e kas-container: mount KAS_REPO_REF_DIR rw
With the added auto-caching logic of the referenced repos in 1c2c859,
kas-container has to mount this directory in read-write mode.
Otherwise, initial clones will fail.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-27 12:33:15 +01:00
Jan Kiszka
0e61fdddaf Release 3.2.2 2023-02-13 21:18:57 +01:00
Jan Kiszka
d85adb9b93 README: Leave notice on inherit integrity weaknesses of repo fetches
Neither git nor hg currently provide a production-ready replacement for
weak SHA-1 commit IDs. Furthermore, kas mixes commit IDs and symbolic
commit names in refspec. This permits attackers who gained control over
a repository that kas fetches from to present manipulated content
without kas noticing this.

Aditya Sirish A Yelgundhalli recently reported one potential attack
scenario, using branches that shadow commit IDs. While trying to
mitigate this particular case, it became clear that there is no simple
solutions with the given tools and interfaces.

For now, warn prominently that only trusted sources should be used.
There are extensions planned to address the issue at its root, likely by
introducing content checksums.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-13 07:18:29 +01:00
Jan Kiszka
78084c05bb tests: Clean up remaining temporary directory setups
This completes what 6b1df5f425 started.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-12 13:55:23 +01:00
Jan Kiszka
f748c17023 kas-container: Make sure privileged podman will find sbin tools
The user environment may lack /usr/sbin while certain podman
configuration will need, e.g., iptables for the setup. This can cause

Error: plugin type="bridge" failed (add): cni plugin bridge failed: failed to locate iptables: exec: "iptables": executable file not found in $PATH

Resolved that by appending /usr/sbin to the PATH in privileged podman
mode.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-12 13:45:12 +01:00
Jan Kiszka
a39d5a00c5 container: Add copyright headers and improve comments
Dockerfile and container-entrypoint were missing that header.

Furthermore, the leading comment in Dockerfile got out-of-date, and we
should rather add section marker for the kas and kas-isar targets.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-07 10:38:39 +01:00
Jan Kiszka
06aae60b65 container: Disable git safe.directory when running without kas-container
This is for the sake of the gitlab-ci runner which does not properly
aligns the ownership of the repo it checks out with the UID:GID of
our builder user. Reason not yet understood and hard to debug (logging
of the runner is incomplete).

Work around this issue by disabling safe.directory checks in case the
container is called without kas-container as wrapper (means, when it is
called without setting "--user=root"). This preserves git's checks for
the common interactive case, the more critical one.

Reported-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-07 10:38:39 +01:00
Jan Kiszka
492b2c56ab container: Rework uid/gid alignment with caller
Already create the builder user/group during container image build and
only align the IDs in the entrypoint if started with a non-zero USER_ID.
The primary gain is code simplification because this removes some
dynamics from the entrypoint.

As this refactoring avoids that gitlab-ci runners start the container as
root, it was also supposed to resolve the mismatch between the owner of
the checked-out repo and builder user. Unfortunately, this does not work
yet, and the reason is still unclear.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-07 10:38:27 +01:00
Jan Kiszka
a596e2ac33 kas-isar: Rework isar-specific entrypoint setup
Leave the isar-only commands commented-out in the container-entrypoint
and simply remove the comment when building kas-isar. This is simpler,
more readable and also more robust against changes of the entrypoint
file.

While at it, avoid a separate layer for modifying container-entrypoint.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-06 10:30:04 +01:00
Jan Kiszka
6b1df5f425 tests: Avoid useless directory creations
The pattern

mkdir test_commands
rm -rf test_commands

makes no sense. If there were test_commands before, mkdir would have
failed. Simply build the path and copy the source content in.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-05 21:18:35 +01:00
Jan Kiszka
039a320631 Add a SECURITY.md
This file shall describe the vulnerability disclosure process for kas
and the security context in which kas should be seen. Reporting
vulnerabilities via github has already been activated.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-04 14:03:00 +01:00
Jan Kiszka
66893480c7 Release 3.2.1 2023-02-02 08:41:17 +01:00
Jan Kiszka
795ecdbcd6 doc: Fix description of bblayers_conf_header and local_conf_header
For a long time already (c9326cc1ed), the documentation suggests a
different ordering of entries generated by *_conf_header than we
actually create. Finally fix this.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-02-01 08:57:07 +01:00
Hannah Kiekens
b27ab659f3 Dockerfile: Add unzip package to kas-base
Was lost in the refactoring of a6b18abc8a.

Signed-off-by: Hannah Kiekens <hannah.kiekens@mind.be>
[Jan: refer to the causing commit]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-24 07:11:12 +01:00
Jan Kiszka
34cf4c9c8f docs: Fix description of container image generation
Since a6b18abc8a, we only have a single Dockerfile and a --target
instead.

We actually also need to touch the unmodified proxy line, wrapping it
around, so that doc8 remains happy.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-17 18:39:36 +01:00
Jan Kiszka
5b15a4d52a scripts: Add an update reminder for readthedocs
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-17 18:27:08 +01:00
Jan Kiszka
dc44638cd8 Release 3.2 2023-01-17 12:49:44 +01:00
Jan Kiszka
e9fac8dda7 ci: Update docker actions
Silences lots of deprecation warnings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-13 18:11:00 +01:00
Frieder Schrempf
f3521f82fa kas-container: Fix engine detection when docker is an alias for podman
The current detection mechanism assumes that if the docker command
is available the engine behind is also docker. In fact nowadays a
lot of people use /usr/bin/docker as an alias for podman. In that
case the script expects a docker engine and misses to use
podman-specific settings.

To fix this, run "docker -v" and check if the output indicates a
podman engine or a real docker engine. Otherwise error out as the
enginge is not supported.

Signed-off-by: Frieder Schrempf <frieder.schrempf@kontron.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-09 13:09:14 +01:00
Jan Kiszka
f5655dcb0c ci: Update to actions/checkout@v3 and actions/setup-python@v4
Silences some NodeJS 12 action deprecation warnings.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 20:22:22 +01:00
Felix Moessbauer
1c2c859a4a add test for KAS_REPO_REF_DIR logic
This patch adds a test for cloning with KAS_REPO_REF_DIR.
It explicitly tests the case that two repos with a single upstream URI
are added and also tests mercurial.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: ensure Python 3.5 compatibility]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 20:03:25 +01:00
Felix Moessbauer
3289ec2a7e re-create tmpdir after successfull rename
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:12:04 +01:00
Felix Moessbauer
f2560588bc auto create repo refs when setting KAS_REPO_REF_DIR
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.

Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.

The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.

The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:11:06 +01:00
Felix Moessbauer
e7896c33d9 do not short-circuit on initial clone of repos
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:10:51 +01:00
Jan Kiszka
c816b483fa ci: Adjust images to cover all Python versions
Only Ubuntu 20.04 is still supported with 3.5 and 3.6, so pin to it.
Expand the test to 3.10 and 3.11 at this chance. Quoting is needed now
to avoid that 3.10 becomes 3.1.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:59:27 +01:00
Felix Moessbauer
88a2b3d47b add test for resolve-env option in dump plugin
This patch adds a test for the --resolve-env option in the dump plugin.
Both cases resolve and non-resolve are tested, each for yaml and json
output format.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:41:00 +01:00