Commit Graph

69 Commits

Author SHA1 Message Date
Jan Kiszka
420bf399b3 repos: git: Strip of heads/ prefix when checking out branches
If a remote branch is referenced as refs/heads/<branch>, also drop the
heads/ from the local branch name to have a consistent naming.

This wasn't noticed so far because the primary use case of refs/ is
addressing symbolic refs in upstream, and the related test case
(test_refspec_absolute) was papering over another hidden kas issues
while using refs/heads/.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Jan Kiszka
a317a0e957 repos: Rename boolean branch variable/argument to is_branch
Makes it clearer that this is a boolean, not the branch name.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-30 13:36:26 +02:00
Felix Moessbauer
5e4da5020b repos: restore error handling of patch apply
In ac4373088 the error handling was made more strict. By that, failures
during patch apply instantly bailed out instead of returning and
providing the more useful error message. This patch restores the error
handling by marking the commands as not-fail and encapsulating the error
into a more specific exception with a useful error message.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:49:53 +02:00
Felix Moessbauer
d3963ff494 refactor: remove dead code paths on task exception
As the tasks with fail=True property return via an exception for a
long time, the handling of the return code is not required anymore.
This simplifies the code by removing dead code paths.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-19 06:49:49 +02:00
Felix Moessbauer
222f07de69 refactor: port all sys.exit over to kas exceptions
This patch replaces all direct invocations of sys.exit outside of the
main invocation to KasUserError based exceptions. By that, only one
method for returning is used and return codes can be handled
consistently. In addition, this makes it possible to handle specific
errors differently.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:56:32 +02:00
Felix Moessbauer
a5750901c6 use custom exceptions to improve error handling
This patch adds the KasUserError exception class to distinguish between
internal kas exceptions and user or configuration errors. Exceptions
previously raised on user errors are ported over by deriving
KasUserError. In case of user errors, only the exception message is
shown, but no stacktrace. This makes it easier for users to locate the
issue as the reason is now stated in the last line of the output.

Kas internal exceptions are not subject to this change to help the
developers to find the root cause more easily.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-05-05 21:54:40 +02:00
Felix Moessbauer
3e0dd10416 add support to override refspec of repos
This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.

Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-04-23 14:04:51 +02:00
Felix Moessbauer
3289ec2a7e re-create tmpdir after successfull rename
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:12:04 +01:00
Felix Moessbauer
f2560588bc auto create repo refs when setting KAS_REPO_REF_DIR
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.

Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.

The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.

The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:11:06 +01:00
Felix Moessbauer
e7896c33d9 do not short-circuit on initial clone of repos
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2023-01-05 18:10:51 +01:00
Felix Moessbauer
23dcf955a2 add revision attribute to repo class
This patch adds a programmatic attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).

Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-12-13 14:39:25 +01:00
Claudius Heine
42e83cb7a3 repos: add --whitespace=nowarn to git apply command
Patches contain tailing white space in empty lines around a change, those
are generally expected by git-apply and cause no issue.

However in cases where a patch file is added, removed or modified
in a patch, that patch would necessarily include `- ` or `+ ` lines,
since they operate on patch files.

Applying those patches with git-apply will cause git to print out
warnings about tailing white spaces per default, but still applies the
patch. Those warnings will be picked up by kas, and since they are
coming from stderr, print them out as `ERRORS`.

To solve this add `--whitespace=nowarn` as a parameter to the git-apply
call to silence those warnings.

In case of kas, it is to be expected that patches are added, removed or
changed in meta layers, so those supposedly errors will cause confusion.

Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-05-16 09:12:48 +02:00
Henning Schild
16a697736c repos: git fetch always with quiet flag
git-fetch prints its progress on stderr which makes harmless progress
output be rendered like ERRORs. Messages we likely do not care about and
certainly do not want to see as errors.
This gets us consistent with most of the other git operations using -q.

Closes: #78
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-26 09:21:13 +01:00
Tobias Schaffner
1c53b817b5 fix: Raise an error if no refspec is provided
Only local repositories may be configured without refspec.
Raise an error if a repository url is given but refspec is not.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2022-02-21 20:05:31 +01:00
Drew Reed
6b65de3e2f Enable gerrit/gitlab/github refspecs
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree.  This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes.  When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.

Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-07-19 07:57:32 +02:00
Konrad Weihmann
b250c341fb repos: strip dot from layer name
when importing nested layer in combination with the top level layer like

meta-security
meta-security/meta-tpm

one can use

layers:
    .:
    meta-tpm:

in the spec file to import the parent layer.
This leads to the effect that

.
meta-tpm             = "HEAD:<hash>"

is shown in the build console and in /etc/build
when using image-buildinfo.bbclass.
To fix that just strip the trailing dot from each layer name.

Add test to avoid future regressions

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
[Jan: use only a single rstrip call, fix style]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2021-05-05 15:43:58 +02:00
Silvano Cirujano Cuesta
0892b37ad6 repos: refactor repo path calculation
Refactored repo path calculation eliminates duplicated code and aligns
the matching logic (with respect to url and path) with the
documentation.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:12 +01:00
Silvano Cirujano Cuesta
1f1a21992d repos: fix path if no url, but path given
The `repo` documentation says:
"If the url is missing and the path defined, this entry references the
directory the path points to. [...] In case of a relative path name
`kas_work_dir` is prepended."

But if no `url` is given and a relative path is given in `path`, then
the layer paths being added to `bblayer.conf` are missing the
`kas_work_dir` prefix.

This patch fixes it.

Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-12-16 18:17:04 +01:00
Paul Barker
d42df24afe repos: Keep repo name as read from config file
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-11-17 17:52:00 +01:00
Jan Kiszka
e5701a03a6 repos: Make set_remote_url_cmd fail loudly
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.

Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-10-13 14:06:31 +02:00
Adrian Stratulat
a5c8345ee6 repos: remove trailing slash from layer path
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.

Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-09-23 09:34:28 +02:00
Nathan Genetzky
0e7c32cc79 repos: GitRepo fetch_cmd shouldn't default to "git fetch --all"
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.

From git help fetch:

> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.

Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-27 08:21:28 +02:00
Paul Barker
b107a60118 config: Allow a default repo to be specified for patches
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:47 +02:00
Paul Barker
2260189fda config: Allow a default refspec to be specified
Also update config file version to 9 due to this format change.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-07 11:48:08 +02:00
Jan Kiszka
f678b24611 repos: Allow for layer-free repositories
While we support

  layers:
    .: excluded

this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.

Add support for that be converting a non-existing layers property into

  layers:
    .:

internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.

Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-07-02 07:46:49 +02:00
Jan Kiszka
acdb774d67 repos: Finally fix typo
Something happened to 53668123c7 while I applied it. Fix it for real.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-29 22:12:05 +02:00
Jan Kiszka
839c936abb Fix flake8 style reports
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 08:09:14 +02:00
Paul Barker
24f2f66cf5 kas: Support --update argument
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:40 +02:00
Paul Barker
1569895001 kas: Support --force-checkout argument
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-17 07:58:32 +02:00
Paul Barker
b998dfa722 repos: Make tracking branch resolution more robust
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.

It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-15 19:18:43 +02:00
Paul Barker
53668123c7 repos: Fix typo in repo type error message
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-06-02 08:35:07 +02:00
Jan Kiszka
e41d5af76b repos: Fix cloning of repos without default branch
Handle gracefully the case that a repo has no branch checked out after
cloning.

Reported-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-29 14:33:09 +02:00
Jan Kiszka
d58a291513 repos: Silence pycodestyle error
New pycodestyle version, new warning: "l" might be confused with "1".

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-05-19 07:53:02 +02:00
Jan Kiszka
2142fd855e Fix reporting of repo patch IDs
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.

Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-04-10 18:10:17 +02:00
Jan Kiszka
6a67d69411 Address deprecated warnings with Python 3.8
All these chances work fine with 3.5, our minimally required version.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2020-01-09 11:53:29 +01:00
Jan Kiszka
26a1a0b03b Create from local branch before applying patches with git
If refspec is pointing to an upstream branch name, the checkout for a
second run will not reset a repo back to the upstream commit. That is
because we applied the repo patches to a local branch that carries the
name of the upstream one.

Fix that by switching to local branch prior to applying any patches.

Reported-by: Belisko Marek <marek.belisko@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-23 13:08:51 +02:00
Henning Schild
81c5beedaa repos: apply patches with version control system and not patch
patch can not handle file mode headers and can therefore not be used to
create executable files. fixes 8a2a21d799

In addition commit every single patch and treat git patches as
potentially just diffs and not mails.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-09-21 12:49:47 +02:00
Henning Schild
8a2a21d799 repos: change the way we apply patches
Switch to applying all patches with "patch" and later do a "<vcs> add;
<vcs> commit". Also iterate over quilt series files ourselfs and apply
the patches one by one.

This means we can apply patches on mercurial as well as on git.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-20 12:53:54 +02:00
Henning Schild
df8d7b929d repos: Fix up the basic mercurial support
The interface cmd prototypes got out of sync so mercurial stopped
working. Align with git again to fix that.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-14 12:10:08 +02:00
Pascal Bach
66bd078e7f repos: make sure the remote url is matching the kas file
Currently the the remote URL is only set during the initial clone.
This might lead to confusion with the user as changing the URL in the kas.yml
does not get reflected and thus errors about references not found show up.

This change makes sure the URL is always set to the value specified in the
kas.yml file.

For mercurial this currently prints a warning as there is no command to change
the default path.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
[Jan: fix up style, adjust comment on hg's set_remote_url_cmd]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2019-08-13 20:38:03 +02:00
Jan Kiszka
2f24bdeac2 repos: Fail on unsuccessful patching
There is no good reason to only warn about an incorrect patch entry or
a missing patch file. This just causes CI systems to succeed with a
build - or fail at unexpected places.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:22:55 +01:00
Jan Kiszka
2f7650bb05 Add KAS_PREMIRRORS support
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2019-02-14 09:20:35 +01:00
Daniel Wagner
25bc0c9c06 scripts: Remove pylint
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.

This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.

Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
2018-12-07 09:55:26 +01:00
Andreas Reichel
10cd13cf9c Correct grammar, spelling and copyright year in files
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
2018-09-05 12:46:35 +02:00
Jan Kiszka
4a1fba912e repos: Add fallback parameter to get_root_path()
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:25 +02:00
Jan Kiszka
b349f86457 repos: Remove ctx parameter from repo functions and their callers
We can use the global context instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:54:18 +02:00
Jan Kiszka
4acde53b94 Repo: Rework factory() interface
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.

Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:53:08 +02:00
Jan Kiszka
0c66b24a12 repos: Remove environ parameter from get_root_path()
This is just the default, namely the environment from the global
context.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:52:07 +02:00
Jan Kiszka
86dc584284 run_cmd: Default to environment from global context
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:51:34 +02:00
Jan Kiszka
76e7339fdb repos: Fix false positives when detecting dirty repos
This fixes a regression of 5b85fba0: diff-index is porcelain and seems
to lack some explicit refresh prior to being called. Therefore, it can
incorrectly detect files as being modified although a "git diff" will
not - and will actually also update the status so that the next
diff-index would be fine again.

Fix those false positives while still detecting also cached changes by
using git status -s instead.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
2018-08-27 07:45:15 +02:00