This patch adds the top-level `overrides` entry, which is used to
override (or pin) the refspec of repositories. The main difference to a
direct override is that this logic only applies to repos that are
already defined. By that, a superset of all repos can be added to this
entry (similar to a global lockfile), but only the currently active ones
are affected. A new top-level keyword is required because everything
below the "repos" keyword is potentially defined by "default" values.
For the locking mechanism, a clear separation between overrides (only
override if existing) and definitions is required to be able to define a
global lockfile with all possible repos, while just defining some repos.
Proposed-by: Ross Burton <ross@burtonini.com>
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
[Jan: also bump __file_version__]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a workaround for python versions < 3.8.
There, the dir created by TemporaryDirectory must still exist when
leaving the context manager.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch reworks the logic when setting KAS_REPO_REF_DIR.
When this variable is set, a two-staged clone is used:
First, a bare-clone (or similar) is created in the ref-dir, according to
the naming scheme. This clone is executed in a way that is both
reentrant as well as race-free across multiple instances of KAS working
on the same dir. Internally we clone into a tmpdir below the refdir and
rename on success to guarantee the atomicity of the operation on POSIX
compliant filesystems.
Second, the clone in the KAS_WORK_DIR is executed against the local
copy. After that, the origin url is redirected to the upstream url.
By that, the KAS_REPO_REF_DIR directory can be cached across builds
which significantly speedsup clone times against large repos.
In case the requested refspec is already in the cache (very likely in CI
builds), no direct access to the upstream repo is required. This logic
is crucial for CI systems in China, where e.g. access to github is
blocked from time to time.
The clone-from-local logic is currently only implemented for the git plugin
as HG misses the caching logic. Repo implementations that do not support this
logic can simply opt-out for the first stage by returning 'true'.
The existing user-facing logic of KAS_REPO_REF_DIR is not affected.
However, internally we no longer clone via --reference as this still
requires access to the remote repo, even if the requested commit is
already in the local copy.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch is a preparation for the cloning via reference logic.
Instead of handling the initial-clone specially, we just run the full
fetch_async logic. This is required as clones via a local mirror have a
different remote url which needs to be rewritten.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This patch adds a programmatic attribute 'revision' to the repo class.
This attribute contains the exact revision at the time of the checkout
of a repository. By that, we can avoid the ambiguity of refspecs
containing tags or branch names. Internally, the revision is not yet
used but just made available for future downstream users (e.g. plugins).
Note, that the revision has to be re-queried on each access, as the
Config class re-instantiates the repos for each consumer.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Patches contain tailing white space in empty lines around a change, those
are generally expected by git-apply and cause no issue.
However in cases where a patch file is added, removed or modified
in a patch, that patch would necessarily include `- ` or `+ ` lines,
since they operate on patch files.
Applying those patches with git-apply will cause git to print out
warnings about tailing white spaces per default, but still applies the
patch. Those warnings will be picked up by kas, and since they are
coming from stderr, print them out as `ERRORS`.
To solve this add `--whitespace=nowarn` as a parameter to the git-apply
call to silence those warnings.
In case of kas, it is to be expected that patches are added, removed or
changed in meta layers, so those supposedly errors will cause confusion.
Signed-off-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
git-fetch prints its progress on stderr which makes harmless progress
output be rendered like ERRORs. Messages we likely do not care about and
certainly do not want to see as errors.
This gets us consistent with most of the other git operations using -q.
Closes: #78
Signed-off-by: Henning Schild <henning.schild@siemens.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Only local repositories may be configured without refspec.
Raise an error if a repository url is given but refspec is not.
Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By default git only fetches references under the refs/heads/ tree,
this patch adds support to kas to enable you to specify references
outside of the ref/heads tree. This is useful as it allows you to
use uncommitted gerrit patchsets, Gitlab merge requests or github
pull requests that live under refs/changes/, refs/merge-requests
and refs/pull as the reference for a repo allowing the use of
in development changes. When a refsepc is defined that starts
with refs/ an additional git fetch operation is preformed on the
repo to explicitly fetch the reference given so it can be checked
out for use.
Signed-off-by: Drew Reed <drew.reed@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
when importing nested layer in combination with the top level layer like
meta-security
meta-security/meta-tpm
one can use
layers:
.:
meta-tpm:
in the spec file to import the parent layer.
This leads to the effect that
.
meta-tpm = "HEAD:<hash>"
is shown in the build console and in /etc/build
when using image-buildinfo.bbclass.
To fix that just strip the trailing dot from each layer name.
Add test to avoid future regressions
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
[Jan: use only a single rstrip call, fix style]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Refactored repo path calculation eliminates duplicated code and aligns
the matching logic (with respect to url and path) with the
documentation.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The `repo` documentation says:
"If the url is missing and the path defined, this entry references the
directory the path points to. [...] In case of a relative path name
`kas_work_dir` is prepended."
But if no `url` is given and a relative path is given in `path`, then
the layer paths being added to `bblayer.conf` are missing the
`kas_work_dir` prefix.
This patch fixes it.
Signed-off-by: Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The Repo.factory method determines the name of each repo from either a
'name' property from the key used for this repo within the repos
dictionary. The Repo object should be initialised with this name instead
of dropping it and falling back to the basename of the path.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
No reason to run this with "fail=False" because we won't ignore errors
anyway. The current setting rather leads to the stderr output being
swallowed.
Reported-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
In order for oe-core to correctly display the layers and revisions
used in a build, the paths used in conf/bblayers.conf must have no
trailing path separator.
Signed-off-by: Adrian Stratulat <adrian.stratulat91@gmail.com>
[Jan: fix overlong line, update test case]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The fetch() method for git will call git fetch --all ; this causes
issues when user has multiple remotes. We are able to simply call
'fetch' because by default the origin remote will be used.
From git help fetch:
> When no remote is specified, by default the origin remote will be
used, unless there's an upstream branch configured for the current
branch.
Signed-off-by: Nathan Genetzky <nathan@genetzky.us>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Also update config file version to 9 due to this format change.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
[Jan: style fix]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
While we support
layers:
.: excluded
this is effectively the same as leaving out the layers property: The
repository itself will always be added as layer to bblayers.conf. This
prevents using kas to checkout layer-free repositories as well, e.g.
bitbake in case oe-core is used.
Add support for that be converting a non-existing layers property into
layers:
.:
internally and removing the special case from Repo.layers which assumed
that layers=None meant the above.
Closes: #16
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
By default we do not update a repository if the desired refspec is
already checked out and so we do not pull any new commits that may have
been added to this refspec upstream. If the new `--update` argument is
passed on the command line then we instead pull in any new upstream
commits.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
When checking out a repository the default behaviour is to abort if
local changes are present. If the new `--force-checkout` argument is
passed on the command line then any local changes will instead be
discarded so that the desired refspec can be checked out.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If refspec is a branch name, ensure that the local branch name is always
set to the refspec. In the opposite case when refspec is a commit hash,
ensure that for git repositories we always end up in the 'detached HEAD'
state.
It's also cheaper and simpler to always run `git checkout` or `hg
checkout` than it is to test if a checkout is actually needed. If the
desired refspec is already checked out then these commands are no-ops.
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Handle gracefully the case that a repo has no branch checked out after
cloning.
Reported-by: Claudius Heine <ch@denx.de>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
We were always printing the ID of the last patch of a repo because the
application loop only iterated over a set of patch paths, not the patch
entries. Fix that be assembling a tuple of path and ID in my_patches.
Reported-by: Mustafa Yuecel <mustafa.yuecel@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If refspec is pointing to an upstream branch name, the checkout for a
second run will not reset a repo back to the upstream commit. That is
because we applied the repo patches to a local branch that carries the
name of the upstream one.
Fix that by switching to local branch prior to applying any patches.
Reported-by: Belisko Marek <marek.belisko@gmail.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
patch can not handle file mode headers and can therefore not be used to
create executable files. fixes 8a2a21d799
In addition commit every single patch and treat git patches as
potentially just diffs and not mails.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Switch to applying all patches with "patch" and later do a "<vcs> add;
<vcs> commit". Also iterate over quilt series files ourselfs and apply
the patches one by one.
This means we can apply patches on mercurial as well as on git.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
The interface cmd prototypes got out of sync so mercurial stopped
working. Align with git again to fix that.
Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Currently the the remote URL is only set during the initial clone.
This might lead to confusion with the user as changing the URL in the kas.yml
does not get reflected and thus errors about references not found show up.
This change makes sure the URL is always set to the value specified in the
kas.yml file.
For mercurial this currently prints a warning as there is no command to change
the default path.
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
[Jan: fix up style, adjust comment on hg's set_remote_url_cmd]
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
There is no good reason to only warn about an incorrect patch entry or
a missing patch file. This just causes CI systems to succeed with a
build - or fail at unexpected places.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
Analogously to bitbake's PREMIRRORS, this allows to define alternative
sources for repo URLs specified in kas files.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
pylint has been a great tool to get the initial code base into shape
but since then we mostely false postives. All those warnings needed to
be annonated.
This allong wouldn't be reason enough to remove it. The main problem
is that the pylint version on gitlab is updated in the back and
suddenly we get new warnings which are bogus.
Signed-off-by: Daniel Wagner <daniel.wagner@siemens.com>
This allows the caller to differentiate between "no repo found" and
"result points to repo". Default behavior remain unchanged (return
provided path itself if no repo is found).
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Rather than handing down a complete config, only pass what is really
needed: a repository fallback path. This can already be determined by
the caller.
Inside Repo.factory(), we can retrieve the global context now and can
stop relying on a reference stored in the config.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If env is not provided, use the global context environment. Then drop
all every passing of that parameter.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
This fixes a regression of 5b85fba0: diff-index is porcelain and seems
to lack some explicit refresh prior to being called. Therefore, it can
incorrectly detect files as being modified although a "git diff" will
not - and will actually also update the status so that the next
diff-index would be fine again.
Fix those false positives while still detecting also cached changes by
using git status -s instead.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
If a kas-file inside a repo includes another kas-file form a repo
which is not checked-out, and this 2nd repo can only be checked-out
with an SSH-key, kas fails. This is, because the constructor of
the Config class used to fetch missing repos already before the
SSH-Agent was setup.
This patch refactors the way in which kas checks-out repositories. This
also required the Config class to be split into Context and Config,
where the new Context is the build-Context, i.e. containing the
environment of commands executed by kas and the new Config is the Config
of kas itself, i.e. containing the repo dictionary.
This way it is possible to initialize the context needed for SSH setup
independently of the kas configuration.
The commands ReposFetch and ReposCheckout are refactored into a
SetupRepos command. This command parses the include files and
successively checks-out needed repos and updates the config as long
as repos are missing. The logic is taken directly from the constructor
of the former Config class (with minor adaptations).
Further refactoring will be needed to clean up the conceptual
programming patterns (i.e. to keep the Macro-Style consistent),
and to re-add the macro pattern regarding the above ReposFetch.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
is_dirty_cmd() uses `git diff --shortstat` to see if the repo can be
safely checked out. This only checks for the modified files and does
not take the cached(staged) files into account.
There can be cases where the cached changes could be overwritten by a
checkout. In this case, the git checkout command aborts with an error
message telling the user to commit or stash their changes.
Use `git diff-index HEAD --shortstat` to determine if there are any
changes in the tracked files, whether cached or not.
https://git-scm.com/docs/git-diff-index
Signed-off-by: Abdur Rehman <abdur_rehman@mentor.com>
With this commit its now possible to patch 3rd party repos before bitbake is started.
Example:
This is our repo:
.
├── kas.yml
├── first-patch.patch
└── quilt-patches
├── second-patch.patch
├── third-patch.patch
└── series
Content of kas.yml:
header:
version: 8
repos:
my:
third-party:
url: "git://example.com/third-party.git"
refspec: "35adf4...34"
patches:
01-first:
repo: my
path: "first-patch.patch"
02-second:
repo: my
path: "quilt-patches"
Currently only 'git' repositories can be patched.
Signed-off-by: Claudius Heine <ch@denx.de>
If a repo path is relative, append the kas work dir to it.
Background: The path ends up in bblayers.conf
Without this change the following example will fail:
--------------------------------------------------------
header:
version: 4
machine: qemux86
repos:
poky:
url: "https://git.yoctoproject.org/git/poky"
refspec: "rocko"
path: "thirdparty/poky"
layers:
meta:
meta-poky:
--------------------------------------------------------
Signed-off-by: Georg Lutz <georg@georglutz.de>
