Splitting configuration files into multiple files and implementing an include mechanism allows to handle configurations more flexible and allowing following scenarios: - including kas configuration file from external meta layer into own project - splitting many different similar configurations into multiple files with only a couple of common files To include file its necessary to add a 'includes' entry into the kas configuration. To include files from the same repo, do it like this: ----- includes: - ../base/base_include.yml ----- The path is relative to the current configuration file. If they start with a path seperator ("/") they are absolute. To include files from a different repository, do it like this: ----- includes: - file: bsps/my_machine_include.yml repo: collected_machines repos: collected_machines: url: https://url.to.git.repo/ revspec: master ----- You have to make sure that the repository definition is available in your current file, or in any include that is available at this point. Yaml ("*.yml") and Json ("*.json") files are supported. Included in this patch are also some changes to the configuration file structure. Here is an overview: The value of the 'repos' key is a dictionary that maps a repo identifier to a dictionary that contains 5 entries: - url: The url to the repository. If that is missing, no git operations are used - refspec: The git refspec of the repository that is used. If that is missing the latest commit of the default branch is used. - name: The is defines under which name the repository is stored. If that is missing the repository identifier is used - path: The path where the repository is stored. If no url is given and a path is missing, the repository is referencing the repository under which the configuration file is stored. If no url is given and a path is specified, the repository is referencing a directory where layers might be stored. If an url is specified, path can be used to overwrite the default (kas_work_dir + repo.name) checkout directory. - layers: This contains a dictionary of layers, that should be included into the bblayers.conf. The keys are paths relative to the repository and the values can be used to exclude layers if they are one of "excluded", "disabled", "false", "0", "n", or "no". Also boolean values are accepted. Any other value, including "None" means that this layer is included into the bblayers.conf. If "layers" is missing or empty, the repository itself is included into the bblayers. If this is specified, the repository itself is not included into the bblayers.conf. Signed-off-by: Claudius Heine <ch@denx.de>
218 lines
8.5 KiB
Python
218 lines
8.5 KiB
Python
# kas - setup tool for bitbake based projects
|
|
#
|
|
# Copyright (c) Siemens AG, 2017
|
|
#
|
|
# Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
# of this software and associated documentation files (the "Software"), to deal
|
|
# in the Software without restriction, including without limitation the rights
|
|
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
# copies of the Software, and to permit persons to whom the Software is
|
|
# furnished to do so, subject to the following conditions:
|
|
#
|
|
# The above copyright notice and this permission notice shall be
|
|
# included in all copies or substantial portions of the Software.
|
|
#
|
|
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
# SOFTWARE.
|
|
|
|
"""
|
|
This module implements how includes of configuration files are handled in
|
|
kas.
|
|
"""
|
|
|
|
import os
|
|
import sys
|
|
import collections
|
|
import functools
|
|
import logging
|
|
|
|
__license__ = 'MIT'
|
|
__copyright__ = 'Copyright (c) Siemens AG, 2017'
|
|
|
|
|
|
class IncludeException(Exception):
|
|
"""
|
|
Class for exceptions that appear in the include mechanism.
|
|
"""
|
|
pass
|
|
|
|
|
|
class IncludeHandler(object):
|
|
"""
|
|
Abstract class that defines the interface of an include handler.
|
|
"""
|
|
|
|
def __init__(self, top_file):
|
|
self.top_file = top_file
|
|
|
|
def get_config(self, repos=None):
|
|
"""
|
|
Parameters:
|
|
repos -- A dictionary that maps repo name to directory path
|
|
|
|
Returns:
|
|
(config, repos)
|
|
config -- A dictionary containing the configuration
|
|
repos -- A list of missing repo names that are needed \
|
|
to create a complete configuration
|
|
"""
|
|
# pylint: disable=no-self-use,unused-argument
|
|
|
|
logging.error('get_config is not implemented')
|
|
sys.exit(1)
|
|
|
|
|
|
class GlobalIncludes(IncludeHandler):
|
|
"""
|
|
Implements a handler where every configuration file should
|
|
contain a dictionary as the base type with and 'includes'
|
|
key containing a list of includes.
|
|
|
|
The includes can be specified in two ways, as a string
|
|
containg the relative path from the current file or as a
|
|
dictionary. The dictionary should have a 'file' key, containing
|
|
the relative path to the include file and optionally a 'repo'
|
|
key, containing the key of the repository. If the 'repo' key is
|
|
missing the value of the 'file' key is threated the same as if
|
|
just a string was defined, meaning the path is relative to the
|
|
current config file otherwise its relative to the repository path.
|
|
|
|
The includes are read and merged depth first from top to buttom.
|
|
"""
|
|
|
|
def get_config(self, repos=None):
|
|
repos = repos or {}
|
|
|
|
def _internal_file_parser(filename):
|
|
(_, ext) = os.path.splitext(filename)
|
|
if ext == '.json':
|
|
import json
|
|
with open(filename, 'rb') as fds:
|
|
return json.load(fds)
|
|
elif ext == '.yml':
|
|
import yaml
|
|
with open(filename, 'rb') as fds:
|
|
return yaml.safe_load(fds)
|
|
logging.error('Config file extension not recognized: %s', ext)
|
|
sys.exit(1)
|
|
|
|
def _internal_include_handler(filename):
|
|
"""
|
|
Recursively load include files and find missing repos.
|
|
|
|
Includes are done in the following way:
|
|
|
|
topfile.yml:
|
|
-------
|
|
includes:
|
|
- include1.yml
|
|
- file: include2.yml
|
|
- repo: repo1
|
|
file: include-repo1.yml
|
|
- repo: repo2
|
|
file: include-repo2.yml
|
|
- include3.yml
|
|
-------
|
|
|
|
Includes are merged in in this order:
|
|
['include1.yml', 'include2.yml', 'include-repo1.yml',
|
|
'include-repo2.yml', 'include-repo2.yml', 'topfile.yml']
|
|
On conflict the latter includes overwrite previous ones and
|
|
the current file overwrites every include. (evaluation depth first
|
|
and from top to buttom)
|
|
"""
|
|
missing_repos = []
|
|
configs = []
|
|
current_config = _internal_file_parser(filename)
|
|
if not isinstance(current_config, collections.Mapping):
|
|
raise IncludeException('Configuration file does not contain a '
|
|
'dictionary as base type')
|
|
for include in current_config.get('includes', []):
|
|
if isinstance(include, str):
|
|
includefile = ''
|
|
if include.startswith(os.path.pathsep):
|
|
includefile = include
|
|
else:
|
|
includefile = os.path.abspath(
|
|
os.path.join(
|
|
os.path.dirname(filename),
|
|
include))
|
|
(cfg, rep) = _internal_include_handler(includefile)
|
|
configs.extend(cfg)
|
|
missing_repos.extend(rep)
|
|
elif isinstance(include, collections.Mapping):
|
|
includerepo = include.get('repo', None)
|
|
if includerepo is not None:
|
|
includedir = repos.get(includerepo, None)
|
|
else:
|
|
raise IncludeException(
|
|
'"repo" is not specified: {}'
|
|
.format(include))
|
|
if includedir is not None:
|
|
try:
|
|
includefile = include['file']
|
|
except KeyError:
|
|
raise IncludeException(
|
|
'"file" is not specified: {}'
|
|
.format(include))
|
|
(cfg, rep) = _internal_include_handler(
|
|
os.path.abspath(
|
|
os.path.join(
|
|
includedir,
|
|
includefile)))
|
|
configs.extend(cfg)
|
|
missing_repos.extend(rep)
|
|
else:
|
|
missing_repos.append(includerepo)
|
|
configs.append((filename, current_config))
|
|
return (configs, missing_repos)
|
|
|
|
def _internal_dict_merge(dest, upd, recursive_merge=True):
|
|
"""
|
|
Merges upd recursively into a copy of dest as OrderedDict
|
|
|
|
If recursive_merge=False, will use the classic dict.update,
|
|
or fall back on a manual merge (helpful for non-dict types
|
|
like FunctionWrapper)
|
|
"""
|
|
if (not isinstance(dest, collections.Mapping)) \
|
|
or (not isinstance(upd, collections.Mapping)):
|
|
raise IncludeException('Cannot merge using non-dict')
|
|
dest = collections.OrderedDict(dest)
|
|
updkeys = list(upd.keys())
|
|
if not set(list(dest.keys())) & set(updkeys):
|
|
recursive_merge = False
|
|
if recursive_merge:
|
|
for key in updkeys:
|
|
val = upd[key]
|
|
try:
|
|
dest_subkey = dest.get(key, None)
|
|
except AttributeError:
|
|
dest_subkey = None
|
|
if isinstance(dest_subkey, collections.Mapping) \
|
|
and isinstance(val, collections.Mapping):
|
|
ret = _internal_dict_merge(dest_subkey, val)
|
|
dest[key] = ret
|
|
else:
|
|
dest[key] = upd[key]
|
|
return dest
|
|
else:
|
|
try:
|
|
for k in upd:
|
|
dest[k] = upd[k]
|
|
except AttributeError:
|
|
# this mapping is not a dict
|
|
for k in upd:
|
|
dest[k] = upd[k]
|
|
return dest
|
|
|
|
configs, missing_repos = _internal_include_handler(self.top_file)
|
|
config = functools.reduce(_internal_dict_merge,
|
|
map(lambda x: x[1], configs))
|
|
return config, missing_repos
|