gecko_taskgraph.util package

Submodules

gecko_taskgraph.util.attributes module

gecko_taskgraph.util.attributes.match_run_on_hg_branches(hg_branch, run_on_hg_branches)

Determine whether the given project is included in the run-on-hg-branches parameter. Allows ‘all’.

gecko_taskgraph.util.attributes.match_run_on_projects(params, run_on_projects)

Determine whether the given project is included in the run-on-projects parameter, applying expansions for things like “integration” mentioned in the attribute documentation.

gecko_taskgraph.util.attributes.task_name(task)

gecko_taskgraph.util.backstop module

gecko_taskgraph.util.backstop.is_backstop(params, push_interval=20, time_interval=240, trust_domain='gecko', integration_projects={'autoland'}, backstop_strategy='backstop')

Determines whether the given parameters represent a backstop push.

Parameters:

  • push_interval (int) – Number of pushes

  • time_interval (int) – Minutes between forced schedules. Use 0 to disable.

  • trust_domain (str) – “gecko” for Firefox, “comm” for Thunderbird

  • integration_projects (set) – project that uses backstop optimization

Returns:

True if this is a backstop, otherwise False.

Return type:

bool

gecko_taskgraph.util.bugbug module

exception gecko_taskgraph.util.bugbug.BugbugTimeoutException

Bases: Exception

gecko_taskgraph.util.bugbug.get_session()
gecko_taskgraph.util.bugbug.patch_schedules(base_rev, patch_content, mode='quick')

Query BugBug API with a patch to get test recommendations.

This is used by ./mach test –auto to get test recommendations for local changes.

Parameters:

  • base_rev (str) – The base revision hash.

  • patch_content (str) – The patch content with commit metadata.

  • mode (str) – The mode of test selection, which determines the confidence threshold. One of ‘extensive’, ‘moderate’, or ‘quick’.

Returns:

A dictionary with containing test recommendations filtered by

confidence threshold.

Return type:

dict

Raises:

BugbugTimeoutException – If the API times out.

gecko_taskgraph.util.bugbug.push_schedules(branch, rev)
gecko_taskgraph.util.bugbug.translate_group(group)

gecko_taskgraph.util.cached_tasks module

gecko_taskgraph.util.cached_tasks.add_optimization(config, taskdesc, cache_type, cache_name, digest=None, digest_data=None)

Allow the results of this task to be cached. This adds index routes to the task so it can be looked up for future runs, and optimization hints so that cached artifacts can be found. Exactly one of digest and digest_data must be passed.

Parameters:

  • config (TransformConfig) – The configuration for the kind being transformed.

  • taskdesc (dict) – The description of the current task.

  • cache_type (str) – The type of task result being cached.

  • cache_name (str) – The name of the object being cached.

  • digest (bytes or None) – A unique string indentifying this version of the artifacts being generated. Typically this will be the hash of inputs to the task.

  • digest_data (list of bytes or None) – A list of bytes representing the inputs of this task. They will be concatenated and hashed to create the digest for this task.

gecko_taskgraph.util.chunking module

Utility functions to handle test chunking.

class gecko_taskgraph.util.chunking.BaseManifestLoader(params)

Bases: object

abstractmethod get_manifests(flavor, subsuite, mozinfo)

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters:

  • flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.

  • subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.

  • mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns:

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

class gecko_taskgraph.util.chunking.BugbugLoader(*args, **kwargs)

Bases: DefaultLoader

Load manifests using metadata from the TestResolver, and then filter them based on a query to bugbug.

CONFIDENCE_THRESHOLD = 0.7
get_manifests(suite, mozinfo)

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters:

  • flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.

  • subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.

  • mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns:

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

class gecko_taskgraph.util.chunking.DefaultLoader(params)

Bases: BaseManifestLoader

Load manifests using metadata from the TestResolver.

get_manifests(suite, frozen_mozinfo)

Compute which manifests should run for the given flavor, subsuite and mozinfo.

This function returns skipped manifests separately so that more balanced chunks can be achieved by only considering “active” manifests in the chunking algorithm.

Parameters:

  • flavor (str) – The suite to run. Values are defined by the ‘build_flavor’ key in moztest.resolve.TEST_SUITES.

  • subsuite (str) – The subsuite to run or ‘undefined’ to denote no subsuite.

  • mozinfo (frozenset) – Set of data in the form of (<key>, <value>) used for filtering.

Returns:

A tuple of two manifest lists. The first is the set of active manifests (will run at least one test. The second is a list of skipped manifests (all tests are skipped).

get_tests(suite)
gecko_taskgraph.util.chunking.chunk_manifests(suite, platform, chunks, manifests)

Run the chunking algorithm.

Parameters:

  • platform (str) – Platform used to find runtime info.

  • chunks (int) – Number of chunks to split manifests into.

  • manifests (list) – Manifests to chunk.

Returns:

A list of length chunks where each item contains a list of manifests that run in that chunk.

gecko_taskgraph.util.chunking.get_manifest_loader(name, params)
gecko_taskgraph.util.chunking.get_runtimes(platform, suite_name)
gecko_taskgraph.util.chunking.get_test_tags(config, env)
gecko_taskgraph.util.chunking.guess_mozinfo_from_task(task, repo='', app_version='', test_tags=[])

Attempt to build a mozinfo dict from a task definition.

This won’t be perfect and many values used in the manifests will be missing. But it should cover most of the major ones and be “good enough” for chunking in the taskgraph.

Parameters:

task (dict) – A task definition.

Returns:

A dict that can be used as a mozinfo replacement.

gecko_taskgraph.util.chunking.resolve_manifest_runtimes(all_runtimes, manifests)

Match manifests to their runtimes, aggregating included sub-manifests.

Runtime data keys can be either “manifest.toml” for direct matches or “manifest.toml:included.toml” for included sub-manifests. This function aggregates both into a single runtime per parent manifest.

Parameters:

  • all_runtimes (dict) – Raw runtime data from get_runtimes().

  • manifests (iterable) – Manifest paths to look up.

Returns:

A dict mapping manifest paths to their total runtime in seconds. Manifests with no runtime data are omitted.

gecko_taskgraph.util.constants module

gecko_taskgraph.util.declarative_artifacts module

gecko_taskgraph.util.declarative_artifacts.get_geckoview_artifact_id(config, platform, package, update_channel=None)
gecko_taskgraph.util.declarative_artifacts.get_geckoview_artifact_map(config, job)
gecko_taskgraph.util.declarative_artifacts.get_geckoview_template_vars(config, platform, package, update_channel)
gecko_taskgraph.util.declarative_artifacts.get_geckoview_upstream_artifacts(config, job, package, platform='')

gecko_taskgraph.util.dependencies module

gecko_taskgraph.util.dependencies.chunk_locale_grouping(config, tasks)

Split by a chunk_locale (but also by platform, build-type, product)

This grouping is written for mac signing with notarization, but might also be useful elsewhere.

gecko_taskgraph.util.dependencies.partner_repack_ids_grouping(config, tasks)

Split by partner_repack_ids (but also by platform, build-type, product)

This grouping is written for release-{eme-free,partner}-repack-signing.

gecko_taskgraph.util.dependencies.platform_grouping(config, tasks)
gecko_taskgraph.util.dependencies.product_grouping(config, tasks)
gecko_taskgraph.util.dependencies.single_grouping(config, tasks)
gecko_taskgraph.util.dependencies.single_locale_grouping(config, tasks)

Split by a single locale (but also by platform, build-type, product)

The locale can be None (en-US build/signing/repackage), a single locale, or multiple locales per task, e.g. for l10n chunking. In the case of a task with, say, five locales, the task will show up in all five locale groupings.

This grouping is written for non-partner-repack beetmover, but might also be useful elsewhere.

gecko_taskgraph.util.dependencies.skip_only_or_not(config, task)

Return True if we should skip this task based on only_ or not_ config.

gecko_taskgraph.util.docker module

class gecko_taskgraph.util.docker.ImagePathsMap(config_path, image_dir='/builds/worker/checkouts/gecko/taskcluster/docker')

Bases: Mapping

ImagePathsMap contains the mapping of Docker image names to their context location in the filesystem. The register function allows Thunderbird to define additional images under comm/taskcluster.

register(jobs_config_path, image_dir)

Register additional image_paths. In this case, there is no ‘jobs’ key in the loaded YAML as this file is loaded via jobs-from in kind.yml.

gecko_taskgraph.util.docker.docker_url(path, **kwargs)
gecko_taskgraph.util.docker.image_path(name)
gecko_taskgraph.util.docker.parse_volumes(image)

Parse VOLUME entries from a Dockerfile for an image.

gecko_taskgraph.util.docker.post_to_docker(tar, api_path, **kwargs)

POSTs a tar file to a given docker API path.

The tar argument can be anything that can be passed to requests.post() as data (e.g. iterator or file object). The extra keyword arguments are passed as arguments to the docker API.

gecko_taskgraph.util.hash module

gecko_taskgraph.util.hash.get_file_finder(base_path)
gecko_taskgraph.util.hash.hash_path(path)

Hash a single file.

Returns the SHA-256 hash in hex form.

gecko_taskgraph.util.hash.hash_paths(base_path, patterns)

Give a list of path patterns, return a digest of the contents of all the corresponding files, similarly to git tree objects or mercurial manifests.

Each file is hashed. The list of all hashes and file paths is then itself hashed to produce the result.

gecko_taskgraph.util.hg module

gecko_taskgraph.util.hg.calculate_head_rev(root)
gecko_taskgraph.util.hg.find_hg_revision_push_info(repository, revision)

Given the parameters for this action and a revision, find the pushlog_id of the revision.

gecko_taskgraph.util.hg.get_hg_commit_message(root, rev='.')
gecko_taskgraph.util.hg.get_hg_revision_branch(root, revision)

Given the parameters for a revision, find the hg_branch (aka relbranch) of the revision.

gecko_taskgraph.util.hg.get_hg_revision_info(root, revision, info)
gecko_taskgraph.util.hg.get_json_pushchangedfiles(repository, revision)
gecko_taskgraph.util.hg.get_push_data(repository, project, push_id_start, push_id_end)

gecko_taskgraph.util.partials module

gecko_taskgraph.util.partials.find_localtest(fileUrls)
gecko_taskgraph.util.partials.get_balrog_platform_name(platform)

Convert build platform names into balrog platform names.

Remove known values instead to catch aarch64 and other platforms that may be added.

gecko_taskgraph.util.partials.get_builds(release_history, platform, locale)

Examine cached balrog release history and return the list of builds we need to generate diffs from

gecko_taskgraph.util.partials.get_partials_artifacts_from_params(release_history, platform, locale)
gecko_taskgraph.util.partials.get_partials_info_from_params(release_history, platform, locale)
gecko_taskgraph.util.partials.get_release_builds(release, branch)
gecko_taskgraph.util.partials.get_sorted_releases(product, branch)

Returns a list of release names from Balrog. :param product: product name, AKA appName :param branch: branch name, e.g. mozilla-central :return: a sorted list of release names, most recent first.

gecko_taskgraph.util.partials.populate_release_history(product, branch, maxbuilds=4, maxsearch=10, partial_updates=None)

gecko_taskgraph.util.partners module

gecko_taskgraph.util.partners.GITHUB_API_ENDPOINT = 'https://api.github.com/graphql'

LOGIN_QUERY, MANIFEST_QUERY, and REPACK_CFG_QUERY are all written to the Github v4 API, which users GraphQL. See https://developer.github.com/v4/

gecko_taskgraph.util.partners.apply_partner_priority(config, jobs)
gecko_taskgraph.util.partners.build_macos_attribution_dmg_command(dmg_app_path, attributions)
gecko_taskgraph.util.partners.check_if_partners_enabled(config, tasks)
gecko_taskgraph.util.partners.check_login(token)
gecko_taskgraph.util.partners.fix_partner_config(orig_config)
gecko_taskgraph.util.partners.generate_attribution_code(defaults, partner)
gecko_taskgraph.util.partners.get_attribution_config(manifestRepo, token)
gecko_taskgraph.util.partners.get_ftp_platform(platform)
gecko_taskgraph.util.partners.get_partner_config_by_kind(config, kind)

Retrieve partner data starting from the manifest url, which points to a repository containing a default.xml that is intended to be drive the Google tool ‘repo’. It descends into each partner repo to lookup and parse the repack.cfg file(s).

Supports caching data by kind to avoid repeated requests, relying on the related kinds for partner repacking, signing, repackage, repackage signing all having the same kind prefix.

gecko_taskgraph.util.partners.get_partner_config_by_url(manifest_url, kind, token, partner_subset=None)

Retrieve partner data starting from the manifest url, which points to a repository containing a default.xml that is intended to be drive the Google tool ‘repo’. It descends into each partner repo to lookup and parse the repack.cfg file(s).

If partner_subset is a list of sub_config names only return data for those.

Supports caching data by kind to avoid repeated requests, relying on the related kinds for partner repacking, signing, repackage, repackage signing all having the same kind prefix.

gecko_taskgraph.util.partners.get_partner_url_config(parameters, graph_config)
gecko_taskgraph.util.partners.get_partners(manifestRepo, token)

Given the url to a manifest repository, retrieve the default.xml and parse it into a list of partner repos.

gecko_taskgraph.util.partners.get_partners_to_be_published(config)
gecko_taskgraph.util.partners.get_repack_configs(repackRepo, token)

For a partner repository, retrieve all the repack.cfg files and parse them into a dict

gecko_taskgraph.util.partners.get_repack_ids_by_platform(config, build_platform)
gecko_taskgraph.util.partners.get_repo_params(repo)