graphql

Query GitHub’s GraphQL API: schema-aware chained queries, one-request batching, and raw GraphQL

from fastcore.test import *
from pyskills import xdir

GitHub’s REST API returns each resource’s fixed shape, so a read that fans out – “the head commit of these 100 repos” – is 100 round trips, most of whose bytes you discard. The GraphQL API is one endpoint that accepts a shape: describe the nesting you want and the server returns exactly that, in one round trip. GhGql is fastspec’s GraphQL client bound to GitHub: the schema ships pre-distilled with this package, so discovery (xdir, attribute completion, rich reprs), schema-checked query chaining with plain-kwargs arguments, and parallel chunked batching all work with no setup beyond a GITHUB_TOKEN. Raw GraphQL text works at every level, and only query fields are exposed as attributes – mutations require deliberately writing raw text.

The distilled schema

The distilling machinery lives in fastspec.gql; ghapi ships the result. build_gql_spec in build_lib runs GitHub’s introspection at build time and saves the compact tables as ghapi.gql_spec – so at runtime there is no schema fetch at all, exactly as gh_spec ships the REST spec.

The shipped tables for GitHub’s real schema (1.35MB on disk, ~180KB in the wheel):

from ghapi.gql_spec import gqlspec

len(gqlspec['types']), gqlspec['query'], gqlspec['types']['Ref']['fields']['target']['type']

(1813, 'Query', 'GitObject')

GhGql is a thin binding: fastspec’s GqlClient pointed at GitHub’s endpoint, authenticated from GITHUB_TOKEN (or an explicit token=), with the shipped schema loaded. Fragments, batch, gql.t, raw calls, and GqlError with partial data are all inherited.

---

source

GhGql

def GhGql(
    token:NoneType=None
):

GitHub GraphQL client using the shipped distilled schema

Discovery

Discovery happens on the same objects you build with: xdir (or tab completion) at any point lists what the schema allows next, and displaying an unfinished fragment shows the field’s signature, its argument docs, and the fields available on its result type – so the lookup and the query never live in different places:

gql = GhGql()
xdir(gql, 'repo')

['repository', 'repositoryOwner']

gql.repository

repository(owner: String!, name: String!, followRenames: Boolean = true) -> Repository
Lookup a given repository by the owner and repository name.
args:
  owner: The login field of a user or organization
  name: The name of the repository
  followRenames: Follow repository renames. If disabled, a repository referenced by its old name will return an error.
fields of Repository: allowUpdateBranch archivedAt assignableUsers autoMergeAllowed branchProtectionRules codeOfConduct codeowners collaborators commitComments contactLinks contributingGuidelines createdAt databaseId defaultBranchRef deleteBranchOnMerge dependencyGraphManifests deployKeys deployments description descriptionHTML ...

A complete fragment displays instead as the query it will send – the repr teaches the raw language as you go. Awaiting executes it, unwrapping along the path, so a scalar leaf comes back as the bare value:

f = gql.repository(owner='AnswerDotAI', name='fastws').ref(qualifiedName='refs/heads/main').target.oid
f

{ repository(owner: "AnswerDotAI", name: "fastws") { ref(qualifiedName: "refs/heads/main") { target { oid } } } }

sha = await f
test_eq(len(sha), 40)
sha

'd64c351ed5c9d7da6a596a7cf5d79647ee06862e'

Batching

Because a query is a shape, “run these N fragments” is just one bigger shape: batch (from GqlClient) takes fragments – or one generator of them – and returns results in input order. repo builds the fragment for an 'owner/name' spec. Checking which of a hundred repos moved is one batch call. GitHub resolves a query’s aliases serially (a 103-alias query measured 5.5s), so GhGql sets batch_chunk = 25: large batches go as parallel chunked requests transparently (the same 103 repos: 1.7s):

repos = [('AnswerDotAI/fastws', 'main'), ('AnswerDotAI/ghapi', 'main'), ('fastai/fastcore', 'main'),
    ('AnswerDotAI/aidialog', 'main'), ('AnswerDotAI/llmdojo', 'main')]
heads = await gql.batch(gql.repo(s).ref(qualifiedName=f'refs/heads/{b}').target.oid for s, b in repos)
test_eq(len(heads), len(repos))
for h in heads: test_eq(len(h), 40)
heads

['d64c351ed5c9d7da6a596a7cf5d79647ee06862e',
 'ad07893daac86de6693bc9bb57ae7216c1b347d0',
 '25c4f3228ccac3c5a63da71b5eaa4be3c428f602',
 '53913aba7c6ef152294953645a5dbb9dee142276',
 '1ea3915e8cc7f348e4ef3bfc855075ab5368cfc2']

GitHub reports a missing repository as a path-scoped error alongside null data for that alias. In a bulk scan that must not kill the other 99 answers, so batch returns None for the errored alias and real results for the rest (a global error – bad syntax, auth – still raises):

res = await gql.batch(*[gql.repository(owner='AnswerDotAI', name=n).ref(qualifiedName='refs/heads/main').target.oid
    for n in ('fastws', 'no-such-repo-xyz', 'ghapi')])
test_eq(res[1], None)
test_eq(len(res[0]), 40)
res

['d64c351ed5c9d7da6a596a7cf5d79647ee06862e',
 None,
 'ad07893daac86de6693bc9bb57ae7216c1b347d0']

Raw selections

Chaining covers linear paths. When a selection branches, or needs an inline fragment to downcast a union or interface, call any node with raw GraphQL text instead – here fetching pyproject.toml from several repos at once (object returns a GitObject interface, so reading blob text requires ... on Blob). This is the one-request version of what dep_graph in ghapi.core does with sequential REST fetches:

frags = [gql.repository(owner='AnswerDotAI', name=n).object(expression='HEAD:pyproject.toml')('... on Blob { text }')
    for n in ('fastws', 'aidialog')]
blobs = await gql.batch(*frags)
assert all('[project]' in b.text for b in blobs)
print(blobs[0].text[:120])

[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "fastws-

Some GitHub data is reachable only through GraphQL – ProjectsV2 boards, for example, have no REST endpoints at all:

await gql.organization(login='github').projectsV2(first=3, orderBy=dict(field='TITLE', direction='ASC'))('nodes { title public }')

{ 'nodes': [{'title': 'BUGS', 'public': True}, {'title': 'Campus Experts + GitHub Docs program', 'public': True}, {'title': 'Coding Standards Public Development Board', 'public': True}]}

Connection fields cap first: at 100, so long lists are walked with Relay cursors instead: paged (also from GqlClient) follows them to the end. Here it crosses that cap – every repo name in the org:

names = [o.name async for o in gql.paged(gql.organization(login='AnswerDotAI').repositories, 'name')]
assert len(names) > 100
len(names)

547

The type index

Not everything sits on a path you are building: enum values, input-object shapes, and union membership are looked up rather than navigated to. gql.t indexes every schema type by name.

gql.t.OrderDirection

OrderDirection (ENUM)
Possible directions in which to order a list of items when provided an `orderBy` argument.
  ASC: Specifies an ascending order for a given `orderBy` argument.
  DESC: Specifies a descending order for a given `orderBy` argument.

Raw queries and errors

Everything above compiles down to calling the client with query text, which is always available directly – with GraphQL variables when values shouldn’t be inlined. This is also the only route to mutations, which fragments deliberately never reach:

res = await gql('query($owner: String!) { organization(login: $owner) { name createdAt } }', owner='AnswerDotAI')
test_eq(res.organization.name, 'Answer.AI')
res

{'organization': {'createdAt': '2024-01-13T09:43:28Z', 'name': 'Answer.AI'}}

Failures are loud everywhere except per-alias in batch: awaiting a fragment that still needs a selection is a client-side TypeError before any network, an unknown field fails at chain time naming the type, and a solo query for a missing repo raises the server’s message:

r = gql.repository(owner='AnswerDotAI', name='fastws')
with expect_fail(TypeError, contains='needs a selection'): await r
with expect_fail(AttributeError, contains='has no field'): r.no_such_field
with expect_fail(GqlError, contains='Could not resolve'):
    await gql.repository(owner='AnswerDotAI', name='no-such-repo-xyz').ref(qualifiedName='refs/heads/main').target.oid