# Pagination
### Paged operations
Some GitHub API operations return their results one page at a time. For
instance, there are many thousands of
[gists](https://docs.github.com/github/writing-on-github/creating-gists),
but if we call `list_public` we only see the first 30:
``` python
api = GhApi()
```
``` python
gists = await api.gists.list_public()
len(gists)
```
30
That’s because this operation takes two optional parameters, `per_page`,
and `page`:
``` python
api.gists.list_public
```
List public gists
Docs: https://docs.github.com/rest/gists/gists#list-public-gists
Parameters: - since (str, optional): Only show results that were last
updated after the given time. This is a timestamp in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) format:
`YYYY-MM-DDTHH:MM:SSZ`. - per_page (int, default: 30): The number of
results per page (max 100). For more information, see “[Using pagination
in the REST
API](https://docs.github.com/rest/using-the-rest-api/using-pagination-in-the-rest-api).” -
page (int, default: 1): The page number of the results to fetch. For
more information, see “[Using pagination in the REST
API](https://docs.github.com/rest/using-the-rest-api/using-pagination-in-the-rest-api).”
This is a common pattern for `list_*` operations in the GitHub API. One
way to get more results is to increase `per_page`:
``` python
len(await api.gists.list_public(per_page=100))
```
100
However, `per_page` has a maximum of `100`, so if you want more, you’ll
have to pass `page=` to get pages beyond the first. An easy way to
iterate through all pages is to use
[`paged`](https://ghapi.fast.ai/page.html#paged), which returns an async
generator:
------------------------------------------------------------------------
source
### paged
``` python
def paged(
oper, *args, per_page:int=30, max_pages:int=9999, **kwargs
):
```
*Convert operation `oper(*args,**kwargs)` into an async iterator,
requesting pages serially until one comes back empty*
We’ll demonstrate this using the `repos.list_for_org` method:
``` python
api.repos.list_for_org
```
List organization repositories
Docs:
https://docs.github.com/rest/repos/repos#list-organization-repositories
Parameters: - org (str, required): The organization name. The name is
not case sensitive. - direction (str, optional): The order to sort by.
Default: `asc` when using `full_name`, otherwise `desc`. - type (str,
default: ‘all’): Specifies the types of repositories you want
returned. - sort (str, default: ‘created’): The property to sort the
results by. - per_page (int, default: 30): The number of results per
page (max 100). For more information, see “[Using pagination in the REST
API](https://docs.github.com/rest/using-the-rest-api/using-pagination-in-the-rest-api).” -
page (int, default: 1): The page number of the results to fetch. For
more information, see “[Using pagination in the REST
API](https://docs.github.com/rest/using-the-rest-api/using-pagination-in-the-rest-api).”
``` python
repos = await api.repos.list_for_org(org='fastai')
len(repos),repos[0].name
```
(30, 'docs')
To convert this operation into a Python iterator, pass the operation
itself, along with any arguments (either keyword or positional) to
[`paged`](https://ghapi.fast.ai/page.html#paged). Note how the function
and arguments are passed separately:
``` python
repos = paged(api.repos.list_for_org, org='fastai')
```
The object returned from
[`paged`](https://ghapi.fast.ai/page.html#paged) is an async generator,
so iterate through it with `async for`:
``` python
async for page in repos: print(len(page), page[0].name)
```
30 docs
30 nbdev_template
30 hugo
30 course22
4 lm-hackers
### Link header (RFC 5988)
GitHub tells us how many pages are available using the [link
header](https://tools.ietf.org/html/rfc5988). Unfortunately the pypi
[LinkHeader](https://pypi.org/project/LinkHeader/) library appears to no
longer be maintained, so we’ve put a refactored version of it here.
------------------------------------------------------------------------
source
### parse_link_hdr
``` python
def parse_link_hdr(
header
):
```
*Parse an RFC 5988 link header, returning a `dict` from rels to a
`tuple` of URL and attrs `dict`*
Here’s an example of a link header with just one link:
``` python
parse_link_hdr('; rel="foo bar"; type=text/html')
```
{'foo bar': ('http://example.com', {'type': 'text/html'})}
``` python
links = parse_link_hdr('; rel="foo bar"; type=text/html')
link = links['foo bar']
test_eq(link[0], 'http://example.com')
test_eq(link[1]['type'], 'text/html')
```
Let’s test it on the headers we received on our last call to GitHub. You
can access the last call’s headers in \`recv_hdrs’:
``` python
api.recv_hdrs['Link']
```
'; rel="prev", ; rel="last", ; rel="first"'
Here’s what happens when we parse that:
``` python
parse_link_hdr(api.recv_hdrs['Link'])
```
{'prev': ('https://api.github.com/organizations/20547620/repos?per_page=30&page=5',
{}),
'last': ('https://api.github.com/organizations/20547620/repos?per_page=30&page=5',
{}),
'first': ('https://api.github.com/organizations/20547620/repos?per_page=30&page=1',
{})}
### Getting pages in parallel
Rather than requesting each page one at a time, we can save some time by
getting all the pages we need in parallel.
------------------------------------------------------------------------
source
### GhApi.last_page
``` python
def last_page():
```
*Parse RFC 5988 link header from most recent operation, and extract the
last page*
To help us know the number of pages needed, we can use `last_page`,
which uses the link header we just looked at to grab the last page from
GitHub.
We will need multiple pages to get all the repos in the `github`
organization, even if we get 100 at a time:
``` python
await api.repos.list_for_org('github', per_page=100)
api.last_page()
```
6
------------------------------------------------------------------------
source
### pages
``` python
async def pages(
oper, n_pages, *args, per_page:int=100, **kwargs
):
```
*Get `n_pages` pages from `oper(*args,**kwargs)`, in parallel*
[`pages`](https://ghapi.fast.ai/page.html#pages) by default passes
`per_page=100` to the operation.
Let’s look at some examples. To get all the pages for the repos in the
`github` organization in parallel, we can use this:
``` python
gh_repos = (await pages(api.repos.list_for_org, api.last_page(), 'github')).concat()
len(gh_repos)
```
554
If you already know ahead of time the number of pages required, there’s
no need to call `last_page`. For instance, the GitHub docs specify that
we can get at most 3000 gists:
``` python
gists = (await pages(api.gists.list_public, 30)).concat()
len(gists)
```
3000
GitHub ignores the `per_page` parameter for some API calls, such as
listing public events, which it limits to 8 pages of 30 items per page.
To retrieve all pages in these cases, you need to explicitly pass the
lower per page limit:
``` python
await api.activity.list_public_events()
api.last_page()
```
10
``` python
evts = (await pages(api.activity.list_public_events, api.last_page(), per_page=30)).concat()
len(evts)
```
292
### Sync clients
On a `GhApi(sync=True)` client (see `core`’s “Sync usage” section)
endpoint calls return pages directly, so pagination gets sync twins.
[`sync_paged`](https://ghapi.fast.ai/page.html#sync_paged) is
[`paged`](https://ghapi.fast.ai/page.html#paged) as a plain generator:
------------------------------------------------------------------------
source
### sync_paged
``` python
def sync_paged(
oper, *args, per_page:int=30, max_pages:int=9999, **kwargs
):
```
*[`paged`](https://ghapi.fast.ai/page.html#paged) for a `sync=True`
client: request pages serially until one comes back empty*
``` python
sapi = GhApi(sync=True)
pgs = L(sync_paged(sapi.repos.list_for_org, org='fastai'))
test_eq(len(pgs[0]), 30)
assert len(pgs) > 1
```
[`sync_pages`](https://ghapi.fast.ai/page.html#sync_pages) mirrors
[`pages`](https://ghapi.fast.ai/page.html#pages), fetching a known
number of pages in parallel; with no event loop to gather on, it uses a
thread pool instead, which is safe because every request creates its own
HTTP client. `last_page` works unchanged on a sync client, since it only
parses the stored link header.
------------------------------------------------------------------------
source
### sync_pages
``` python
def sync_pages(
oper, n_pages, *args, per_page:int=100, n_workers:int=16, **kwargs
):
```
*Get `n_pages` pages from `oper(*args,**kwargs)` on a `sync=True`
client, in parallel via threads*
``` python
gists_s = sync_pages(sapi.gists.list_public, 3).concat()
test_eq(len(gists_s), 300)
```
### GH Notifications
------------------------------------------------------------------------
source
### gh_notifs
``` python
async def gh_notifs(
days:int=5, reasons:tuple=('mention', 'review_requested', 'author', 'assign'), include_read:bool=False,
per_page:int=100
):
```
*Notifications from the past `days` (all available if None) as
[`NotifRows`](https://ghapi.fast.ai/page.html#notifrows), marked
`[closed]`/`[merged]` when the subject is no longer open; `reasons=None`
includes every reason*
------------------------------------------------------------------------
source
### NotifRows
``` python
def NotifRows(
items:NoneType=None, *rest, use_list:bool=False, match:NoneType=None
):
```
*Notification rows; displays ‘no notifications’ when empty*
------------------------------------------------------------------------
source
### NotifRow
``` python
def NotifRow(
*args, **kwargs
):
```
*One notification thread as an actionable line*
``` python
await gh_notifs(2, include_read=True)
```
'**2 notifications** (past 2 days; mention, review_requested, author, assign)\n\n- [PullRequest #5](https://github.com/AnswerDotAI/fastcflare/pull/5) AnswerDotAI/fastcflare — Use new fastspec `AttrDict` response (assign)\n- [PullRequest #14](https://github.com/AnswerDotAI/fastspec/pull/14) AnswerDotAI/fastspec — fix: walk anyOf/oneOf in _schema_props_required (review_requested)'
Each row leads with its thread id, which is what `mark_done` takes, so
the triage loop is read the rows, act on what needs acting, then clear
by id. GitHub’s API can mark a thread done but never lists done threads
back, so `mark_done` is irreversible from the API side (the web UI’s
Done tab still shows them).
------------------------------------------------------------------------
source
### GhApi.mark_done
``` python
async def mark_done(
*ids
):
```
*Mark notification threads as done, removing them from the inbox (the
API cannot list done threads back)*