GitHub Connector
Ingest GitHub pull requests, issues, comments, reviews and pushes into CortexDB.
Syncs GitHub pull requests, issues, comments, reviews, inline review comments and pushes, and writes them to CortexDB as experiences.
Two ways to run this connector
Self-hosted — pip install 'cortexdb-connectors[github]' then cortexdb-sync sync github.
Managed — CortexDB runs the connector as a worker. See the Connectors overview.
1. Prepare credentials
Self-host and dashboard use different auth
The managed dashboard flow can authenticate as a GitHub App (App ID / Installation ID / private
key PEM). The self-hosted connector uses a personal access token. cortexdb-sync list shows its
required settings: GITHUB_TOKEN and GITHUB_REPOS.
Create a fine-grained or classic PAT with read access to the repositories' issues, pull requests and contents, and copy it.
2. Choose repositories and history
| Setting | Required | What it does |
|---|---|---|
GITHUB_TOKEN | yes | the personal access token |
GITHUB_REPOS | yes | comma-separated owner/repo list. There is no repository discovery: without it nothing syncs |
GITHUB_BACKFILL_DAYS | no (default 30) | how many days of existing pull requests, issues, comments and reviews the first sync brings in |
GITHUB_EVENTS | no | comma-separated subset of push, pull_request, issues, issue_comment, pull_request_review, pull_request_review_comment (all six by default) |
Size the first sync for busy repositories
The first sync fetches the whole backfill window and writes when the fetch completes, so nothing
appears in CortexDB until then and an interrupted first sync saves nothing. Measured on llvm/llvm-project
with GITHUB_BACKFILL_DAYS=1: about 15 minutes, roughly 850 GitHub API calls, 1,340 events. For a very
busy repository start with a small GITHUB_BACKFILL_DAYS and raise it later. Later syncs are incremental
and re-read the last 6 hours so events GitHub reports late are not skipped.
What gets written
Each record becomes a POST /v1/experience on the resolved scope (default scope template
org:<org>/source:github/project:{repo}), attributed to the GitHub user who acted (observed_actor).
The text of each record:
| GitHub record | Stored text |
|---|---|
| Pull request | number, state, title, author, branches and the full description |
| Push | branch, commit range and every commit's message |
| Issue | number, state, title, author and the full body |
| Issue or PR comment | the full comment, however long |
| Review | review state (approved, commented, …) and the review text |
| Inline review comment | the file and line it is on, the comment, and the diff hunk it refers to |
Every event carries labels such as source:github, repository:<owner>/<repo>,
github_event_type=<type> and, for anything belonging to a pull request or issue, a
thread=<owner>/<repo>:pr:<n> label that groups the PR, its comments and its reviews into one
conversation. The connector never sets context.preceded_by.
Record identity and the key on the wire
Each record has a source identity such as github:issue_comment:<repo id>:<comment id> or
github:push:<repo id>:<ref>:<before>..<head>; it decides which source record an event belongs to. The
idempotency_key actually sent to CortexDB is a derived cxsrc:<56 hex> value that also hashes the
content, labels, observed time and actor, so an unchanged redelivery converges on the same event while any
edit lands as a new one. Every event also carries a source descriptor with the record's fingerprint. See
Source identity and keys.
Run it yourself
pip install 'cortexdb-connectors[github]'
export GITHUB_TOKEN=ghp_... # required (PAT)
export GITHUB_REPOS=acme/api,acme/web # required
export GITHUB_BACKFILL_DAYS=7 # optional, default 30
export CORTEXDB_URL=http://localhost:3141
export CORTEXDB_API_KEY=<your server key> # required, see below
export CORTEXDB_ACTOR=user:local # required: the identity your key acts as
cortexdb-sync sync github
cortexdb-sync watch github --interval 120CORTEXDB_API_KEYis required even if your server runs without auth (CORTEX_INSECURE_NO_AUTH=1): without it every event fails withCortexDB API key missing. Against a no-auth server any placeholder value works. Ignore the error's suggestion to runcortexdb auth signup— that's for CortexDB Cloud.CORTEXDB_ACTORis required and must match the identity the key acts as (user:localfor a static server key). Without it every event fails withCortexDB actor missing.
Cursor state persists in ~/.cortexdb/sync_state.json (override with --state-file). Run the connector
where that file survives restarts; a throwaway container without a volume starts over each time.