Skip to content

GitHub App setup (webhook trigger)

This document describes how to configure a GitHub App to trigger GitTinkerer runs via the issue_comment webhook.

Overview

  • GitHub delivers an issue_comment webhook to the VPS service endpoint.
  • The service verifies the webhook signature.
  • The service only reacts to comments that begin with /tinker.
  • The service uses issue.pull_request.url from the webhook payload to fetch PR metadata (head/base refs and SHAs).
  • The service starts a run by spawning bin/gittinkerer run --payload-file ....
  • GitTinkerer posts the final response as a new PR conversation comment (by PR number).
  • Run artifacts are written to artifacts/<timestamp>/agent-run/.

1) Create the GitHub App

In GitHub:

  1. Go to Settings → Developer settings → GitHub Apps → New GitHub App.
  2. Set Webhook URL to your VPS service endpoint:
  3. https://<your-host>/api/github/webhook
  4. Set a Webhook secret (random, high-entropy).
  5. Install the app on the org/repo that GitTinkerer will operate on.

2) Subscribe to events

Enable these webhook events:

  • Issue comment

GitTinkerer only uses issue_comment events for PR-backed issues (where issue.pull_request exists).

3) Permissions (minimum)

The exact minimum can vary depending on whether you also use the App token for git pushes.

Recommended baseline permissions:

  • Issues: Read & write
  • Needed to read issue comments (incoming) and create a new comment for the agent response.
  • Pull requests: Read-only
  • Needed to fetch PR metadata (head.ref, head.sha, base.ref) via the PR API URL.
  • Contents: Read & write (only if you plan to use the App token for git pushes over HTTPS)

If you push via SSH deploy keys instead, Contents write may not be required.

4) VPS environment variables

Configure the service with:

  • GH_WEBHOOK_SECRET
  • Used to verify X-Hub-Signature-256.

For GitHub API authentication, you have two options:

Set:

  • GH_APP_ID
  • GH_APP_PRIVATE_KEY
  • PEM-encoded private key (or use service/private-key.pem as described below).
  • GH_APP_INSTALLATION_ID (optional)
  • For webhook-triggered runs, the installation ID is read from the webhook payload (hook.installation.id).
  • For web-mode runs (POST /api/runs with source: "web"), the service looks up the installation via GET /repos/{owner}/{repo}/installation.
  • Only set GH_APP_INSTALLATION_ID if you need to force a specific installation (e.g., custom flows that skip webhooks) or as a fallback when dynamic resolution is not possible.

The service will mint an installation access token at runtime and inject it into the spawned run as GITHUB_TOKEN, satisfying the CLI requirement.

4.1) Providing the private key

Download the GitHub App private key from GitHub App settings → Private keys → Generate and save the PEM file as service/private-key.pem. The service prefers this file and falls back to the GH_APP_PRIVATE_KEY env var if the file is missing or invalid.

Option B: Static token (fallback)

Set:

  • GITHUB_TOKEN

This should be an installation token or other GitHub token with the required scopes.

5) /tinker command behavior

  • Only comments that begin with /tinker trigger a run.
  • /tinker supports multiline instructions.
  • The service strips /tinker and passes the remaining text as comment_body.
  • The full raw comment body is recorded for audit in pr_comment.txt (via comment_raw_body).

Example:

/tinker
Please update the README.
Then fix failing tests.

6) Local testing

Quick-start for running locally with .env and a tunnel:

cd service
npm install
cp ../.env.example .env  # edit .env with your GitHub App values
npm start

Then expose your local port (default 3000) via a tunneling tool (e.g., ngrok) and point the GitHub App webhook URL at the HTTPS tunnel.

Common options:

  • Use a tunneling tool (e.g. ngrok) to expose your local service and point the GitHub App webhook URL at the tunnel.
  • Use GitHub’s webhook “Recent Deliveries” UI to redeliver events.

Install ngrok on WSL Ubuntu:

  1. sudo apt-get update && sudo apt-get install -y wget tar
  2. Download ngrok v3 (x86_64): wget https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz
  3. For ARM64 laptops, use ngrok-v3-stable-linux-arm64.tgz.
  4. tar -xvf ngrok-v3-stable-linux-*.tgz && sudo mv ngrok /usr/local/bin/
  5. Add your auth token: ngrok config add-authtoken <your-ngrok-token>
  6. Verify version ngrok version (should be 3.x). If you have an old ngrok in your PATH, remove or override it.

Using ngrok:

  1. Start the Node service (requires Node 24): cd service && npm install && npm start (set SERVICE_PORT/SERVICE_HOST/ARTIFACTS_DIR as needed; defaults to port 3000).
  2. Run ngrok http <local-port> and copy the generated HTTPS forwarding URL.
  3. In your GitHub App settings, set Webhook URL to <ngrok-https>/api/github/webhook and keep your existing Webhook secret.
  4. Trigger a fresh event (or redeliver in “Recent Deliveries”) and inspect the logs to confirm the request reaches your local service.

When debugging, verify:

  • Signature verification is enabled (GH_WEBHOOK_SECRET matches the App setting).
  • Only /tinker comments trigger runs.
  • PR enrichment works (the service calls issue.pull_request.url).

7) Security notes

  • Always verify webhook signatures.
  • Consider adding additional allowlisting (e.g., repo/org allowlist) before spawning runs.
  • Treat comment bodies as untrusted input.

8) Webhook signature verification

Inline sequence of the verification path (Fastify webhook route → usecase → HMAC check):

sequenceDiagram
  participant GH as GitHub
  participant API as /api/github/webhook
  participant Verify as verifyWebhook
  participant HMAC as verifyGithubSignature

  GH->>API: issue_comment webhook (raw Buffer + X-Hub-Signature-256)
  API->>Verify: secret + rawBody + signature
  Verify->>HMAC: sha256 HMAC + timingSafeEqual
  HMAC-->>Verify: isValid
  Verify-->>API: boolean (warn Sentry on failure)
  API-->>GH: 401 if invalid
  API-->>GH: continue if valid

9) Webhook handling path

The Fastify controller processes GitHub deliveries as follows:

10) PR enrichment and run queue

handleIssueComment orchestrates the /tinker flow (service/src/usecases/github/handleIssueComment.ts):

11) GitHub App token minting

Inline sequence of how API tokens are produced and used:

sequenceDiagram
  participant Service
  participant Auth as getGithubToken
  participant GitHub

  Service->>Auth: need token (optional installationId)
  alt App creds + installationId present
    Auth->>Auth: build RS256 JWT (9 min exp)
    Auth->>GitHub: POST /app/installations/{id}/access_tokens
    GitHub-->>Auth: installation access token
  else fallback
    Auth-->>Service: GITHUB_TOKEN
  end
  Service-->>GitHub: API calls with Bearer token (e.g., pull request fetch)

12) Comment reply publishing

When a run completes for a PR source, the agent publishes a new issue comment on the PR conversation (lib/reply/github.sh):

  • Only runs with PAYLOAD_SOURCE=pr and non-dry-run proceed; otherwise replies are skipped (lib/reply/github.sh).
  • Requires GITHUB_TOKEN, PR number, repo full name, and the rendered response; missing fields fail fast with artifacts (lib/reply/github.sh).
  • Posts to https://api.github.com/repos/{full_name}/issues/{pr_number}/comments using curl; logs the resulting comment id (lib/reply/github.sh).

13) Troubleshooting webhook signatures

  • Ensure GH_WEBHOOK_SECRET matches the GitHub App secret exactly (no whitespace trimming differences).
  • Confirm Fastify receives the raw Buffer body; JSON pre-parsing will break the HMAC check.
  • Verify the X-Hub-Signature-256 header is present; GitHub only signs when a secret is set on the App.
  • Regenerate and redeploy the secret if you rotated it in GitHub but not in the service.
  • If using a tunnel/proxy, ensure it preserves the body bytes; transformations (compression/re-encoding) will cause constant-time compare to fail.