# Apache GitHub issue/PR activity

Standalone tool to scrape **who filed** issues and pull requests across
`github.com/apache` repositories (metadata only—no titles or bodies), classify
authors as **committer / unknown / bot**, and produce monthly trend charts.

## Requirements

- Python 3 with: `pyyaml`, `easydict`, `requests`, `python-ldap`, `ezt`
- GitHub personal access token (public repo read)
- ASF LDAP bind credentials (for the committer → GitHub map)
- Optional for charts: `./fetch-echarts.sh` (curl or wget)

## Setup

```bash
cd apache_gh_activity
cp config.example.yaml config.yaml
# edit config.yaml: github.token, ldap.bind_dn, ldap.bind_password
./fetch-echarts.sh    # once; writes echarts.min.js (not versioned)
```

`config.yaml` and `data/` are svn-ignored (secrets and runtime DB/reports).

## Commands

```bash
./main.py scrape [--refresh-repos]   # GraphQL → SQLite events
./main.py import-committers          # LDAP → committers table
./main.py report                     # CSV + HTML trends
```

Typical order: scrape (long, resume-safe) → import-committers → report.

| Command | Purpose |
|---------|---------|
| **scrape** | List org repos (cached), page all issue/PR **opens**, store author id/login + `created_at` |
| **import-committers** | Load ASF committer group + GitHub attrs from LDAP |
| **report** | Classify every open; write monthly CSV and ECharts HTML |

### Scrape notes

- **Resume:** unfinished repos keep `phase` (`issues` / `prs`) + GraphQL cursor; re-run continues.
- **Repo list cache:** reused until `--refresh-repos` or empty DB.
- **Smoke tests:** `only_repos: [name]` and/or `max_to_try: N` in config.
- **Rate limits:** primary + secondary detection, backoff, retry cap; fails leave state unfinished (not marked done).

### Classification

| Group | Rule |
|-------|------|
| **bot** | Login in `bots.logins`, or ends with `bots.suffix` (default `[bot]`) |
| **committer** | Author matches LDAP map (numeric GitHub id preferred, else login) |
| **unknown** | Everyone else |

**Important:** “committer” means *GitHub account linked in ASF LDAP*, not every ASF id. Unlinked committers who file issues/PRs appear as **unknown**. The committer series is a lower bound on true committer participation.

### Report outputs (`data/reports/`)

| File | Content |
|------|---------|
| `activity_by_month.csv` | `period, kind, group_name, count` |
| `trends.html` | Monthly line charts (issues + PRs) + metadata blurb |
| `echarts.min.js` | Symlink to package `echarts.min.js` |

Open `trends.html` in a browser after `report`.

## Data store (`data/events.sqlite`)

| Table | Role |
|-------|------|
| `repos` | Org repository list |
| `events` | One row per issue/PR open |
| `committers` | ASF id ↔ GitHub id/login (from LDAP) |
| `scrape_state` | Per-repo scrape phase / cursor / last error |
| `meta` | e.g. last repo-list time |

## Layout

```
apache_gh_activity/
  main.py              # all modes
  config.example.yaml  # template (committed)
  config.yaml          # local secrets (ignored)
  trends.ezt           # HTML report template
  fetch-echarts.sh     # download echarts.min.js
  echarts.min.js       # local only (ignored)
  data/                # ignored
    events.sqlite
    reports/
```

## Dependencies (Python)

```text
pyyaml  easydict  requests  python-ldap  ezt
```