SIFT: Scoring PRs by Real Work

Overview

Many companies judge engineers partly by how many pull requests they merge. With AI, pushing lots of PRs is easier than ever, so the raw count says less every year. I built a pipeline that replaces the count with one score per PR, based on how much meaningful work actually went into it.

1. StartIncoming requestTeams of GitHub users, last 90 days by default
   GitHub APIFetch merged PRsCommits and diffs, comments, reviews, timeline
2. Skip what shouldn’t countPre-filter
   • Simple: drop merge, empty, lockfile, bot and other-author commits (co-authors kept)
   • Revert pairs: a diff that is a 90% inverse of an earlier one nets both to 0, so revert/recommit chains count once
3. Runs for every commitMeaningful Commit OrganizerChecks run in order; a commit stops at the first one that assigns a bucket.
   1. 1. Size & config · config-only files, ≤2 lines, whitespace; docs-only filesLint/ConfigDocs
   2. 2. WIP flag · WIP keywords, placeholder linesflags only
   3. 3. Test check · ≥80% of lines in test files, <20 other linesTesting
   4. 3.5 Conversation · reviewer asked for changes, then approved (by commit position)Addressed
   5. 4. Lint by content · comment/format-only; placeholders or <50% of lines reach the final PRLint/ConfigWIP
   6. AI · Claude Haiku · judges the rest; diff trimmed to 1,500 lines; ignores instructions in diffs; fails closedMeaningfulLintWIPTestingUnclassified
4. Score each PRCalculation3·log2(1 + meaningful) + log2(1 + addressed) + log2(1 + testing) + log2(1 + docs) + log2(1 + min(lines, 2000))Lines are code only, additions weigh 1.5×; docs, config and moved lines are left out
5. OutputResultsBucket and reason per commit; each person vs. team and org average
   ValidationCheck against hand labelsevaluate.py reports agreement per bucket and per check

Every commit records which check decided its bucket, and the AI’s one-sentence reason when it was involved, so any score can be explained or disputed. I tested it on a well-known open-source maintainer’s public work: 34 merged PRs and 59 commits over 90 days.

What I tried that didn’t work

Most of what didn’t work only showed up once I ran the pipeline on real repositories.

• Trusting commit messages. Anyone can type “refactor”. Keywords now only raise a flag; the diff and GitHub’s review data decide the bucket.
• Ordering reviews by timestamp. 13 of the 34 PRs (38%) had been force-pushed, and a rebase rewrites commit dates. I switched to each commit’s position in the PR, which survives a rebase.
• Counting commits linearly. Splitting the same 500 lines into ten commits scored 10× more than one commit. A log scale cuts that gap to 1.6×.
• Counting raw lines. One commit counted 4,662 added lines, nearly all copied documentation. Size now counts code only, and moved lines cancel out. That commit now counts 20 lines.

A tradeoff I made, and why

Rules first, AI last. Sending every commit to a model would be simpler and might catch more edge cases. But a rule’s decision is free, instant and explainable: “this commit only touched config files” takes an engineer five seconds to check. The AI only sees what the rules can’t settle.

Failing closed. When the AI errors out, the commit scores zero and is listed for a rerun instead of getting a best guess. That’s unfair in the short term, but a silent guess is worse: nobody could tell which parts of a score were judged and which were made up.

What I learned

• Real data beats unit tests at finding the real problems. My tests passed, yet one run on real repositories surfaced copied docs counted as code, test data in a testdata/ folder the rules didn’t recognize, and refactors misfiled as lint. Every threshold in the pipeline is still a starting guess until it’s checked against real commits.
• Any metric people are judged by will be gamed. Designing against that shaped almost every decision: damped counts, a size cap, keywords that only flag, and an AI told to ignore instructions hidden in the code it reads.
• AI isn’t ground truth either. The AI misfiled real refactors as lint and a regenerated data file as meaningful until I fixed its prompt. Its answers need checking against human judgement just like the rules do, which is why I built a tool to compare both against my own hand labels.
• Small samples prove less than they look. 19 of 20 agreeing only shows the true rate is at least 76%; 98 of 100 shows at least 93%. I built that confidence bound into the tooling so I couldn’t fool myself.
• Explainability is a feature, not a nice-to-have. A score is only fair if the person being scored can see why. Recording the deciding check for every commit, plus the AI’s one-sentence reason, made the pipeline easier to debug too.