In production since Jun 2026
AI agent for entry-form triage
An AI agent that reads entry forms, produces a score and decides approval — with a human fallback when the case is ambiguous.
- Period
- Jun 2026 — ongoing
- Role
- Solo
- +1,500
- forms/week
- 15k+
- training cases
- 0–1
- score
#Context
Brasil Capital Roleplay gets a high volume of entry forms — players who want server access and need to be assessed against what the community values. Done by hand, that means a team reading free text, applying subjective judgment and becoming a bottleneck: the candidate waits, the reviewer tires, and similar cases get different answers depending on who read them.
The problem wasn't only speed. It was consistency: the same form had to get the same answer regardless of the hour or who was on shift.
#What the product does
- Reads the candidate's submitted form (free text, open answers).
- Compares the answers against what the operation expects and considers acceptable.
- Produces a score from 0 to 1 representing the candidate's fit.
- Decides on the score: approve, reject, or route to human review when it lands in the gray zone.
#Architecture
#Technical decisions
Why doesn't the AI approve every case on its own?
Because the cost of a false positive is high and asymmetric: approving someone who shouldn't get in creates moderation work later. The agent only decides on its own at the extremes of the score; the ambiguous band falls to human review. I traded a bit of full automation for not outsourcing the edge-case decision.
Why a continuous 0-to-1 score instead of a straight yes/no?
A boolean hides the uncertainty. With the continuous score I choose where to place the approve and reject cutoffs, and move them as the operation gets stricter or looser — with no retraining. The cost is having to define and maintain those thresholds.
How do you keep the decision aligned with the operation, not the model's 'common sense'?
The agent was calibrated on 15k+ real cases from the operation itself, not a generic prompt. The yardstick is the community's, not the model's. The trade-off is that quality depends on curating that history — garbage in, garbage out.
#What went wrong
The first version was too strict at the cutoff: it sent too many cases to human review, which defeated the purpose. The fix wasn't touching the model — it was measuring the real score distribution and recalibrating the thresholds against it. The right decision was almost always in the data, not in the guess.
#Outcome
- Over 1,500 forms triaged per week with no human bottleneck on the clear cases.
- Consistent decisions: the same form gets the same answer, regardless of shift.
- Ambiguous cases preserved for the human eye, where it actually adds value.
Stack
- TypeScript
- Node.js
- LLM
- PostgreSQL