Constraint 01 · Volume
Specialists answer the same questions about visa timelines, required documents, form completion and eligibility rules across hundreds of cases.
← All work
Case study 04 · Settly · 2024–Present
Relocation AI Agent Platform
A multi-agent AI system that turns relocation expert knowledge into scalable platform capability — with structured evaluation, retrieval-augmented generation, and human-in-the-loop deployment.
- Role
- Senior Product Manager
- Timeline
- 2024–Present
- Scope
- Architecture · Evaluation · Deployment
- Team
- 1 PM · eng · domain experts
30%
Reduction in immigration query handling effort
3
Specialized domain agents deployed
RAG
Trusted-source knowledge retrieval system
HITL
Human-in-the-loop for safe deployment
01
Strategic context
Settly's long-term vision is to scale relocation services through a platform-first operational model — workflows, knowledge and communication embedded in the product rather than distributed across internal teams. A structural challenge was clear: the most time-consuming activities, from answering immigration questions to coordinating travel bookings, still relied on manual knowledge work performed by domain experts.
Scaling that linearly meant hiring more specialists. That is not a platform strategy.
“The goal was not to build a chatbot. It was to design a scalable AI capability layer — an orchestrated multi-agent architecture embedded directly into Settly's operational platform.”
Three limitations made it urgent: operational scaling caps on expertise, knowledge fragmented across Slack channels, past cases, government websites and vendor systems, and a clear opportunity to centralise operational reasoning and make it reusable across thousands of cases.
02
Problem identification
Immigration workflows are strong candidates for AI — and carry specific constraints that make naive approaches dangerous.
Constraint 02 · Fragmentation
Correct answers combine official government sources, law-firm guidance, internal operational knowledge and historical cases. No single source has the full answer — this needs retrieval-augmented reasoning, not rules.
Constraint 03 · Asymmetric risk
Incorrect immigration advice has legal and human consequences. Evaluation must prioritise factual correctness and policy compliance over stylistic quality.
Some workflows also require authenticated portal access and multi-step navigation — demanding explicit escalation logic for when the system defers to a human. And prompt changes move model behaviour materially, so evaluation datasets have to evolve with the edge cases.
03
What I did
I translated operational workflows into AI-compatible system architecture across four tracks.
- 01Workflow discovery with domain experts — collected real immigration questions from internal Slack, identified the most frequent request types, and mapped which sources experts consult to construct an answer.
- 02Multi-agent architecture design — instead of one general assistant, an orchestrated architecture where each workflow is handled by a specialised agent, with a central orchestrator routing requests and supplying context.
- 03AI evaluation framework — curated datasets of real questions, expert-written golden answers, and automated scoring, enabling quantitative benchmarking and regression testing.
- 04Gradual prototyping with human-in-the-loop deployment — agents draft, hosts review and edit, approved responses go out, and real-world performance data accrues safely.
The defining decision
The evaluation framework was designed from day one, not retrofitted. In this domain you do not discover failure modes after shipping — you build the infrastructure to detect them before users do.
04
The solution
An orchestrated multi-agent architecture: specialised agents, tiered retrieval, structured workflow logic and evaluation infrastructure.
ORCHESTRATOR → AGENTS
01
Agent orchestrator
A central routing layer that reads intent and dispatches to the right specialist agent with the case context attached.
PRIMARYSECONDARYHISTORICAL
GOV · LAW FIRM · CASES
02
Immigration agent
Answers guidance questions using trusted sources and expert knowledge, with policy compliance scored on every response.
REQUEST → VENDOR → BOOKED
03
Travel agent
Handles vendor communication and travel booking coordination — the highest-volume, lowest-risk workflow, so it went first.
FORM VERSION AS TRUTH
04
Document agent
Answers questions about forms and documentation requirements, where the source of truth is a specific version of a specific form.
PRIMARYSECONDARYHISTORICAL
05
Tiered knowledge retrieval
Official government sources first, law-firm guidance and internal docs second, historical case outcomes third — never open-internet retrieval.
→→
DRAFT → REVIEW → SEND
06
Human-in-the-loop review
Agents draft, hosts approve. Every edit is implicit feedback, and automation thresholds rise as confidence scores do.
05
Outcome & impact
30%
reduction in immigration query handling effort
3
specialized agents with domain accuracy
↑
evaluation metrics improve each iteration
Early prototypes drafted operational responses with strong contextual awareness, evaluation metrics improved through iterative prompt tuning, and domain experts systematically validated outputs using the framework. The strategic impact is operational scalability — expert reasoning reused across thousands of cases without proportional headcount — plus knowledge centralisation away from key-person dependency, a foundation for further automation, and faster user support.
06
What I'd do differently
Instrument the review interface harder from the start. Edit distance became the passive quality signal, but we added it after the first cohort rather than before.
Set the escalation thresholds with the domain experts in the room, not afterwards. The first pass was too conservative and buried hosts in low-risk drafts they did not need to review.