Enterprise AI-Powered Multi-Agent Booking Platform
AI Assistant and Booking Backend for a Travel Agency
A travel agency needed an AI assistant that could answer detailed Umrah and Hajj questions and turn them into bookings, without inventing facts. I built a standalone AI backend from scratch: a custom multi-agent RAG system that answers only from the agency's own catalogue, keeps conversations in memory, and hands ready-to-book leads to the sales flow. It let the agency give accurate, round-the-clock support in two languages, and, by their report, it lifted booking conversion.
The problem
A travel agency sells Umrah and Hajj packages. Every day, customers ask the same kinds of questions before they book: which hotel is closest to the Haram, what a package includes, how to prepare, what documents are needed. The questions come in Turkish and English, at all hours, and each one needs a specific, correct answer drawn from the agency's own tours, hotels, and guides. A generic chatbot fails here, because it invents details when it does not know, which is dangerous when money and a religious trip are involved, and it cannot pull a real hotel or package out of the agency's live catalogue. The client needed an assistant that answers only from their own data, keeps the thread of a conversation, and runs as its own backend service behind the website and the user system.
Constraints
- ·A backend tier, not a replacement. It had to slot behind an existing website and user system and talk to both, without owning either.
- ·Grounded answers only. For a paid religious journey, a confident wrong answer is worse than no answer.
- ·Two first-class languages, Turkish and English, not one translated into the other.
- ·Managed models. Inference runs on AWS Bedrock, so the design lives within its latency, per-call cost, and regional availability.
- ·One AI engineer, built to hand over. The system had to stay simple enough for a second person to run after training.
- ·A scale target of ten million-plus users from day one.
Architecture
A Flask backend. Each chat question is read first by an LLM router (one Bedrock call, Claude 3.7 Sonnet) that returns a single 12-field JSON: the chosen agent plus extracted trip fields and a few rewritten search phrases. The request is dispatched to one of four specialised agents (hotel, tour, blog, general), which retrieves from in-memory vector stores and streams a grounded answer from Bedrock. Sessions and history live in DynamoDB. Two in-process background-thread queues handle persistence and live quality scoring. The router and agent prompts are drawn from an editable prompt layer and personalized with the user's quiz answers.
- ›Flask API: /stream (public) and /stream_secure (token-checked), plus /update to refresh vectors
- ›Agent router: one non-streaming Bedrock call (Claude 3.7 Sonnet, 3.5 fallback) returning a 12-field JSON
- ›Four specialised agents: hotel (multi-query dual-vector), tour, blog, general (no retrieval)
- ›Retrieval: in-memory brute-force cosine over pandas copies of three LanceDB stores, Titan embeddings
- ›Generation: streaming Bedrock, with an in-band metadata marker for chat title and source ids
- ›DynamoDB: ChatsDB (sessions, history, 4 modes, TTL) and an LLMJudge table
- ›Two in-process queues on background threads: message persistence and the live LLM-Judge
- ›Prompt layer: editable prompt files with an admin UI, plus quiz personalization wrapped into router and agent prompts
How one request flows
- 01The question arrives at the Flask API, public or behind a token check. The API reads the session and past history from DynamoDB first, so the answer has context.
- 02A router reads the question first. One Bedrock call (Claude 3.7 Sonnet) returns a single JSON object: the chosen agent, the extracted fields (destination, hotel, star, budget, people, date), and a few rewritten search phrases. The prompt is editable and personalized with the user's quiz answers.
- 03The right agent takes over: hotel, tour, blog, or general. Each has its own prompt and its own slice of the data.
- 04Retrieval, not guessing. The agent embeds the query and scores it against records held in memory. The hotel path fans the question into several phrases and searches two vectors, one for the description and one for the name.
- 05The model writes the answer from the retrieved records and streams it out, plus a machine-readable tail with the conversation title and the source ids.
- 06Memory and quality are handled off the request thread: the turn is queued for saving and, separately, scored by the live LLM-Judge, both on background threads.
Engineering decisions
Routing by extraction, not keywords
What. The router extracts meaning into a fixed set of fields and picks an agent from that, rather than matching keywords.
Why. Keyword rules break the moment a user phrases things their own way, and the extracted fields are logged in plain text, so wrong routes are easy to see and fix.
Tradeoff. One extra model call per turn, accepted because a misrouted answer costs more than a few hundred milliseconds.
A custom agent framework instead of a library
What. Built the agent layer by hand rather than adopting a heavy framework like LangChain.
Why. Direct control over how many model calls each turn costs, how state passes between steps, and how a failing agent is contained.
Tradeoff. More code to write and own, and no community plugins. Worth it for cost and failure control at scale.
Retrieval kept in memory
What. Each store is loaded into memory once at startup and searched there. The hotel path is multi-query dual-vector, the lighter agents run a single query.
Why. A query never waits on a database round-trip, which removes a moving part from the hot path.
Tradeoff. Each instance holds the full catalogue in memory and re-scores it per query, fine at this catalogue size.
Behavior tunable without a deploy
What. Router and agent prompts live in editable text files, changed through an admin screen with a reset to original.
Why. The team could adjust the assistant's behavior without waiting on an engineer or a release.
Tradeoff. These edits are runtime state, so a redeploy resets them unless the files are persisted. The limit was flagged, not hidden.
Personalized by the user's own answers
What. The user's travel-quiz answers are folded into both the router prompt and the chosen agent's prompt.
Why. The same context shapes both the routing decision and the reply, instead of being bolted on at the end.
Tradeoff. A per-user fetch on the request path, cached in-process.
Sessions that fit real usage
What. Four session modes from the same code: anonymous continuous, fresh-every-message, fully persistent, and stateless.
Why. Not every user logs in, and the model matches how people actually use the assistant.
Tradeoff. Temporary sessions expire on their own through a DynamoDB TTL.
A judge that scores every turn
What. A second model scores each interaction on three axes: right agent, relevant context, good answer.
Why. It turns a vague sense that the bot feels off into a measurable signal.
Tradeoff. Runs live on a background thread and writes to its own table, so it never slows the reply.
Slow work off the request thread
What. Saving history and scoring quality each sit behind an in-process queue and worker thread.
Why. The reply streams while writes are batched to the database in the background, with no external broker.
Tradeoff. This state lives inside the process, accepted for the persistence path and watched elsewhere.
Grounded answers with sources named
What. Retrieval agents answer only from what they pulled and return the ids of the records used, and a general agent handles off-catalogue chat.
Why. For a paid religious trip, refusing to invent is the whole point.
Tradeoff. The general fallback is not grounded, by design, because there is nothing to retrieve.
What changed along the way
- →Started as a single app, then split the agents into their own modules and moved persistence and quality scoring onto background threads as load grew.
- →Moved to flexible database schemas once new data types kept arriving with different fields.
- →Gave each agent its own prompt after one shared prompt proved too blunt.
- →Added routing rules from real misroutes, for example sending price and payment questions to the guide agent.
Security and reliability
- ·A public endpoint and a token-checked secure endpoint that validates the caller's token before answering.
- ·Secrets kept out of the code and read from the environment.
- ·Sessions in DynamoDB with a time-to-live, so temporary data cleans itself up.
- ·Graceful under pressure: when the primary model is throttled, generation falls back to a second model.
- ·Recovery: vector stores are backed by S3 and rebuilt by the sync job, so a bad index can be restored.
- ·Observability: every request records its own performance metrics and quality scores to a dedicated table.
Metrics
What shipped
- ✓Shipped a standalone AI backend that answers only from the agency's own tours, hotels, and guides, and names the records it used.
- ✓Routes each question to one of four specialised agents and runs at all hours in Turkish and English.
- ✓Turns a chat into a ready-to-book lead by passing the extracted booking fields to the sales flow.
- ✓Behavior tunable without a deploy through an editable prompt layer, and personalized with each user's quiz answers.
- ✓Scores its own answers on every turn through the live LLM-Judge, and records per-request metrics.
- ✓Handed over clean, with a second engineer trained to run and extend it.
Lessons learned
- ·Routing by extraction, with every decision logged in plain text, made wrong answers debuggable in minutes instead of hours.
- ·For trust, grounding and naming your sources mattered more than reaching for a bigger model.
- ·An automated judge turned a vague sense that the bot felt off into a number I could track and push on.
- ·A hand-built agent framework paid for itself in cost control and failure containment.
- ·Keeping the search in memory removed a moving part from the hot path, the right call at this data size.