Bedrock Multi-Model PDF Chat & Semantic Search Prototype
A Multi-Model Bedrock Prototyping Sandbox
A mobile services company wanted to know what AWS Bedrock could do for document Q&A and semantic search before committing to a production build. This was a one-month prototype: a Flask app that lets a user pick from five Bedrock models to answer questions about an uploaded PDF, and a separate set of scripts comparing three ways to find the right passage to answer from. It gave the client a working demo and a tested basis for the model and retrieval choices that followed.
The problem
The client is a mobile services company that wanted an AI assistant able to answer questions about its documents. AWS Bedrock had just made five structurally different foundation models available behind one API, and none of them were tested yet on the client's kind of question. Picking one without comparing it against the others meant picking blind. The two obvious ways to find the right passage in a document, a full embeddings pipeline or no retrieval at all, were both extremes: building production RAG before knowing if Bedrock fit would waste time, and answering from a whole document with no retrieval would not scale past a page or two. What they needed was a fast, side-by-side comparison of models and retrieval strategies before committing to infrastructure.
Constraints
- ·One-month prototyping window, not a production build.
- ·No client data available yet, so retrieval tests ran against public datasets.
- ·Five different Bedrock model APIs in scope, each with its own request and response shape.
- ·Both the source document and the user's question needed to work in languages other than English.
- ·Built and run by a sole engineer.
Architecture
A Flask endpoint fetches a chosen PDF from S3, detects its language and the question's language with Comprehend, translates both to English with Translate, then routes the request to one of five Bedrock model branches (Claude, Cohere Command, AI21 Jurassic-2, Stability Diffusion, Titan Image), each building its own request body for invoke_model. Text answers are translated back to the user's chosen language before being shown in the chat log; image outputs are written to the static folder. Separately, a hardcoded retrieval hack has Claude classify a question against a fixed list of section titles and maps the result to hardcoded PDF page numbers, tested as an alternative to real retrieval. A third, unconnected set of scripts indexes a public movie dataset into OpenSearch with a knn_vector mapping, using both Titan Embeddings and sentence-transformers, and compares that against a TF-IDF vectorizer over the same extracted text.
- ›Flask /send_message endpoint (flaskpython.py): model, PDF, and language dropdowns, routes to one of five model branches
- ›AWS Comprehend + Translate: detect-then-translate-to-English for both the document and the question, then translate the reply back
- ›Hardcoded topic-to-page lookup (ga_test_sematic.py, work.py): Claude picks one of ~40 fixed section titles, an if-chain maps it to page numbers, PyPDF2 extracts those pages
- ›OpenSearch KNN index (opensearch_test.py/test2.py/test3.py): knn_vector/hnsw mapping over movie-plot embeddings from Titan Embeddings or sentence-transformers
- ›TF-IDF alternative (semantic_test.py, my_work/work.py): PyMuPDF extraction, nltk sentence tokenizing, scikit-learn TfidfVectorizer and cosine similarity
How one request flows
- 01The user picks a PDF, a model, and a language from three dropdowns, and types a question.
- 02The source PDF is fetched from S3, and if a hardcoded page range applies, Claude first classifies the question against the fixed section-title list to narrow which pages to read.
- 03Comprehend detects the language of the document and the question, and anything not already in English goes through Translate.
- 04The Flask endpoint routes the request to the chosen model's branch, each building its own request body for bedrock.invoke_model.
- 05The model answers the question using the extracted PDF text, or generates an image if an image model was chosen.
- 06Text answers are translated back into the user's chosen language before being appended to the chat log.
- 07Image outputs are written to the static folder and the page reloads to show them.
Engineering decisions
Five models, one endpoint
What. Wired all five in-scope Bedrock models behind one Flask endpoint with a branch per model.
Why. Let the client compare Claude, Cohere, AI21, Stability Diffusion, and Titan Image from one UI instead of five separate demos.
Tradeoff. A longer if-chain to maintain, accepted because it cost less than building and hosting five apps for a one-month comparison.
A lookup table instead of embeddings, on purpose
What. Before building an embeddings pipeline, tested a hand-coded shortcut: Claude classifies a question against a fixed list of roughly 40 section titles from the PDF's own table of contents, and an if-chain maps the title to page numbers.
Why. Got a working PDF Q&A demo in days rather than weeks, and gave a reference point to judge whether the extra cost of real embeddings was worth it.
Tradeoff. Does not generalize past this one document's structure, so it was never presented as retrieval, only as a fast baseline.
Managed embeddings versus open source
What. Ran the same OpenSearch knn_vector index twice: once with Titan Embeddings, once with the open-source sentence-transformers model.
Why. Comparing both against the same movie dataset showed whether Titan's managed convenience was worth its cost over a model that runs for free.
Tradeoff. Titan Embeddings adds an AWS call and cost per document; sentence-transformers runs locally but needs its own hosting later.
TF-IDF as a cheaper baseline
What. Ran a classic TF-IDF vectorizer over the same extracted PDF text and compared its matches by hand against the embedding-based results.
Why. Made the case for embeddings concrete instead of assumed, before spending on a neural model.
Tradeoff. TF-IDF has no real semantic understanding, so it was kept as a sanity check, not a candidate for production.
Public data while the client's data was not ready
What. Ran every retrieval test against a public movie dataset and a generic salary dataset instead of client documents.
Why. The client's own documents were not available for this exploration, and using public data kept the comparison honest without touching anything client-owned.
Tradeoff. None of the retrieval numbers transfer directly to the client's real documents, so the comparison judged setup cost and mechanism, not final accuracy.
What changed along the way
- →The OpenSearch indexing script went through three passes (opensearch_test.py through opensearch_test3.py) before settling on a single bulk-index call.
- →Started with the hardcoded topic-to-page lookup to get something working fast, then used it as the baseline the embedding-based tests were judged against.
- →A separate IAM-signed OpenSearch client (opensearch_upload.py) was tried as an alternative to basic auth, but stayed a disconnected spike and was not carried into the main comparison.
Security and reliability
- ·No client data in the tests: every retrieval experiment ran against public datasets.
- ·Credentials kept out of source: the OpenSearch cluster's host and basic-auth credentials, hardcoded during the prototype, were moved to environment variables during cleanup.
- ·No production hardening: the Flask app has no authentication layer and no monitoring, appropriate for a one-month comparison, not carried into a production recommendation.
Metrics
What shipped
- ✓A working multi-model Bedrock chat app that answers questions about an uploaded PDF, translating the document and the reply through the user's chosen language.
- ✓A side-by-side comparison of a hardcoded topic-to-page lookup versus real embedding-based retrieval as two different ways to answer questions about a document.
- ✓Working OpenSearch KNN vector search index and query path proven against a public movie dataset, using both AWS Titan Embeddings and open-source sentence-transformers.
- ✓A TF-IDF-based retrieval alternative implemented and compared against the embedding-based approach on the same extracted PDF text.
Lessons learned
- ·A hand-coded lookup table is a legitimate way to get a fast demo in front of a client, as long as it is never described as retrieval.
- ·Comparing a managed embedding model against an open-source one on the same data is worth doing before committing to either.
- ·TF-IDF is a cheap, useful sanity check to run before assuming an embedding model is necessary.
- ·Keeping prototype tests on public data, not client documents, is the safer default when the client's own data is not ready yet.