Enterprise End-to-End ML Pipeline Development & Advanced Analytics Platform
Fine-Tuning a Small Model to Judge CVs Against Job Postings
A recruitment technology company wanted a model that could write a job posting, draft interview questions, and judge how well a CV fits a role, without paying for a frontier model API call every time. I built a four-stage pipeline that generates a labeled training set with GPT-4o, a job posting, five CVs, and five structured evaluations per posting, then fine-tuned a small open-weights model on it with a LoRA adapter. The result is a working proof of concept that shows the approach fits before committing to a larger training run.
The problem
A recruitment technology company wanted an AI assistant for screening: write a job posting from a few inputs, generate interview questions for a candidate, and score how well a CV matches the role. Calling a frontier model API for every one of these requests works, but the cost scales directly with volume and the output cannot be tuned to a house style or scoring criteria without prompt engineering that has to be maintained forever. There was no existing labeled dataset of job postings and CV evaluations to fine-tune on, and using real candidate CVs to build one raises its own privacy problem. They needed a training set built without real candidate data, one that included CVs that fail the role as well as ones that pass it, so the resulting model would actually discriminate instead of approving everything it sees.
Constraints
- ·No real candidate data. Training data had to be synthetic, generated rather than sourced from real CVs or job applications.
- ·Both outcomes needed. The dataset had to include CVs that fail a posting, not only ones that pass, or the fine-tuned model learns to say yes to everything.
- ·Small dataset, small budget. 500 job postings generated through a paid API, which meant parameter-efficient fine-tuning (4-bit, LoRA) instead of a full fine-tune.
- ·Single engineer covering data design, the generation pipeline, validation, and the fine-tune.
Architecture
The full cross product of 3 company sizes, 10 sectors, and 20 positions is shuffled and the first 500 rows become the training set, split into 20 chunks. Four GPT-4o stages then run per chunk: a job posting generator, a CV generator that returns 5 CVs per posting in one call, and a CV evaluator that returns 5 structured fit evaluations per posting in one call. A validation layer re-sorts each chunk and checks it has the expected shape, with tooling to pull, delete, or retry specific bad indices instead of a whole chunk. Once all 20 chunks pass, they are combined and re-indexed into one final dataset, which is formatted into instruction/input/output triples and used to fine-tune a 4-bit Llama-3-8B model with a LoRA adapter via unsloth.
- ›JobPostingsDataProcessor: builds the 500-row input space and splits it into 20 chunks
- ›JobPostingGenerator: GPT-4o, 1 call per row, 4 concurrent workers, fixed job-posting template
- ›CVGenerator: GPT-4o, 5 CVs per call, split on a literal separator, 30-50% variance from the posting, mixed pass/fail by design
- ›CVEvaluator: GPT-4o, 5 structured evaluations per call, position/education/sector/company-size relevance plus an overall fit verdict
- ›DataOrganizer + DataChecker: re-sort by index, verify chunk shape (25 postings, 5 CVs each), pull/delete/retry specific bad indices
- ›EvaluationsCombiner: merges and re-indexes the 20 validated chunks into one final.json (500 records)
- ›finetuning.ipynb: Llama-3-8B loaded in 4-bit via unsloth, LoRA adapter (rank 16, alpha 16, all attention/MLP projection layers), trained with SFTTrainer
How one request flows
- 01Build the input space: shuffle the 3x10x20 cross product of company size, sector, and position, keep the first 500 rows, split into 20 chunks of 25.
- 02Generate a job posting per row with GPT-4o, 4 rows processed concurrently per chunk.
- 03Generate 5 CVs per posting in one GPT-4o call, deliberately varied 30 to 50 percent from the posting's stated requirements, with a mix of CVs meant to pass and to fail.
- 04Evaluate the 5 CVs in one GPT-4o call, covering position, education, sector, and company-size relevance plus an overall fit call. This becomes the training label.
- 05Validate the chunk: re-sort back into original order, check for 25 postings and 5 CVs each, pull and rerun only the specific bad indices found.
- 06Assemble the final dataset once all 20 chunks pass, combining and re-indexing them into one file of 500 labeled examples.
- 07Fine-tune: format the dataset into instruction/input/output triples and train a LoRA adapter on a 4-bit Llama-3-8B model.
Engineering decisions
Synthetic data, on purpose, with built-in failures
What. Generated CVs with GPT-4o instead of sourcing real candidate data, with an explicit instruction to vary 30-50% from the posting and include failing candidates.
Why. Real candidate CVs raise a privacy problem on their own, and a dataset of only strong matches trains a model that approves everything.
Tradeoff. Synthetic data is less noisy than real-world CVs, traded for a dataset that is legal to build and includes negative examples by design.
Five items per call, not five calls
What. CVGenerator and CVEvaluator each ask for 5 items in a single completion, split on a literal separator, instead of looping 5 separate API calls.
Why. A fifth of the billed requests for the same output.
Tradeoff. A parsing step that depends on the model following the separator exactly, covered by a length-based validation check downstream rather than avoided.
Chunking as the unit of retry
What. Every generation stage works on 20 chunks of 25 rows, with tooling to pull, delete, or retry specific bad indices inside a chunk.
Why. A concurrent API call can fail on any single row, and chunking bounds the damage to 25 rows instead of the full 500.
Tradeoff. More bookkeeping (chunk files, index tracking) than one long job, in exchange for cheap, targeted recovery.
LoRA and 4-bit quantization instead of a full fine-tune
What. Loaded Llama-3-8B in 4-bit and trained a rank-16 LoRA adapter across the attention and MLP projection layers.
Why. Fits the training run in a single GPU session and keeps the base model reusable.
Tradeoff. Smaller effective capacity to change the model than a full fine-tune, accepted for a first proof of concept.
A proof-of-concept training run, stated as one
What. Trained for 60 steps at an effective batch size of 8, seeing roughly 480 of the 500 examples once.
Why. Enough to confirm the data format and pipeline work end to end before committing more GPU budget.
Tradeoff. Not enough training to claim a production-grade model, and this log states that plainly rather than inflating the result.
A managed GPU platform was priced out, not built
What. An AWS SageMaker notebook setup (ml.p3.8xlarge, 2 data scientists, Frankfurt region) was estimated for a heavier training workflow.
Why. The pipeline as delivered runs the OpenAI API for generation and a single GPU notebook for fine-tuning, which was enough for the proof of concept.
Tradeoff. The managed infrastructure stays a considered option for scaling up, not something built and left unused.
What changed along the way
- →The first version of the pipeline (flat scripts, hardcoded chunk ranges per run, in what is now old_program/) was refactored into a package with explicit constructor arguments once the retry-by-chunk pattern proved out.
- →The CV variance target grew from an initial plan of about 20 percent (files/steps.md) to 30-50 percent in the shipped prompt, after early CVs came back too close to the posting to produce meaningful failing examples.
- →Validation started as a simple re-sort and grew into pull/delete/retry tooling for specific bad indices, once whole-chunk reruns proved wasteful when only 1 or 2 rows in a chunk of 25 had failed.
Security and reliability
- ·No hardcoded credentials: the OpenAI key loads from the environment at runtime and every stage fails fast with a clear error if it is missing.
- ·Bounded concurrency: each generation stage threads 4 requests at a time, not the whole chunk at once, to keep API load and cost predictable.
- ·Chunk-level fault isolation: a failed API call drops only its row, logged, and the chunk it belongs to is small enough to rerun cheaply.
- ·Validation before assembly: every chunk is checked for the expected shape before it is combined into the final training file.
Metrics
What shipped
- ✓Delivered a four-stage synthetic data pipeline that produces a labeled dataset of 500 job postings, 2,500 CVs, and 2,500 structured evaluations.
- ✓Built chunk-level validation and retry tooling so a failed batch can be repaired without regenerating the whole dataset.
- ✓Fine-tuned a 4-bit Llama-3-8B model with a LoRA adapter on the assembled dataset as a working proof of concept.
- ✓Left a documented, resumable pipeline, by chunk range and by individual bad index, ready to scale the dataset further.
Lessons learned
- ·Batching multiple items into one LLM call cuts cost, but it trades away response reliability, and that tradeoff needs a validation layer built in from the start.
- ·A synthetic dataset that is all positive examples trains a model that just says yes. Deliberately generating some failing CVs is what makes the fit judgment meaningful.
- ·Chunking a generation pipeline into small, retryable units matters more than parallelizing harder. Most of the practical value came from being able to resume, not from raw speed.
- ·A small LoRA fine-tune is a fast, cheap way to confirm an approach works before committing GPU budget to a longer run.