
TL;DR
Fine-tune a security model for a narrow behavior: turn permitted evidence into a structured finding and a reviewable remediation proposal. Keep current vulnerability facts in the evidence bundle, preserve failed assessments and uncertainty, and compare the adapted model against the unchanged model using identical inputs.
This article selects Qwen2.5-1.5B-Instruct and specifies a LoRA training path, a small synthetic dataset plan, explicit loss masking, comparison interfaces, and model/adapter records. The examples explain the mechanics; they do not establish enterprise security capability. No downloadable training package or measured model result is supplied.
Train a candidate, not an authority. Retain the baseline unless measured task improvement justifies the additional artifact and operating burden.
In This Article
- Introduction
- Define the Task Before Preparing the Dataset
- Select a Small, Inspectable Model Candidate
- Prepare Examples That Teach Evidence Use
- Separate Training from the Comparison
- Establish the Environment and Unchanged Baseline
- Make the Loss Mask Explicit
- Train a LoRA Adapter Without Updating the Base Weights
- Compare Decisions and Evidence, Not Just Training Loss
- Register the Adapter with Its Dependencies and Rollback Path
- Troubleshoot Before Expanding the Experiment
- Conclusion
Introduction
Example Enterprise has a versioned advisory store and an asset inventory that preserves missing observations. The next question is whether a model can consistently turn that evidence into a useful draft.
Consider two proposed answers. Both identify the correct component and recommend a supported candidate version. One preserves the required staging test and warns that rollback can restore an affected build. The other omits both. That difference matters to the reviewer even when both answers contain valid JSON.
Supervised fine-tuning is a candidate response to a repeatable behavior problem like that. It is not the response to a missing deployment observation or a withdrawn supplier fix. This installment moves from the earlier model-selection decision into a bounded experiment, without giving the model scanner credentials, ticket-write access, or deployment authority.
The outcome is a reference implementation plan for preparation and comparison. The command examples describe interfaces to implement, rather than scripts available to download from this article. Qwen tokenization, inference, GPU training, masking tests, and task-quality comparison require execution and retained results before this design can be treated as validated.
Define the Task Before Preparing the Dataset
The model receives an asset and finding identifier, an assessment execution state, and a permitted bundle of versioned evidence passages. It returns a draft. It does not retrieve additional information, decide whether a scan is authorized, or mark remediation complete.
Define a finding-draft.schema.json contract with these groups of fields:
| Output group | Required content | Boundary |
|---|---|---|
| Identity and disposition | Asset ID, finding ID, proposed applicability, assessment execution | Preserve identifiers and execution state; applicability remains a proposal |
| Recommendation | Summary, supported target version, proposed action | A supplier-supported candidate is not an approved local change |
| Operating conditions | Prerequisites, rollback limitations, verification requirements | Use supplied procedures; disclose missing procedures |
| Evidence and uncertainty | Evidence IDs, unresolved conditions, review requirement | References must exist; human review remains required |
An evidence ID that resolves successfully is only the beginning of review. The cited passage must support the particular recommendation, including its product conditions and limitations.
For simple findings, a deterministic template may already satisfy this contract. Establish that alternative before paying to train a model to reproduce fields ordinary code can assemble.
Select a Small, Inspectable Model Candidate
Use Qwen/Qwen2.5-1.5B-Instruct for this lab. Its official model card identifies a 1.54-billion-parameter instruction-tuned causal language model with a 32,768-token context. The repository contains an Apache-2.0 license. These properties make it a concrete candidate for a bounded local experiment, not evidence that it is the best model for security analysis. Qwen: Qwen2.5-1.5B-Instruct
Pin the model and tokenizer to repository revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306. Do not use a moving main reference for the comparison. Preserve the license and review permitted use before internal distribution. Hugging Face
The reference placement is self-hosted lab inference and training. It makes the base model, adapter, template, and runtime available for inspection while keeping the synthetic evidence local. A hosted training service would introduce a separate data-processing and retention decision; approved inference access would not settle that decision.
The release pins below were checked against public releases and declared package dependencies on November 18, 2026. They are a candidate environment, not an end-to-end validated GPU stack. Hugging Face: TRL v1.18.1 project dependencies
| Component | Selected pin | Role |
|---|---|---|
| Python | 3.13 reference environment | Script execution |
| PyTorch | 2.10.0, appropriate CUDA build | Tensor computation and training |
| Transformers | 5.23.0 | Model, tokenizer, generation, and trainer infrastructure |
| TRL | 1.18.1 | Supervised fine-tuning trainer |
| PEFT | 0.21.3 | Parameter-efficient adaptation |
| Datasets | 5.2.0 | Local training and development tables |
| Accelerate | 1.16.0 | Training-device integration |
Use one visible CUDA GPU with bfloat16 support for this candidate profile. The implementation should reject unsupported hardware instead of silently changing precision or moving to CPU.
At two bytes per parameter, 1.54 billion frozen weights alone represent approximately 3.08 GB of storage. That is arithmetic, not a training-memory requirement. Activations, adapter parameters, gradients, optimizer state, temporary buffers, and runtime overhead require additional memory. Measure the specified one-step run before planning capacity or cost.
Prepare Examples That Teach Evidence Use
Each dataset record contains the task input, expected output, family identifier, split assignment, and synthetic-data admission information. Only the task input reaches inference. Expected answers and split labels must not leak into the model prompt.
A small starter dataset can use 22 synthetic examples: 12 training, four development, and six final demonstration cases. Cover affected and not-affected conditions, stale inventory, failed authentication, missing scanner checks, backported fixes, conflicting guidance, withdrawn fixes, and missing procedures. This is a dataset design, not an attached corpus.
The targets were checked against their authored conditions and machine-readable contract. They are not independently human-reviewed incident records. Before adapting this workflow to real findings, require security-owner review of the evidence, expected answer, permitted use, and operational consequences.
Sanitize Before Admission, Not During Training
Prepare real examples from a controlled export. Remove credentials, private keys, session material, personal information, and customer-specific details that the training task does not require. Preserve the relationships needed to understand the finding through consistent replacement identifiers.
Run appropriate secret detection and classification checks, then review the result. A basic credential-pattern rejection rule is only a tripwire, not proof that arbitrary data has been sanitized.
The prior knowledge-ingestion registry restricted training use. This tutorial does not override that restriction or automatically convert its source corpus into training data. The new synthetic dataset has a separate admission path. Keep the same distinction when adding licensed advisories, internal procedures, or reviewed operational records.
Split Related Cases Together
Keep every related product/advisory family within one split. Place all Example Renderer cases in the final demonstration set, including the stale-deployment and withdrawn-fix branches. Use different fictional component families for training and development.
This is a proposed group split, not a demonstrated temporal holdout. A small public example set can expose mechanical failures but cannot establish production false-positive rates, generalization across customers, or resistance to unfamiliar attacks.
For a later operational dataset, deduplicate related incidents and advisory revisions before splitting. Reserve later evidence and unfamiliar product families for final evaluation. Once a team tunes against that final set, it becomes development material and needs replacement.
Separate Training from the Comparison
The important boundary is that development evidence selects the candidate; final evidence does not participate in gradient updates or checkpoint selection.

Both variants receive the same frozen evidence, system instruction, tokenizer, generation settings, and output checks. Updating retrieval for one variant and not the other would confound the adaptation result.
Record hashes for the dataset, prompt, configuration, model manifest, and prediction files. Also preserve software versions and device information. Hashes help detect changes; they do not establish who approved an artifact.
Establish the Environment and Unchanged Baseline
Implement and review the script interfaces below before running these commands. First create an isolated environment and a requirements file containing the selected pins. Install the appropriate CUDA-enabled PyTorch 2.10.0 build for the lab’s driver using the official distribution, then install the remaining requirements. The validation and test files must be authored for the actual implementation.
python3.13 -m venv .venv . .venv/bin/activate # Install the approved CUDA-enabled PyTorch build first. python -m pip install -r requirements-training.txt python -m pip check python -m pip freeze > environment.resolved.txt python validate.py python -m unittest -v test_lab.py
Use these checks to validate dependency consistency and fixture structure, then assess model quality separately. Preserve the resolved environment and approved package hashes with the experiment. Top-level pins alone do not freeze every transitive dependency or GPU-driver component.
Download Once, Then Operate Locally
Implement prepare_model.py as the explicit connected-intake step: download selected files from the pinned public repository and create a local SHA-256 manifest. Hugging Face’s download API supports revision-specific snapshots and file selection. Hugging Face: Download files from the Hub
python prepare_model.py --download export CUDA_VISIBLE_DEVICES=0 export HF_HUB_DISABLE_TELEMETRY=1 export HF_HUB_OFFLINE=1 python train.py --check-only
Review and protect the downloaded model directory before transferring it into the restricted lab. Configure loaders to use local files, safetensors weights, and trust_remote_code=False. These choices constrain loading behavior; they are not an assertion that the model or dependency stack is harmless.
Disable experiment-reporting integrations and library telemetry separately. TRL documents a trainer-instantiation telemetry request and supports the Hugging Face telemetry-disable and offline environment variables. Enforce the network boundary outside Python as well; an environment variable is not a firewall. Hugging Face: Usage Stats Collection
The actual-tokenizer preflight checks the train/development sequences and masks without loading model weights. Once it succeeds, capture the unchanged model’s development answers:
python predict.py --split development \ --output runs/baseline-development.jsonl python score.py --split development \ --baseline runs/baseline-development.jsonl \ --output runs/baseline-development-checks.json
Read the raw answers against their passages. Fix missing evidence, ambiguous instructions, or an unsuitable output contract before training. If the unchanged model already meets the task requirements, stop with that baseline.
Make the Loss Mask Explicit
Supervised fine-tuning minimizes prediction error on selected target tokens. For this task, the input evidence supplies context, while the assistant’s reviewed answer supplies the supervised target. Masking an input token removes its direct contribution to the loss; it does not prevent that input from influencing learning or make sensitive input safe to include. Hugging Face: SFT Trainer
Use the model’s native chat template rather than inventing role separators. Hugging Face’s template documentation distinguishes a prompt ending at the assistant-generation boundary from a complete conversation containing the answer. Hugging Face: Chat templates
The preparation code tokenizes both forms, verifies that the prompt tokens are an exact prefix of the full sequence, then builds labels:
# Exact token-prefix equality is checked before this assignment. labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]
Here, -100 marks positions excluded from the training loss. Implement the completion collator to mask padding by position while retaining the assistant’s end-of-turn token as a target. Do not mask every occurrence of a token merely because that token can be used for padding. TRL 1.18.1 SFTTrainer implementation
Reject an example that exceeds the 2,048-token lab limit. Do not truncate away an applicability condition, rollback warning, or the end of the target JSON. The runner precomputes input_ids, attention_mask, and labels, then tells TRL to skip dataset preparation. Packing is disabled so unrelated examples are not concatenated.
This explicit path avoids depending on whether a template supplies an assistant-token mask. The native-tokenizer prefix check remains mandatory after changing the model or template.
Train a LoRA Adapter Without Updating the Base Weights
Low-Rank Adaptation, or LoRA, adds small trainable matrices to selected model layers. Hugging Face PEFT exposes their rank, scaling, dropout, and target modules. In this experiment, adapt the attention query and value projections while leaving the base parameters frozen. Hugging Face: LoRA
The following excerpt from the training path shows the adaptation and data-boundary choices. The complete train.py includes loading, version checks, device checks, checkpoint handling, and artifact records.
peft_config = LoraConfig(
r=8,
lora_alpha=16,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj"],
bias="none",
task_type="CAUSAL_LM",
)
# train_dataset and development_dataset already contain masked labels.
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=development_dataset,
processing_class=tokenizer,
data_collator=CompletionCollator(tokenizer.pad_token_id),
peft_config=peft_config,
)
The configuration uses a microbatch of one, gradient accumulation of four, learning rate 0.0001, one epoch, and bfloat16 base weights. These are teaching defaults, not tuned recommendations. The runner checks that only LoRA parameters are trainable and records their actual count.
Use activation checkpointing to trade recomputation for reduced activation storage. Transformers 5.23.0 names the settings activation_checkpointing and activation_checkpointing_kwargs; use those version-specific names in the implementation. This profile deliberately avoids quantization and its additional implementation variables. It is LoRA, not QLoRA. Transformers 5.23.0 TrainingArguments
Begin with a one-optimizer-step smoke run. Then, only after baseline review establishes a worthwhile experiment, run the separate candidate:
python train.py --run-training --smoke --output runs/smoke python train.py --run-training --output runs/lora-v1
The smoke artifact is labeled separately and rejected by the candidate-comparison path. A full run starts from the unchanged base, not the smoke adapter. With only twelve training examples, its purpose is to exercise the adaptation pipeline, not produce a deployable security specialist.
Record peak device allocation, elapsed work, and review effort. Fixed seeds help control variation, but PyTorch explicitly warns that identical seeds do not guarantee reproducibility across releases, platforms, or CPU/GPU execution. PyTorch Documentation
Compare Decisions and Evidence, Not Just Training Loss
Generate the candidate’s development answers and compare them with the saved baseline:
python predict.py --split development \ --adapter runs/lora-v1/adapter \ --output runs/candidate-development.jsonl python score.py --split development \ --baseline runs/baseline-development.jsonl \ --candidate runs/candidate-development.jsonl \ --output runs/development-comparison.json
The comparator rejects mismatched input, prompt, configuration, software, device, CUDA, or generation metadata. Its automated checks cover schema validity, exact decision fields, evidence-reference existence, omitted uncertainty, and completion termination.
Those checks are deliberately narrower than a quality verdict. A citation can exist without supporting the sentence. A nonempty rollback field can contain the wrong procedure. Human review must assess passage support, missing prerequisites, supported alternatives, and operational consequences.
The final Renderer cases preserve these distinctions:
| Case | Expected interpretation |
|---|---|
| Old EE-DEP-001 observation, unresolved EE-DEP-002 | Current applicability stays unknown even though revision 1 supplies candidate 1.4.3 |
| Same inventory, withdrawn-fix revision 2 | Applicability stays unknown; supported target becomes null |
| Failed authenticated assessment without fresh observations | Execution remains failed; no clean disposition is inferred |
| Separate counterfactual with confirmed affected conditions but no scanner check | Applicability can be affected while assessment execution remains unavailable |
These are expected targets, not observed model answers. The missing-check case is a separate authored branch, not a modification to the earlier stale inventory.
After selecting and freezing the candidate through development review, run both variants with --split test and compare their saved outputs. Define rejection conditions before that run. For this lab, a fabricated fix, corrupted assessment state, or unsupported clean disposition is reason to reject the candidate regardless of improved formatting.
No baseline-versus-adapter improvement has been measured for this article. Dataset validation, masking tests, GPU compatibility, training convergence, and security effectiveness all require recorded execution against the completed implementation. A dependency pin or code excerpt is not a passing test result.
Register the Adapter with Its Dependencies and Rollback Path
A PEFT adapter is not a standalone replacement model. Its checkpoint contains adapter parameters and configuration; inference still needs the compatible base model. Preserve that relationship when storing or distributing the result. Hugging Face: PEFT checkpoint format
Begin EE-MODEL-001 as an experiment specification with unmeasured fields set to null. A completed run should write actual artifact hashes, configuration and dataset identities, parameter counts, and training metrics. Keep its status trained but not evaluated until comparison is complete. Training must not promote it or enable tools.
Keep inference artifacts separate from resumable training state. Resuming requires the relevant checkpoint, optimizer/scheduler state, and random-state information, not merely an adapter directory. Use only trusted internal checkpoints and reject a resume when the base, dataset, prompt, configuration, or pinned runtime differs. Hugging Face: Transformers v5.23.0 TrainingArguments implementation
For rollback, restore the previously evaluated base/adapter and compatible prompt/runtime bundle. Do not roll back current advisory corrections or access revocations with it. Removing an adapter cannot justify resurrecting a withdrawn fix.
Security engineering owns target-answer review; the AI platform team owns the training and serving artifacts; the existing change authority still owns operational access. Dataset permission, successful training, model approval, and authorization to act remain separate decisions.
Troubleshoot Before Expanding the Experiment
| Symptom | Inspect first | Corrective action |
|---|---|---|
| Context exceeds the limit | Full rendered input and target | Reduce irrelevant context or revise the documented profile; never silently truncate |
| Prompt-prefix check fails | Model revision and native template | Stop and correct tokenization; do not guess a character-to-token boundary |
| Loss is invalid or no useful learning occurs | Label mask, supervised-token count, trainable parameters | Confirm nonempty completion targets and intended adapter attachment |
| GPU runs out of memory | Sequence length, activation storage, optimizer and runtime allocation | Measure a new bounded profile and rerun both comparison paths |
| JSON improves but fixes become unsupported | Raw answers and passage-level review | Reject the candidate; inspect examples and overfitting rather than celebrating loss reduction |
A small successful run is a reason to continue evaluation, not to add permissions. Expand reviewed data and failure coverage before increasing task scope.
Conclusion
A useful fine-tuning experiment begins with a defect in a defined task and ends with evidence about whether the adaptation corrected it. The deliverable is not merely an adapter file. It is the relationship among the base model, permitted examples, native template, loss mask, comparison inputs, results, and operating decision.
For Example Enterprise, keep the unchanged model as the reference, use synthetic cases to understand the mechanics, and admit real examples only through an explicit review process. Preserve current facts in retrieval and consequential authority outside the model.
The next installment, Test Your Security Model Before Giving It Operational Access, expands the evaluation beyond this small adaptation experiment into retrieval quality, model behavior, scanner coverage, and workflow enforcement.
Promote demonstrated behavior, not the fact that training completed.
Related Reading
- Test Your Security Model Before Giving It Operational Access
- Keep Security Agents Current with Trusted Advisories and Best Practices
- Build Security Agents with Controlled Access to Scanning Tools
External References
- Qwen: Qwen2.5-1.5B-Instruct
- Qwen: LICENSE at revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306
- Hugging Face: TRL v1.18.1 project dependencies
- Hugging Face: SFT Trainer
- Hugging Face: TRL v1.18.1 SFTTrainer implementation
- Hugging Face: Transformers v5.23.0 TrainingArguments implementation
- Hugging Face: Chat templates
- Hugging Face: LoRA
- Hugging Face: PEFT checkpoint format
- Hugging Face: Download files from the Hub
- Hugging Face: Usage Stats Collection
- PyTorch: Reproducibility, version 2.10 documentation
2 thoughts on “Fine-Tune Your Own Security Model on Reviewed Findings and Remediations”