Fine-Tune Your Own Security Model on Reviewed Findings and Remediations

TL;DR

Fine-tune a security model for a narrow behavior: turn permitted evidence into a structured finding and a reviewable remediation proposal. Keep current vulnerability facts in the evidence bundle, preserve failed assessments and uncertainty, and compare the adapted model against the unchanged model using identical inputs.

This article selects Qwen2.5-1.5B-Instruct and specifies a LoRA training path, a small synthetic dataset plan, explicit loss masking, comparison interfaces, and model/adapter records. The examples explain the mechanics; they do not establish enterprise security capability. No downloadable training package or measured model result is supplied.

Train a candidate, not an authority. Retain the baseline unless measured task improvement justifies the additional artifact and operating burden.

In This Article

Introduction

Example Enterprise has a versioned advisory store and an asset inventory that preserves missing observations. The next question is whether a model can consistently turn that evidence into a useful draft.

Consider two proposed answers. Both identify the correct component and recommend a supported candidate version. One preserves the required staging test and warns that rollback can restore an affected build. The other omits both. That difference matters to the reviewer even when both answers contain valid JSON.

Supervised fine-tuning is a candidate response to a repeatable behavior problem like that. It is not the response to a missing deployment observation or a withdrawn supplier fix. This installment moves from the earlier model-selection decision into a bounded experiment, without giving the model scanner credentials, ticket-write access, or deployment authority.

The outcome is a reference implementation plan for preparation and comparison. The command examples describe interfaces to implement, rather than scripts available to download from this article. Qwen tokenization, inference, GPU training, masking tests, and task-quality comparison require execution and retained results before this design can be treated as validated.

Define the Task Before Preparing the Dataset

The model receives an asset and finding identifier, an assessment execution state, and a permitted bundle of versioned evidence passages. It returns a draft. It does not retrieve additional information, decide whether a scan is authorized, or mark remediation complete.

Define a finding-draft.schema.json contract with these groups of fields:

Output groupRequired contentBoundary
Identity and dispositionAsset ID, finding ID, proposed applicability, assessment executionPreserve identifiers and execution state; applicability remains a proposal
RecommendationSummary, supported target version, proposed actionA supplier-supported candidate is not an approved local change
Operating conditionsPrerequisites, rollback limitations, verification requirementsUse supplied procedures; disclose missing procedures
Evidence and uncertaintyEvidence IDs, unresolved conditions, review requirementReferences must exist; human review remains required

An evidence ID that resolves successfully is only the beginning of review. The cited passage must support the particular recommendation, including its product conditions and limitations.

For simple findings, a deterministic template may already satisfy this contract. Establish that alternative before paying to train a model to reproduce fields ordinary code can assemble.

Select a Small, Inspectable Model Candidate

Use Qwen/Qwen2.5-1.5B-Instruct for this lab. Its official model card identifies a 1.54-billion-parameter instruction-tuned causal language model with a 32,768-token context. The repository contains an Apache-2.0 license. These properties make it a concrete candidate for a bounded local experiment, not evidence that it is the best model for security analysis. Qwen: Qwen2.5-1.5B-Instruct

Pin the model and tokenizer to repository revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306. Do not use a moving main reference for the comparison. Preserve the license and review permitted use before internal distribution. Hugging Face

The reference placement is self-hosted lab inference and training. It makes the base model, adapter, template, and runtime available for inspection while keeping the synthetic evidence local. A hosted training service would introduce a separate data-processing and retention decision; approved inference access would not settle that decision.

The release pins below were checked against public releases and declared package dependencies on November 18, 2026. They are a candidate environment, not an end-to-end validated GPU stack. Hugging Face: TRL v1.18.1 project dependencies

ComponentSelected pinRole
Python3.13 reference environmentScript execution
PyTorch2.10.0, appropriate CUDA buildTensor computation and training
Transformers5.23.0Model, tokenizer, generation, and trainer infrastructure
TRL1.18.1Supervised fine-tuning trainer
PEFT0.21.3Parameter-efficient adaptation
Datasets5.2.0Local training and development tables
Accelerate1.16.0Training-device integration

Use one visible CUDA GPU with bfloat16 support for this candidate profile. The implementation should reject unsupported hardware instead of silently changing precision or moving to CPU.

At two bytes per parameter, 1.54 billion frozen weights alone represent approximately 3.08 GB of storage. That is arithmetic, not a training-memory requirement. Activations, adapter parameters, gradients, optimizer state, temporary buffers, and runtime overhead require additional memory. Measure the specified one-step run before planning capacity or cost.

Prepare Examples That Teach Evidence Use

Each dataset record contains the task input, expected output, family identifier, split assignment, and synthetic-data admission information. Only the task input reaches inference. Expected answers and split labels must not leak into the model prompt.

A small starter dataset can use 22 synthetic examples: 12 training, four development, and six final demonstration cases. Cover affected and not-affected conditions, stale inventory, failed authentication, missing scanner checks, backported fixes, conflicting guidance, withdrawn fixes, and missing procedures. This is a dataset design, not an attached corpus.

The targets were checked against their authored conditions and machine-readable contract. They are not independently human-reviewed incident records. Before adapting this workflow to real findings, require security-owner review of the evidence, expected answer, permitted use, and operational consequences.

Sanitize Before Admission, Not During Training

Prepare real examples from a controlled export. Remove credentials, private keys, session material, personal information, and customer-specific details that the training task does not require. Preserve the relationships needed to understand the finding through consistent replacement identifiers.

Run appropriate secret detection and classification checks, then review the result. A basic credential-pattern rejection rule is only a tripwire, not proof that arbitrary data has been sanitized.

The prior knowledge-ingestion registry restricted training use. This tutorial does not override that restriction or automatically convert its source corpus into training data. The new synthetic dataset has a separate admission path. Keep the same distinction when adding licensed advisories, internal procedures, or reviewed operational records.

Keep every related product/advisory family within one split. Place all Example Renderer cases in the final demonstration set, including the stale-deployment and withdrawn-fix branches. Use different fictional component families for training and development.

This is a proposed group split, not a demonstrated temporal holdout. A small public example set can expose mechanical failures but cannot establish production false-positive rates, generalization across customers, or resistance to unfamiliar attacks.

For a later operational dataset, deduplicate related incidents and advisory revisions before splitting. Reserve later evidence and unfamiliar product families for final evaluation. Once a team tunes against that final set, it becomes development material and needs replacement.

Separate Training from the Comparison

The important boundary is that development evidence selects the candidate; final evidence does not participate in gradient updates or checkpoint selection.

Both variants receive the same frozen evidence, system instruction, tokenizer, generation settings, and output checks. Updating retrieval for one variant and not the other would confound the adaptation result.

Record hashes for the dataset, prompt, configuration, model manifest, and prediction files. Also preserve software versions and device information. Hashes help detect changes; they do not establish who approved an artifact.

Establish the Environment and Unchanged Baseline

Implement and review the script interfaces below before running these commands. First create an isolated environment and a requirements file containing the selected pins. Install the appropriate CUDA-enabled PyTorch 2.10.0 build for the lab’s driver using the official distribution, then install the remaining requirements. The validation and test files must be authored for the actual implementation.

python3.13 -m venv .venv
. .venv/bin/activate

# Install the approved CUDA-enabled PyTorch build first.
python -m pip install -r requirements-training.txt
python -m pip check
python -m pip freeze > environment.resolved.txt

python validate.py
python -m unittest -v test_lab.py

Use these checks to validate dependency consistency and fixture structure, then assess model quality separately. Preserve the resolved environment and approved package hashes with the experiment. Top-level pins alone do not freeze every transitive dependency or GPU-driver component.

Download Once, Then Operate Locally

Implement prepare_model.py as the explicit connected-intake step: download selected files from the pinned public repository and create a local SHA-256 manifest. Hugging Face’s download API supports revision-specific snapshots and file selection. Hugging Face: Download files from the Hub

python prepare_model.py --download

export CUDA_VISIBLE_DEVICES=0
export HF_HUB_DISABLE_TELEMETRY=1
export HF_HUB_OFFLINE=1

python train.py --check-only

Review and protect the downloaded model directory before transferring it into the restricted lab. Configure loaders to use local files, safetensors weights, and trust_remote_code=False. These choices constrain loading behavior; they are not an assertion that the model or dependency stack is harmless.

Disable experiment-reporting integrations and library telemetry separately. TRL documents a trainer-instantiation telemetry request and supports the Hugging Face telemetry-disable and offline environment variables. Enforce the network boundary outside Python as well; an environment variable is not a firewall. Hugging Face: Usage Stats Collection

The actual-tokenizer preflight checks the train/development sequences and masks without loading model weights. Once it succeeds, capture the unchanged model’s development answers:

python predict.py --split development \
  --output runs/baseline-development.jsonl

python score.py --split development \
  --baseline runs/baseline-development.jsonl \
  --output runs/baseline-development-checks.json

Read the raw answers against their passages. Fix missing evidence, ambiguous instructions, or an unsuitable output contract before training. If the unchanged model already meets the task requirements, stop with that baseline.

Make the Loss Mask Explicit

Supervised fine-tuning minimizes prediction error on selected target tokens. For this task, the input evidence supplies context, while the assistant’s reviewed answer supplies the supervised target. Masking an input token removes its direct contribution to the loss; it does not prevent that input from influencing learning or make sensitive input safe to include. Hugging Face: SFT Trainer

Use the model’s native chat template rather than inventing role separators. Hugging Face’s template documentation distinguishes a prompt ending at the assistant-generation boundary from a complete conversation containing the answer. Hugging Face: Chat templates

The preparation code tokenizes both forms, verifies that the prompt tokens are an exact prefix of the full sequence, then builds labels:

# Exact token-prefix equality is checked before this assignment.
labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]

Here, -100 marks positions excluded from the training loss. Implement the completion collator to mask padding by position while retaining the assistant’s end-of-turn token as a target. Do not mask every occurrence of a token merely because that token can be used for padding. TRL 1.18.1 SFTTrainer implementation

Reject an example that exceeds the 2,048-token lab limit. Do not truncate away an applicability condition, rollback warning, or the end of the target JSON. The runner precomputes input_ids, attention_mask, and labels, then tells TRL to skip dataset preparation. Packing is disabled so unrelated examples are not concatenated.

This explicit path avoids depending on whether a template supplies an assistant-token mask. The native-tokenizer prefix check remains mandatory after changing the model or template.

Train a LoRA Adapter Without Updating the Base Weights

Low-Rank Adaptation, or LoRA, adds small trainable matrices to selected model layers. Hugging Face PEFT exposes their rank, scaling, dropout, and target modules. In this experiment, adapt the attention query and value projections while leaving the base parameters frozen. Hugging Face: LoRA

The following excerpt from the training path shows the adaptation and data-boundary choices. The complete train.py includes loading, version checks, device checks, checkpoint handling, and artifact records.

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

# train_dataset and development_dataset already contain masked labels.
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=development_dataset,
    processing_class=tokenizer,
    data_collator=CompletionCollator(tokenizer.pad_token_id),
    peft_config=peft_config,
)

The configuration uses a microbatch of one, gradient accumulation of four, learning rate 0.0001, one epoch, and bfloat16 base weights. These are teaching defaults, not tuned recommendations. The runner checks that only LoRA parameters are trainable and records their actual count.

Use activation checkpointing to trade recomputation for reduced activation storage. Transformers 5.23.0 names the settings activation_checkpointing and activation_checkpointing_kwargs; use those version-specific names in the implementation. This profile deliberately avoids quantization and its additional implementation variables. It is LoRA, not QLoRA. Transformers 5.23.0 TrainingArguments

Begin with a one-optimizer-step smoke run. Then, only after baseline review establishes a worthwhile experiment, run the separate candidate:

python train.py --run-training --smoke --output runs/smoke
python train.py --run-training --output runs/lora-v1

The smoke artifact is labeled separately and rejected by the candidate-comparison path. A full run starts from the unchanged base, not the smoke adapter. With only twelve training examples, its purpose is to exercise the adaptation pipeline, not produce a deployable security specialist.

Record peak device allocation, elapsed work, and review effort. Fixed seeds help control variation, but PyTorch explicitly warns that identical seeds do not guarantee reproducibility across releases, platforms, or CPU/GPU execution. PyTorch Documentation

Compare Decisions and Evidence, Not Just Training Loss

Generate the candidate’s development answers and compare them with the saved baseline:

python predict.py --split development \
  --adapter runs/lora-v1/adapter \
  --output runs/candidate-development.jsonl

python score.py --split development \
  --baseline runs/baseline-development.jsonl \
  --candidate runs/candidate-development.jsonl \
  --output runs/development-comparison.json

The comparator rejects mismatched input, prompt, configuration, software, device, CUDA, or generation metadata. Its automated checks cover schema validity, exact decision fields, evidence-reference existence, omitted uncertainty, and completion termination.

Those checks are deliberately narrower than a quality verdict. A citation can exist without supporting the sentence. A nonempty rollback field can contain the wrong procedure. Human review must assess passage support, missing prerequisites, supported alternatives, and operational consequences.

The final Renderer cases preserve these distinctions:

CaseExpected interpretation
Old EE-DEP-001 observation, unresolved EE-DEP-002Current applicability stays unknown even though revision 1 supplies candidate 1.4.3
Same inventory, withdrawn-fix revision 2Applicability stays unknown; supported target becomes null
Failed authenticated assessment without fresh observationsExecution remains failed; no clean disposition is inferred
Separate counterfactual with confirmed affected conditions but no scanner checkApplicability can be affected while assessment execution remains unavailable

These are expected targets, not observed model answers. The missing-check case is a separate authored branch, not a modification to the earlier stale inventory.

After selecting and freezing the candidate through development review, run both variants with --split test and compare their saved outputs. Define rejection conditions before that run. For this lab, a fabricated fix, corrupted assessment state, or unsupported clean disposition is reason to reject the candidate regardless of improved formatting.

No baseline-versus-adapter improvement has been measured for this article. Dataset validation, masking tests, GPU compatibility, training convergence, and security effectiveness all require recorded execution against the completed implementation. A dependency pin or code excerpt is not a passing test result.

Register the Adapter with Its Dependencies and Rollback Path

A PEFT adapter is not a standalone replacement model. Its checkpoint contains adapter parameters and configuration; inference still needs the compatible base model. Preserve that relationship when storing or distributing the result. Hugging Face: PEFT checkpoint format

Begin EE-MODEL-001 as an experiment specification with unmeasured fields set to null. A completed run should write actual artifact hashes, configuration and dataset identities, parameter counts, and training metrics. Keep its status trained but not evaluated until comparison is complete. Training must not promote it or enable tools.

Keep inference artifacts separate from resumable training state. Resuming requires the relevant checkpoint, optimizer/scheduler state, and random-state information, not merely an adapter directory. Use only trusted internal checkpoints and reject a resume when the base, dataset, prompt, configuration, or pinned runtime differs. Hugging Face: Transformers v5.23.0 TrainingArguments implementation

For rollback, restore the previously evaluated base/adapter and compatible prompt/runtime bundle. Do not roll back current advisory corrections or access revocations with it. Removing an adapter cannot justify resurrecting a withdrawn fix.

Security engineering owns target-answer review; the AI platform team owns the training and serving artifacts; the existing change authority still owns operational access. Dataset permission, successful training, model approval, and authorization to act remain separate decisions.

Troubleshoot Before Expanding the Experiment

SymptomInspect firstCorrective action
Context exceeds the limitFull rendered input and targetReduce irrelevant context or revise the documented profile; never silently truncate
Prompt-prefix check failsModel revision and native templateStop and correct tokenization; do not guess a character-to-token boundary
Loss is invalid or no useful learning occursLabel mask, supervised-token count, trainable parametersConfirm nonempty completion targets and intended adapter attachment
GPU runs out of memorySequence length, activation storage, optimizer and runtime allocationMeasure a new bounded profile and rerun both comparison paths
JSON improves but fixes become unsupportedRaw answers and passage-level reviewReject the candidate; inspect examples and overfitting rather than celebrating loss reduction

A small successful run is a reason to continue evaluation, not to add permissions. Expand reviewed data and failure coverage before increasing task scope.

Conclusion

A useful fine-tuning experiment begins with a defect in a defined task and ends with evidence about whether the adaptation corrected it. The deliverable is not merely an adapter file. It is the relationship among the base model, permitted examples, native template, loss mask, comparison inputs, results, and operating decision.

For Example Enterprise, keep the unchanged model as the reference, use synthetic cases to understand the mechanics, and admit real examples only through an explicit review process. Preserve current facts in retrieval and consequential authority outside the model.

The next installment, Test Your Security Model Before Giving It Operational Access, expands the evaluation beyond this small adaptation experiment into retrieval quality, model behavior, scanner coverage, and workflow enforcement.

Promote demonstrated behavior, not the fact that training completed.

External References

Continue reading

Test Your Security Model Before Giving It Operational Access

Evaluate retrieval, model answers, scanner coverage, and workflow enforcement separately before granting security agents operational access. Read the article

2 thoughts on “Fine-Tune Your Own Security Model on Reviewed Findings and Remediations”

Leave a Comment