QA Security Testing

Automated security testing pipelines that integrate scanners into CI/CD, enforce vulnerability gates, and drive findings through remediation. This skill covers the testing automation side of security; for secure design, threat modeling, and architecture review, use software-security-appsec.

Start with the quick start workflow below, then dive into specific reference guides. Use current official sources from data/sources.json for tool documentation.

Quick Start

1. Run a lightweight threat model to identify attack surface and risk areas.
2. Select tools per category (SAST, SCA, DAST, secrets, container/IaC).
3. Integrate into CI with clear gate policies per stage.
4. Establish a triage workflow: confirm, classify, assign, track.
5. Define vulnerability SLAs as a starting policy, then tune them to exploitability, business impact, and compliance obligations.
6. Add security regression tests for every confirmed vulnerability.

Inputs to Gather

• Application type: web app, API, mobile, CLI, infrastructure.
• Languages, frameworks, and build toolchain.
• Deployment model: containers, serverless, VMs, PaaS.
• Current security tooling and CI platform.
• Compliance requirements: SOC 2, PCI DSS, HIPAA, ISO 27001 if applicable.
• Existing vulnerability management process and acceptable risk thresholds.
• Code hosting platform: GitHub, GitLab, Bitbucket (affects native tool availability).

Security Testing Categories

1. SAST (Static Application Security Testing)

Analyze source code for vulnerabilities without executing it.

• Recommended tools: Semgrep (fast, customizable rules, free tier), CodeQL (deep dataflow analysis, GitHub-native), Snyk Code.
• CI pattern: run on every PR; a common starter is block merge on high/critical findings, then tune to your risk policy.
• Key practices: maintain custom rules for your codebase patterns, manage suppressions with documented reasons, use baseline files to avoid noise from pre-existing findings.
• Reference: references/sast-integration.md

2. SCA / Dependency Scanning

Detect known vulnerabilities in direct and transitive dependencies.

• Recommended tools: Dependabot (GitHub-native, zero config), Snyk Open Source, Renovate + npm audit / pip-audit, Trivy fs mode.
• Vulnerability SLAs (common starting point): critical 24h, high 7d, medium 30d, low 90d. Adjust these to exploitability, asset value, compensating controls, and regulatory commitments.
• Key practices: auto-merge patch updates with passing tests, generate SBOMs (CycloneDX or SPDX), track transitive dependency risk.
• Reference: references/dependency-scanning.md

3. DAST (Dynamic Application Security Testing)

Test the running application for vulnerabilities by sending crafted requests.

• Recommended tools: ZAP by Checkmarx (open source, automation framework — left OWASP in 2023, same tool), Nuclei (template-based, fast), Burp Suite (manual + CI plugin).
• CI pattern: run on staging deploy, not on every PR (too slow). Schedule full scans weekly.
• Key practices: configure authenticated scanning, maintain baselines for known findings, scan APIs with OpenAPI specs.
• Reference: references/dast-automation.md

4. Secret Scanning

Detect credentials, tokens, and keys committed to source code.

• Recommended tools: gitleaks (pre-commit + CI), TruffleHog (entropy + regex), GitHub secret scanning (push protection).
• CI pattern: hard fail on any active credential or secret material. False positives and documented test fixtures still need an explicit suppression workflow.
• Key practices: install pre-commit hooks to catch secrets before push, scan full git history for historical leaks, rotate exposed secrets immediately (removal from code is not sufficient).
• Reference: references/secret-scanning.md

5. Container and IaC Scanning

Scan container images and infrastructure-as-code for misconfigurations and CVEs.

• Recommended tools: Trivy (containers + IaC + SBOM, single tool), Checkov (Terraform, CloudFormation, Kubernetes). tfsec is deprecated — all checks merged into trivy config.
• CI pattern: scan on image build; a common starter is block on critical CVEs. Scan IaC on every PR with thresholds matched to environment risk.
• Key practices: enforce base image policy (approved images only), scan registry images on schedule, use multi-stage builds to reduce attack surface.
• Reference: references/container-iac-scanning.md

6. Security Regression Testing

Write test cases that prevent reintroduction of fixed vulnerabilities.

• Key areas: auth boundary tests (IDOR, privilege escalation), input validation suites, CORS/CSP/security header verification, business logic abuse cases.
• CI pattern: include in standard test suites, run on every PR like functional tests.
• Reference: references/security-regression-testing.md

CI Gate Design

Treat gate thresholds as organization policy, not universal defaults. Severity alone is not enough; combine scanner severity with exploitability, reachability, asset sensitivity, and business impact.

Stage	Tools	Gate Policy
Pre-merge (every PR)	SAST + secret scanning + dependency audit	Common starter: block on high/critical SAST, active secrets, and critical/high exploitable CVEs
Pre-deploy (staging)	DAST on staging + container scan	Common starter: block on high/critical confirmed DAST findings and critical container CVEs
Scheduled (weekly)	Full DAST scan, dependency review, registry scan	Findings feed into triage backlog
Release	All gates green + SLA compliance check	Block release when open findings violate the org's release policy or SLA commitments

Vulnerability Management Workflow

1. Triage: confirm finding is real, classify severity and exploitability. Severity (CVSS) alone is not triage — combine it with EPSS (probability of real-world exploitation) and reachability (can your build actually hit the vulnerable path) before setting priority. See references/owasp-top-10-coverage.md § Triage: Severity vs. Exploitability vs. Reachability for the full model and worked rules of thumb.
2. Track: record in issue tracker with severity, EPSS/reachability context, SLA deadline, and remediation plan.
3. Remediate: fix, verify fix with regression test, close finding.
4. Suppress: if false positive, document reason and reviewer. Review suppressions quarterly — see references/owasp-top-10-coverage.md § False-Positive Economics for why unmanaged false positives cost more than the noise itself.
5. Measure: track mean time to remediate, open vulnerability count by severity, scan coverage percentage.
6. Escalate to human testing when scanners cannot see the risk: business-logic abuse, exploit chains across multiple low-severity findings, and freshly re-architected surfaces need a pen-test or red-team engagement, not another scanner run — see references/owasp-top-10-coverage.md § When Pen-Testing Beats Scanning.

Quick Reference

Category	Recommended Tool	CI Stage	Gate Policy
SAST	Semgrep	Every PR	Common starter: block high/critical
SCA	Dependabot + Trivy	Every PR	Common starter: block critical/high exploitable CVE
DAST	ZAP / Nuclei	Staging deploy	Common starter: block high/critical confirmed findings
Secrets	gitleaks	Every PR + pre-commit	Fail on active secret material
Containers	Trivy	Image build	Common starter: block critical CVE
IaC	Checkov / Trivy	Every PR	Common starter: block high/critical misconfigurations
Regression	Custom test suites	Every PR	Standard test pass/fail

Decision Tree

Starting security testing pipeline:
    │
    ├─ New project, no security tooling?
    │   └─ Start with: SAST (Semgrep) + secret scanning (gitleaks) + SCA (Dependabot)
    │       └─ These three give the best signal-to-effort ratio
    │
    ├─ Have SAST/SCA, need runtime testing?
    │   └─ Add DAST: ZAP on staging + Nuclei for targeted templates
    │
    ├─ Running containers?
    │   └─ Add Trivy for image scanning + base image policy
    │
    ├─ Using Terraform/CloudFormation/Kubernetes?
    │   └─ Add Checkov or Trivy IaC scanning on every PR
    │
    ├─ Compliance requirement (SOC 2, PCI, HIPAA)?
    │   └─ Full pipeline + SBOM generation + evidence retention + SLA tracking
    │
    └─ Past security incidents?
        └─ Write regression tests for each, add to standard test suite

Do / Avoid

Do:

• Start with SAST + secrets + SCA — cheapest signal, highest coverage.
• Triage findings before enabling CI blocking to avoid developer frustration.
• Tune block thresholds and SLAs to the business risk model instead of copying canned defaults blindly.
• Maintain suppressions with documented reasons and periodic review.
• Rotate any detected secret immediately; removing from code is not enough.
• Write regression tests for every confirmed vulnerability.
• Test authentication boundaries explicitly (IDOR, privilege escalation, tenant isolation).

Avoid:

• Blocking CI on every finding without initial triage and baseline.
• Running full DAST scans on every PR (too slow, use staging deploys).
• Treating scanner output as ground truth without human verification.
• Scanning without a remediation workflow (findings without owners rot).
• Ignoring transitive dependency vulnerabilities.
• Storing suppression rules without documented justification.

Scripts

Script	Purpose
scripts/vuln_tracker.py	Vulnerability tracker and security posture scorer

Run from the qa-security-testing/ directory:

# Overall security posture: counts by severity, SLA rate, overdue items, score
python scripts/vuln_tracker.py status --input data/sample-vulnerabilities.json

# SLA compliance check: list overdue items with days overdue
python scripts/vuln_tracker.py sla --input data/sample-vulnerabilities.json

# Scanner coverage across attack surfaces: flag gaps
python scripts/vuln_tracker.py coverage --input data/sample-scan-coverage.json

# Full Markdown security testing report
python scripts/vuln_tracker.py report \
  --input data/sample-vulnerabilities.json \
  --coverage data/sample-scan-coverage.json \
  --output report.md

Resources

Resource	Purpose
references/sast-integration.md	Semgrep and CodeQL setup, custom rules, CI integration
references/dast-automation.md	ZAP (by Checkmarx) and Nuclei automation, authenticated scanning
references/dependency-scanning.md	SCA tools, vulnerability SLAs, SBOM generation
references/secret-scanning.md	gitleaks setup, pre-commit hooks, remediation workflow
references/container-iac-scanning.md	Trivy and Checkov for containers and IaC
references/security-regression-testing.md	Writing security test cases for past vulnerabilities
references/supply-chain-security.md	SLSA v1.1 build levels, Sigstore/Cosign keyless signing, in-toto attestations, SBOM signing, GitHub provenance, npm/PyPI sigstore
references/owasp-top-10-coverage.md	CI scanner mapping for OWASP API Top 10 (2023), OWASP LLM Top 10 v2 (2025), OWASP Top 10 for Agentic Applications (2026, ASI01-10), and OWASP Top 10:2025; notes on ASVS 5.0; plus the severity/exploitability/reachability triage model, pen-test-vs-scanning judgment, and false-positive economics
data/sources.json	Curated external sources and documentation links
data/sample-vulnerabilities.json	Sample B2B SaaS vulnerability list for vuln_tracker.py
data/sample-scan-coverage.json	Sample scanner coverage map for vuln_tracker.py

Templates

Template	Purpose
assets/template-security-test-plan.md	Security testing scope, tool selection, and gate policy
assets/template-security-gate-checklist.md	Pre-merge, pre-deploy, and release gate checklist
assets/template-vulnerability-sla.md	Severity-based SLA table with escalation paths

ASCII Flow

Security testing request
  -> Identify attack surface, asset value, compliance needs, and release stage
  -> Select SAST, SCA, secrets, DAST, container, and IaC checks by risk
  -> Wire CI gates with severity, exploitability, and suppression policy
  -> Triage findings: confirm, classify, assign, and set SLA
  -> Add regression tests for confirmed vulnerabilities
  -> Publish evidence for merge, deploy, and release decisions

Navigation

• ## Vulnerability Management Workflow, ## Decision Tree, and ## Do / Avoid for the main sequence
• ## Scripts, ## Resources, and ## Templates for deeper materials and automation
• ## Related Skills for AppSec, CI, and testing-strategy handoffs
• Game theory (scan scheduling, fuzz seed selection, red-team iteration, BAS): references/game-theory-applied.md

Related Skills

Skill	Purpose
software-security-appsec	Secure design, threat modeling, and security review
qa-testing-strategy	Risk-based test strategy
qa-api-testing-contracts	API contract and security testing
ops-devops-platform	CI/CD pipeline design
dev-dependency-management	Dependency management and update policy
qa-resilience	Failure mode testing

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.