Splunk Observability NVIDIA GPU Integration

Workflow Overview

┌───────────┐   ┌───────────────┐   ┌───────────────┐   ┌─────────────────┐
│ Preflight │ → │ Render/review │ → │ Apply/handoff │ → │ Validate evidence │
└───────────┘   └───────────────┘   └───────────────┘   └─────────────────┘

When to Activate

• Send NVIDIA GPU, DCGM, DCGM Exporter, GPU Operator, DGX, AI Pod, or CUDA workload telemetry to Splunk Observability Cloud, configure receiver_creator/dcgm-cisco, enable per-pod DCGM labels, or render GPU dashboards and detectors.
• Preview and review the splunk observability nvidia gpu integration workflow before any live apply phase.
• Diagnose failed prerequisites, generated assets, configuration, or validation evidence.

Scope

Follow the documented read-only or render-first path whenever it is available. This skill does not imply permission to mutate live systems. Require explicit apply flags, protected credentials, and operator review for state changes.

Examples

Inspect the supported setup modes before selecting one:

bash skills/splunk-observability-nvidia-gpu-integration/scripts/setup.sh --help

Expected output: usage, supported modes, and required arguments are displayed without changing the target environment.

Inspect validation modes before running completion checks:

bash skills/splunk-observability-nvidia-gpu-integration/scripts/validate.sh --help

Expected output: offline, live, and completion options are displayed when the skill supports them; help exits without mutation.

Troubleshooting

Issue	Cause	Resolution
Preflight fails	A required tool or access path is missing	Resolve it before rendering or applying
Rendered assets are incomplete	Required non-secret inputs are absent	Complete intake and render again
Apply is blocked	Review, credentials, or explicit acceptance is missing	Use the documented handoff
Validation is incomplete	Live evidence is unavailable	Record the gap and keep completion open

This is a standalone reusable skill for NVIDIA GPU telemetry (DCGM Exporter) in Splunk Observability Cloud. It is independent of the AI Pod umbrella — works for NVIDIA DGX clusters, AI Pods, generic K8s + GPUs, anywhere DCGM Exporter is installed.

Critical naming + discovery details

• Receiver name receiver_creator/dcgm-cisco (parameterized via --receiver-creator-name, default dcgm-cisco). Explicitly NOT receiver_creator/nvidia — that name collides with the Splunk OTel chart's autodetect receiver_creator when autodetect.prometheus: true is set, and the collision silently breaks GPU discovery.
• Discovery rule matches both label conventions: app=nvidia-dcgm-exporter (older standalone deployments) AND app.kubernetes.io/name=nvidia-dcgm-exporter (newer GPU Operator deployments).
• Default unfiltered pipeline: metrics/nvidia-metrics ships all DCGM_FI_* series so dashboards have everything. Pass --filter strict to enable the canonical signalfx allow-list when cardinality control is critical.

What it renders

• splunk-otel-overlay/values.overlay.yaml — agent.config.receivers.receiver_creator/dcgm-cisco parent (watch_observers: [k8s_observer]) owning prometheus/dcgm-cisco child (port 9400). Discovery rule with the dual-label match. Plus the metrics/nvidia-metrics pipeline (unfiltered by default).
• dcgm-pod-labels-patch/ — when --enable-dcgm-pod-labels: env-var patch (DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS=true, DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID=true), ClusterRole + ClusterRoleBinding for the DCGM ServiceAccount, AutoMount ServiceAccountToken, kubelet-path volume mount.
• dashboards/<name>.signalflow.yaml — GPU utilization, memory used/free, GPU temp, power, SM/MEM clocks, PCIe TX/RX, total energy, profiling DRAM/GR engine/PIPE tensor activity.
• detectors/<name>.yaml — GPU temp ceiling, GPU power floor, GPU utilization regression, energy consumption anomaly.
• scripts/setup.sh, render_assets.py, validate.sh, handoff-base-collector.sh, handoff-dashboards.sh, handoff-detectors.sh, apply-dcgm-pod-labels-patch.sh.
• metadata.json.

Prerequisites surfaced as preflight checks (not installed)

• NVIDIA GPU Operator or standalone DCGM Exporter: install via helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts then helm install --generate-name gpu-helm-charts/dcgm-exporter. The receiver_creator pattern relies on the standard pod label app=nvidia-dcgm-exporter (or app.kubernetes.io/name=nvidia-dcgm-exporter).

Safety Rules

• O11y token via --o11y-token-file (chmod 600 enforced; passed through to base collector). Reject --o11y-token, --access-token, --token, --bearer-token, --api-token, --sf-token.

Primary Workflow

1. Verify DCGM Exporter is installed and pods carry the standard label:

   kubectl get pods -A -l app=nvidia-dcgm-exporter -o wide
   # OR (newer GPU Operator):
   kubectl get pods -A -l app.kubernetes.io/name=nvidia-dcgm-exporter -o wide
   

2. Render:

   bash skills/splunk-observability-nvidia-gpu-integration/scripts/setup.sh \
     --render --validate \
     --realm us0 \
     --cluster-name lab-cluster \
     --output-dir splunk-observability-nvidia-gpu-rendered
   

3. (Optional) When you need pod/namespace labels in DCGM_FI_* metrics — they are NOT exposed by GPU Operator by default — render the patch:

   bash skills/splunk-observability-nvidia-gpu-integration/scripts/setup.sh \
     --render --enable-dcgm-pod-labels \
     --output-dir splunk-observability-nvidia-gpu-rendered
   

   Then apply directly via the skill (recommended; refuses without --accept-k8s-apply and prints the active kube-context):

   bash skills/splunk-observability-nvidia-gpu-integration/scripts/setup.sh \
     --render --enable-dcgm-pod-labels \
     --apply-pod-labels-patch --accept-k8s-apply
   

   --apply-pod-labels-patch --accept-k8s-apply --dry-run runs kubectl --dry-run=server without mutating the cluster. The DaemonSet env patch (04-daemonset-env-patch.yaml) remains a strategic-merge patch the operator applies separately so GPU Operator reconciles cleanly.

4. Hand off:

   bash splunk-observability-nvidia-gpu-rendered/scripts/handoff-base-collector.sh
   bash splunk-observability-nvidia-gpu-rendered/scripts/handoff-dashboards.sh
   bash splunk-observability-nvidia-gpu-rendered/scripts/handoff-detectors.sh
   

Hand-offs

• Splunk OTel Collector base install: splunk-observability-otel-collector-setup.
• Dashboards: splunk-observability-dashboard-builder.
• Detectors: splunk-observability-native-ops.

Validation

bash skills/splunk-observability-nvidia-gpu-integration/scripts/validate.sh

Static checks: receiver_creator name not equal to receiver_creator/nvidia, dual-label rule present, allow-list opt-in shape. With --live: DCGM Exporter pod presence, optional SignalFlow probe for DCGM_FI_DEV_GPU_UTIL series.

See reference.md and references/dcgm-exporter.md, dcgm-pod-labels.md, gpu-operator-prereq.md, receiver-creator-naming.md, dual-label-discovery.md, dashboards-catalog.md, troubleshooting.md for details.