Cisco ThousandEyes App Setup Automation

Prerequisites

Tool or access	Purpose	Verify
Bash, curl, and jq	Run setup and REST configuration helpers	command -v bash curl jq
Splunk administrative access	Create indexes, HEC, accounts, and inputs	Confirm search-tier REST access
ThousandEyes account-group access	Authorize OAuth and selected test data	Record the numeric account-group ID

Workflow Overview

┌───────────┐   ┌────────────────┐   ┌──────────────────┐   ┌────────────────────┐
│ Preflight │ → │ Configure auth │ → │ Enable data path │ → │ Validate dashboards │
└───────────┘   └────────────────┘   └──────────────────┘   └────────────────────┘

When to Activate

• Configure ThousandEyes OAuth, HEC streaming, or API polling in Splunk.
• Onboard cloud, enterprise-agent, or endpoint test results.
• Diagnose missing path data, empty dashboards, or optional ITSI readiness.

Scope

This skill configures the Splunk application and its data paths. It does not create ThousandEyes tests, expose OAuth or HEC tokens in chat, or enable both streaming and polling for the same data without duplicate-ingest review.

Examples

Run preflight diagnostics before authentication:

bash skills/cisco-thousandeyes-setup/scripts/validate.sh

Expected output: package, local tools, Splunk access, and account prerequisites are reported without creating tokens or inputs.

Run strict completion checks after enabling the selected path:

bash skills/cisco-thousandeyes-setup/scripts/validate.sh --completion

Expected output: account, HEC or polling input, index, source type, recent data, and dashboard evidence report [PASS] or exit nonzero.

Troubleshooting

Issue	Cause	Resolution
Device flow expires	Approval timed out	Restart and approve promptly
No HEC events	Token, endpoint, or index is wrong	Verify without printing token
Duplicate test events	Streaming and polling overlap	Keep one path per dataset
Empty dashboards/ITSI	Index, macro, or type is wrong	Validate events first

TA Completion Gate

For every TA/add-on or dashboard companion run, satisfy the shared TA completion gate: configure and enable the data ingest path owned by this skill or its required companion, validate events or metrics in the target indexes/source types, and verify any pre-built/package-shipped dashboards are visible, macro-aligned, and returning data. If the package ships no dashboards, record that evidence explicitly and hand off dashboard use to the consuming app, ES/ITSI/ARI content, or readiness doctor.

Automates the Cisco ThousandEyes App for Splunk (ta_cisco_thousandeyes).

How This App Differs From Other Cisco TAs

This app is architecturally different from the Catalyst, Meraki, Intersight, and DC Networking TAs in several ways:

• OAuth 2.0 Device Code Flow — no API keys or passwords. The user must visit a URL in their browser to authorize.
• HEC-based data delivery — ThousandEyes pushes most data TO Splunk via HEC (HTTP Event Collector) streams, rather than Splunk polling an API.
• Hybrid collection — 3 delivery mechanisms: streaming (HEC push), API polling, and webhooks.
• Full app with dashboards — not just a TA. Includes network, application, voice, alerts, traces, and configuration status dashboards.
• ITSI integration — optional bidirectional integration with Splunk IT Service Intelligence.

Package Model

Pull from Splunkbase first (latest version), fall back to splunk-ta/. Use splunk-app-install with --source splunkbase --app-id 7719 to get the latest release. If Splunkbase is unavailable, fall back to the local package in splunk-ta/. This applies to both Splunk Cloud (ACS) and Splunk Enterprise.

After installation, use this skill to authenticate via OAuth, configure HEC, create indexes, and enable inputs. Any splunk-ta/_unpacked/ tree is review-only.

Agent Behavior — Credentials

The agent must NEVER ask for passwords, API keys, or secrets in chat.

Splunk credentials are read automatically from the project-root credentials file (falls back to ~/.splunk/credentials). If neither exists, guide the user to create it:

bash skills/shared/scripts/setup_credentials.sh

ThousandEyes OAuth Authentication

Unlike other TAs, this app uses an interactive OAuth 2.0 device code flow. No password file or API key file is needed. Instead:

1. The configure_account.sh script initiates the OAuth flow and displays a verification URL and user code.
2. The agent instructs the user: "Visit the URL shown in your terminal and enter the code to authorize."
3. The script polls until the user completes authorization in their browser.
4. Tokens are stored and encrypted automatically by Splunk.

The agent never handles or sees the OAuth tokens.

For prerequisite collection, use skills/cisco-thousandeyes-setup/template.example as the intake worksheet. Copy it to template.local, capture the non-secret account details there, and keep the completed file local only.

Environment

Item	Value
Search-tier API	SPLUNK_SEARCH_API_URI env var (legacy alias: SPLUNK_URI)
Cloud stack	SPLUNK_CLOUD_STACK for Cloud installs
App name	ta_cisco_thousandeyes
Credentials	Project-root credentials file (falls back to ~/.splunk/credentials)
HEC requirement	Required for metrics, traces, activity logs, and alerts inputs
Skill scripts	skills/cisco-thousandeyes-setup/scripts/ (relative to repo root)

Remote Splunk Connection

export SPLUNK_SEARCH_API_URI="https://splunk-host:8089"

Splunk Authentication

Scripts read Splunk credentials from the project-root credentials file (falls back to ~/.splunk/credentials) automatically:

bash skills/cisco-thousandeyes-setup/scripts/validate.sh

If credentials are not yet configured:

bash skills/shared/scripts/setup_credentials.sh

Setup Workflow

Step 1: Install the App

bash skills/splunk-app-install/scripts/install_app.sh \
  --source splunkbase --app-id 7719

Step 2: Create HEC Token and Indexes

bash skills/cisco-thousandeyes-setup/scripts/setup.sh

The default flow verifies or creates a HEC token named thousandeyes, then creates six indexes. When run interactively (TTY), the script prompts to continue with OAuth authentication and input enablement after the initial setup completes.

For Splunk Cloud, HEC tokens are managed through ACS and indexes are created through ACS. For Enterprise, both use the REST API.

Partial runs: --hec-only, --indexes-only.

Creates six indexes:

Index	Purpose	Sourcetype
thousandeyes_metrics	Test metrics data	cisco:thousandeyes:metric
thousandeyes_traces	Test traces data	cisco:thousandeyes:trace
thousandeyes_events	ThousandEyes events	cisco:thousandeyes:event
thousandeyes_activity	Activity/audit logs	cisco:thousandeyes:activity
thousandeyes_alerts	Alert notifications	cisco:thousandeyes:alerts
thousandeyes_pathvis	Path visualization	cisco:thousandeyes:path-vis

The metrics and event collectors also write account-group metadata as cisco:thousandeyes:account-group into their configured metrics or events index. The token-refresh input writes operational logs to _internal as cisco:thousandeyes:refresh-tokens; neither sourcetype needs a separate customer index.

When you enable metrics inputs, the setup script now enables related path collection into thousandeyes_pathvis by default. Although it is configured on the metrics stream input, path visualization data is collected via API polling on its own --pathvis-interval (default 3600s) — hence the "API polling" delivery shown for cisco:thousandeyes:path-vis in the sourcetype table below. Use --no-pathvis if you need a metrics-only stream, or override the defaults with --pathvis-index / --pathvis-interval.

Step 3: Authenticate via OAuth

bash skills/cisco-thousandeyes-setup/scripts/configure_account.sh

The script will:

1. Display a verification URL and user code
2. Wait for the user to authorize in their browser
3. Store the OAuth tokens in Splunk's encrypted credential store

The agent should instruct the user to visit the URL shown in the terminal output and enter the displayed code.

Step 4: Enable Inputs

Before running, the agent must ask the user for:

• ThousandEyes account name (the email shown after OAuth)
• ThousandEyes account group (ask user or use script to list available groups)
• Which input types to enable
• HEC token name (default: thousandeyes)
• Alert rule IDs when enabling alerts or all

bash skills/cisco-thousandeyes-setup/scripts/setup.sh --enable-inputs \
  --account "user@example.com" \
  --account-group "My Account Group" \
  --hec-token "thousandeyes" \
  --alert-rules "RULE_ID_1~RULE_ID_2" \
  --input-type all

Input Type	Delivery	Description
metrics	HEC stream	Test metrics via ThousandEyes Streaming API
traces	HEC stream	Test traces via ThousandEyes Streaming API
events	API polling	ThousandEyes events (default interval: 3600s)
activity	HEC stream	Activity/audit logs via Streaming API
alerts	HEC webhook	Alert notifications via webhook
all	Mixed	All of the above

The alerts and all groups require --alert-rules with ThousandEyes alert rule IDs separated by ~, matching the add-on's REST handler format.

Step 5: ITSI Integration (Optional)

If Splunk ITSI (SA-ITOA) is installed, the app can forward Splunk notable events to ThousandEyes and receive alert data. The validate script checks for ITSI presence automatically.

Step 6: Restart If Required

On Splunk Enterprise, restart Splunk after new index creation. On Splunk Cloud, check acs status current-stack and only run acs restart current-stack when ACS reports restartRequired=true.

Step 7: Validate

bash skills/cisco-thousandeyes-setup/scripts/validate.sh --completion

Checks: app installation, HEC token, indexes, OAuth account, token refresh input, data inputs, data flow, settings, and optional ITSI status.

Sourcetypes

Sourcetype	Delivery	Content
cisco:thousandeyes:metric	HEC stream	Test metrics (OpenTelemetry v2)
cisco:thousandeyes:trace	HEC stream	Test traces (OpenTelemetry v2)
cisco:thousandeyes:path-vis	API polling	Path visualization data
cisco:thousandeyes:event	API polling	ThousandEyes events
cisco:thousandeyes:activity	HEC stream	Activity/audit logs
cisco:thousandeyes:alerts	HEC webhook	Alert notifications
cisco:thousandeyes:account-group	API metadata	Account-group metadata written by metrics and event collectors
cisco:thousandeyes:refresh-tokens	Internal monitor	Token-refresh operational logs in _internal

Dashboards

The app ships dashboards in the package. They appear in Splunk Web automatically after installation.

To access them: Apps → Cisco ThousandEyes Add-On for Splunk

Built-in dashboards include: Network, Application, Voice, Alerts, Traces, and Configuration Status views.

Prerequisites for dashboards to show data:

1. The applicable thousandeyes_* indexes must exist and inputs must be enabled (Steps 2–4).
2. HEC must be reachable from ThousandEyes for streaming inputs (metrics, traces, activity, alerts). Validate the HEC target URL format — see Known Issue #6 below.
3. At least one polling input (events) or streaming input must have data flowing before dashboard panels populate.

On Splunk Cloud, the HEC URL must use the cloud format (https://http-inputs-{stack}.splunkcloud.com:443). The setup script auto-detects this when SPLUNK_CLOUD_STACK is set. If streaming inputs were created with the wrong HEC URL, re-run setup.sh --enable-inputs to correct them.

MCP Server Integration

bash skills/cisco-thousandeyes-setup/scripts/load_mcp_tools.sh

Key Learnings / Known Issues

1. OAuth device code flow: This is the only authentication method. There is no API key or username/password option. The user must complete authorization in a browser.
2. HEC is required: Metrics, traces, activity logs, and alerts all require a working HEC endpoint. Verify HEC is enabled and the token is valid before creating streaming inputs.
3. Token refresh: The app includes a thousandeyes_refresh_tokens modular input that runs weekly to regenerate OAuth tokens. Ensure this input is enabled.
4. Streaming API creates ThousandEyes-side resources: When you create a metrics or traces input, the app creates a "stream" object in ThousandEyes that pushes data to Splunk. Deleting the input should clean up the stream.
5. Alerts use webhooks: The alerts input creates webhook operations and connectors in ThousandEyes that push data to Splunk's HEC endpoint.
6. Cloud vs Enterprise HEC URL format: The HEC target URL differs by platform. Splunk Cloud uses https://http-inputs-{stack}.splunkcloud.com:443 (or .stg.splunkcloud.com for staging). Enterprise uses https://{host}:8088. The detect_hec_target() function in setup.sh auto-detects the correct format, but falls back to the Enterprise pattern if SPLUNK_CLOUD_STACK is empty. If streaming inputs have a search-head hostname or port 8088 on a Cloud stack, ThousandEyes cannot reach the HEC endpoint and no data flows. The validate script checks for this mismatch. Fix by re-running setup.sh --enable-inputs.
7. Client ID is hardcoded: The OAuth client ID (0oalgciz1dyS1Uonr697) is built into the app; you do not need to provide it.
8. ITSI is optional: The ITSI integration only activates if SA-ITOA is installed. The app works fully without it.
9. Existing account conflict: If configure_account.sh fails with a 409/500 ("Name already in use"), the account already exists from a prior session. The existing OAuth tokens are retained and typically still valid. Run validate.sh to check the account status before re-authenticating.
10. Interactive continuation: When run from a TTY, setup.sh prompts to continue with OAuth authentication and input enablement after HEC/index setup completes. This is skipped in non-interactive (piped) contexts.

Additional Resources

• reference.md — Complete input catalog, account fields, API endpoints, HEC configuration
• mcp_tools.json — MCP tool definitions

Validation Modes

Run scripts/validate.sh for diagnostics. Use --completion (alias --strict) to require indexes, an OAuth account, token refresh, an enabled data input, nonzero app data, expected account-group metadata when metrics or event data is present, and visible shipped dashboards. A confirmed HEC token is required when streaming inputs are present.