AI Search with RAG Configuration

Overview

This skill covers configuring and optimizing ServiceNow AI Search with Retrieval Augmented Generation (RAG):

• Creating and configuring search sources (knowledge bases, catalog items, service portal content)
• Setting up search profiles that define search behavior and source priorities
• Configuring result ranking and relevance tuning for quality search results
• Enabling RAG-based answer generation from retrieved knowledge content
• Managing search indexes and field configurations for optimal retrieval
• Tuning search quality through feedback loops and analytics
• Integrating AI Search with Now Assist and Virtual Agent

When to use: When setting up AI-powered search across knowledge bases, configuring RAG for generative answers from enterprise content, or optimizing search result quality in ServiceNow.

Prerequisites

• Roles: admin, search_admin, or now_assist_admin
• Plugins: com.snc.ai_search (AI Search), com.snc.generative_ai_controller (Generative AI Controller), sn_gen_ai (Generative AI)
• Access: sn_ai_search_source, sn_ai_search_profile, sn_gen_ai_config tables
• Knowledge: Understanding of search concepts (indexing, ranking, relevance), knowledge management basics
• Related Skills: genai/now-assist-qa for conversational AI, knowledge/article-generation for KB content

Procedure

Step 1: Assess Available Knowledge Sources

Identify what content should be searchable and available for RAG.

MCP Approach:

Use SN-Query-Table on kb_knowledge_base:
  - query: active=true
  - fields: sys_id,title,description,kb_version,active,article_count
  - limit: 20

Check existing knowledge articles:

Use SN-Query-Table on kb_knowledge:
  - query: workflow_state=published^kb_knowledge_base=<kb_sys_id>
  - fields: sys_id,short_description,kb_knowledge_base,workflow_state,sys_updated_on
  - limit: 20

REST Approach:

GET /api/now/table/kb_knowledge_base
  ?sysparm_query=active=true
  &sysparm_fields=sys_id,title,description,active
  &sysparm_limit=20

Step 2: Create Search Sources

Search sources define where AI Search retrieves content from.

Knowledge Base Search Source:

MCP Approach:

Use SN-Create-Record on sn_ai_search_source:
  - name: "IT Knowledge Base"
  - description: "Primary IT support knowledge articles"
  - source_type: "knowledge"
  - source_table: "kb_knowledge"
  - source_condition: "workflow_state=published^kb_knowledge_base=<kb_sys_id>"
  - active: true
  - enable_rag: true
  - rag_content_field: "text"
  - search_fields: "short_description,text,meta"
  - display_fields: "short_description,kb_category,sys_updated_on"

REST Approach:

POST /api/now/table/sn_ai_search_source
Body: {
  "name": "IT Knowledge Base",
  "description": "Primary IT support knowledge articles",
  "source_type": "knowledge",
  "source_table": "kb_knowledge",
  "source_condition": "workflow_state=published",
  "active": true,
  "enable_rag": true,
  "rag_content_field": "text"
}

Catalog Item Search Source:

Use SN-Create-Record on sn_ai_search_source:
  - name: "Service Catalog Items"
  - description: "Available service catalog offerings"
  - source_type: "catalog"
  - source_table: "sc_cat_item"
  - source_condition: "active=true^hide_sp=false"
  - active: true
  - enable_rag: false
  - search_fields: "name,short_description,description"
  - display_fields: "name,short_description,category"

Incident Solutions Search Source:

Use SN-Create-Record on sn_ai_search_source:
  - name: "Resolved Incidents"
  - description: "Previously resolved incidents for pattern matching"
  - source_type: "table"
  - source_table: "incident"
  - source_condition: "state=6^close_notes!=NULL^universal_requestISNOTEMPTY"
  - active: true
  - enable_rag: true
  - rag_content_field: "close_notes"
  - search_fields: "short_description,description,close_notes"

Step 3: Configure Search Profiles

Search profiles control which sources are searched and how results are ranked.

MCP Approach:

Use SN-Create-Record on sn_ai_search_profile:
  - name: "IT Support Search"
  - description: "Search profile for IT help desk agents and self-service"
  - active: true
  - default_profile: false
  - rag_enabled: true
  - max_results: 10
  - rag_max_sources: 5
  - rag_confidence_threshold: 0.7
  - search_sources: "<kb_source_sys_id>,<catalog_source_sys_id>,<incident_source_sys_id>"

REST Approach:

POST /api/now/table/sn_ai_search_profile
Body: {
  "name": "IT Support Search",
  "description": "Search profile for IT help desk agents and self-service",
  "active": true,
  "rag_enabled": true,
  "max_results": 10,
  "rag_max_sources": 5,
  "rag_confidence_threshold": 0.7
}

Step 4: Configure Search Field Weighting

Field configurations control how different fields contribute to relevance scoring.

MCP Approach:

Use SN-Create-Record on sn_ai_search_field_config:
  - search_source: "<kb_source_sys_id>"
  - field_name: "short_description"
  - boost_factor: 3.0
  - searchable: true
  - displayable: true
  - order: 100

Use SN-Create-Record on sn_ai_search_field_config:
  - search_source: "<kb_source_sys_id>"
  - field_name: "text"
  - boost_factor: 1.0
  - searchable: true
  - displayable: false
  - order: 200

Use SN-Create-Record on sn_ai_search_field_config:
  - search_source: "<kb_source_sys_id>"
  - field_name: "meta"
  - boost_factor: 2.0
  - searchable: true
  - displayable: false
  - order: 300

Boost factor guidelines:

Boost Level	Value	Use Case
High	3.0-5.0	Title, short description -- primary match fields
Medium	1.5-2.5	Tags, metadata, categories -- supporting match fields
Standard	1.0	Body text, full content -- broad matching
Low	0.5	Comments, notes -- supplementary information

Step 5: Configure RAG Answer Generation

Set up the generative AI configuration for producing answers from retrieved content.

MCP Approach:

Use SN-Create-Record on sn_gen_ai_config:
  - name: "AI Search RAG Configuration"
  - description: "Controls how RAG generates answers from search results"
  - active: true
  - llm_provider: "now_llm"
  - model: "default"
  - temperature: 0.3
  - max_tokens: 500
  - system_prompt: "You are a helpful IT support assistant. Answer questions using only the provided context. If the context does not contain enough information, say so clearly. Always cite the source article."
  - context_window: 4000
  - enable_citations: true
  - citation_format: "inline"

REST Approach:

POST /api/now/table/sn_gen_ai_config
Body: {
  "name": "AI Search RAG Configuration",
  "description": "Controls how RAG generates answers from search results",
  "active": true,
  "llm_provider": "now_llm",
  "temperature": 0.3,
  "max_tokens": 500,
  "enable_citations": true
}

Temperature guidelines for RAG:

Temperature	Behavior	Use Case
0.0-0.2	Very factual, deterministic	Policy lookups, compliance answers
0.3-0.5	Balanced, mostly factual	IT support, troubleshooting guidance
0.6-0.8	More creative, varied phrasing	Content suggestions, recommendations
0.9-1.0	Highly creative	Not recommended for RAG

Step 6: Set Up Search Index Configuration

Control how content is indexed for optimal retrieval.

MCP Approach:

Use SN-Create-Record on sn_ai_search_index:
  - search_source: "<kb_source_sys_id>"
  - name: "KB Article Index"
  - index_type: "full_text"
  - active: true
  - rebuild_schedule: "daily"
  - chunk_size: 500
  - chunk_overlap: 50
  - embedding_model: "default"

REST Approach:

POST /api/now/table/sn_ai_search_index
Body: {
  "search_source": "<kb_source_sys_id>",
  "name": "KB Article Index",
  "index_type": "full_text",
  "active": true,
  "rebuild_schedule": "daily",
  "chunk_size": 500,
  "chunk_overlap": 50
}

Chunking strategy guidelines:

Content Type	Chunk Size	Overlap	Rationale
Short KB articles	300-500	30-50	Preserve complete article context
Long documentation	500-800	50-100	Balance context with specificity
FAQ content	200-300	20-30	Keep Q&A pairs together
Policy documents	800-1200	100-150	Maintain section-level context

Step 7: Configure Result Ranking

MCP Approach:

Use SN-Create-Record on sn_ai_search_result_config:
  - search_profile: "<profile_sys_id>"
  - name: "IT Support Ranking"
  - ranking_model: "hybrid"
  - semantic_weight: 0.6
  - keyword_weight: 0.3
  - recency_weight: 0.1
  - personalization: true
  - active: true

Ranking model options:

Model	Description	Best For
keyword	Traditional keyword/BM25 matching	Exact term searches, error codes
semantic	Vector-based semantic similarity	Natural language questions
hybrid	Combined keyword + semantic	General-purpose (recommended)

Step 8: Integrate with Virtual Agent and Now Assist

Connect AI Search to conversational interfaces.

Virtual Agent Search Source:

Use SN-Create-Record on sys_cs_ai_search_source:
  - name: "VA AI Search Integration"
  - search_profile: "<profile_sys_id>"
  - active: true
  - auto_summarize: true
  - fallback_action: "transfer_to_agent"
  - confidence_threshold: 0.65
  - max_results_shown: 3

REST Approach:

POST /api/now/table/sys_cs_ai_search_source
Body: {
  "name": "VA AI Search Integration",
  "search_profile": "<profile_sys_id>",
  "active": true,
  "auto_summarize": true,
  "fallback_action": "transfer_to_agent",
  "confidence_threshold": 0.65
}

Step 9: Monitor Search Quality

Query search analytics to understand performance.

MCP Approach:

Use SN-Query-Table on sn_ai_search_log:
  - query: search_profile=<profile_sys_id>^sys_created_on>javascript:gs.daysAgo(7)
  - fields: sys_id,query_text,result_count,click_through,feedback_score,rag_generated
  - limit: 50
  - orderBy: sys_created_on
  - orderDirection: desc

Track RAG answer quality:

Use SN-Query-Table on sn_ai_search_feedback:
  - query: search_profile=<profile_sys_id>^rating<3
  - fields: sys_id,query_text,answer_text,rating,feedback_comment
  - limit: 20

Step 10: Tune and Optimize

Based on analytics, adjust search configuration.

Adjust confidence threshold:

Use SN-Update-Record on sn_ai_search_profile:
  - sys_id: "<profile_sys_id>"
  - rag_confidence_threshold: 0.75

Update field boosting:

Use SN-Update-Record on sn_ai_search_field_config:
  - sys_id: "<field_config_sys_id>"
  - boost_factor: 4.0

Trigger index rebuild:

Use SN-Update-Record on sn_ai_search_index:
  - sys_id: "<index_sys_id>"
  - rebuild_requested: true

Tool Usage

Tool	Purpose	When to Use
SN-Query-Table	Find existing sources, profiles, analytics	Discovery and monitoring
SN-Create-Record	Create sources, profiles, field configs	Initial setup and expansion
SN-Update-Record	Tune ranking, thresholds, rebuild indexes	Optimization and maintenance
SN-Get-Table-Schema	Discover configuration fields	Understanding available settings

Best Practices

1. Start with knowledge bases as the primary RAG source -- they have structured, curated content
2. Use hybrid ranking combining semantic and keyword search for best results
3. Set conservative confidence thresholds (0.7+) initially and lower only if needed
4. Keep temperature low (0.2-0.4) for RAG to maintain factual accuracy
5. Enable citations so users can verify answers against source material
6. Chunk content appropriately -- too small loses context, too large dilutes relevance
7. Boost title fields 3-5x over body text for better result relevance
8. Monitor click-through rates and feedback scores to identify quality gaps
9. Rebuild indexes regularly after knowledge base updates for fresh content
10. Test with real user queries from search logs to validate ranking changes

Troubleshooting

Issue	Cause	Resolution
No search results returned	Search source inactive or no matching content	Verify source is active and content matches source_condition
RAG answer is hallucinated	Temperature too high or insufficient context	Lower temperature, increase rag_max_sources, verify content quality
Irrelevant results ranked high	Field boost weights misconfigured	Increase boost on title/description, decrease on body text
Search is slow	Large index or too many sources searched	Limit source_condition scope, optimize chunk_size, reduce max_results
RAG answer missing citations	Citations not enabled in gen AI config	Set enable_citations=true in sn_gen_ai_config
Index out of date	Rebuild schedule too infrequent	Trigger manual rebuild or increase rebuild_schedule frequency
Duplicate results	Same content indexed from multiple sources	Add source deduplication or narrow source_condition filters

Examples

Example 1: IT Self-Service AI Search

Configure AI Search for employee self-service portal:

• Sources: IT Knowledge Base (published articles), Service Catalog (active items), FAQ knowledge base
• Profile: Self-service with RAG enabled, max 5 results, confidence threshold 0.7
• Ranking: Hybrid model, semantic_weight=0.6, keyword_weight=0.3, recency=0.1
• RAG Config: Temperature 0.3, max 500 tokens, inline citations enabled
• Integration: Virtual Agent with auto-summarize, fallback to live agent

Example 2: HR Policy Search with RAG

Configure AI Search for HR policy questions:

• Sources: HR Knowledge Base (policy documents), HR Catalog (request forms)
• Profile: HR-specific with strict RAG (temperature 0.1 for policy accuracy)
• Chunking: 800 tokens with 100 overlap (preserves policy section context)
• Field Boost: Policy title 5.0x, section headers 3.0x, body 1.0x
• Guardrails: System prompt requires exact policy citations, prohibits paraphrasing legal language

Example 3: Multi-Source Technical Knowledge Search

Configure AI Search across multiple technical knowledge bases:

• Sources: Infrastructure KB, Application KB, Security KB, resolved incidents
• Profile: Technical support with high semantic weight (0.7) for natural language queries
• Chunking: 500 tokens with 50 overlap for technical articles
• Ranking: Boost recent articles (recency_weight=0.2) for evolving tech content
• Monitoring: Track low-rated answers weekly, retrain on misses

Related Skills

• genai/now-assist-qa - Conversational AI that consumes AI Search results
• genai/skill-kit-custom - Custom skills that can invoke AI Search
• knowledge/article-generation - Managing the knowledge content that feeds search
• reporting/executive-dashboard - Building search analytics dashboards