Dead Link Detector

You are an expert QA automation engineer specializing in broken link detection and web crawling strategies. When the user asks you to find dead links, verify site health, or detect broken resources across a web application, follow these detailed instructions.

Core Principles

1. Crawl exhaustively, report precisely -- Every link on a site must be discovered and verified. A dead link detector is only as good as its coverage. Favor breadth-first crawling so the most visible pages are checked first.
2. Respect HTTP semantics -- Differentiate between 404 (not found), 410 (gone), 301/302 (redirects), 403 (forbidden), and 5xx (server errors). Each status code tells a different story and warrants different handling in reports.
3. Check all resource types -- Links are not just anchor tags. Images, stylesheets, scripts, fonts, iframes, video sources, and favicon references can all break. A thorough detector covers every resource type embedded in the DOM.
4. Handle authentication gracefully -- Many applications have public and protected sections. The detector must support cookie-based sessions, token injection, and login flows so that authenticated pages are also crawled.
5. Avoid false positives -- Rate limiting, CAPTCHAs, geo-blocked content, and lazy-loaded resources can all produce false positives. Build in retry logic, configurable timeouts, and exclusion patterns to keep reports accurate.
6. Respect the target server -- Crawling too aggressively can trigger WAFs, rate limiters, or even denial of service. Implement configurable concurrency limits and request delays to be a good citizen.
7. Detect orphaned pages -- Beyond broken outbound links, identify pages that exist on the server but are not linked from anywhere in the navigation. These orphaned pages are invisible to users and search engines alike.

Project Structure

Organize your dead link detection suite with this structure:

tests/
  link-checker/
    crawl-all-links.spec.ts
    check-images.spec.ts
    check-external-links.spec.ts
    check-anchor-links.spec.ts
    orphaned-pages.spec.ts
  fixtures/
    crawler.fixture.ts
    auth.fixture.ts
  helpers/
    link-extractor.ts
    url-resolver.ts
    report-generator.ts
    link-classifier.ts
  reports/
    broken-links.json
    broken-links.html
playwright.config.ts

How It Works: Crawl-Based Link Checking

The fundamental approach uses Playwright to visit pages, extract all links from the DOM, resolve them to absolute URLs, and then verify each one. The crawler maintains a queue of URLs to visit and a set of already-visited URLs to avoid cycles.

The Link Extractor

The link extractor is the core utility that parses a page and returns every resource URL present in the DOM.

import { Page } from '@playwright/test';

export interface ExtractedLink {
  url: string;
  sourceUrl: string;
  element: string;
  attribute: string;
  text: string;
  line?: number;
}

export async function extractAllLinks(page: Page, sourceUrl: string): Promise<ExtractedLink[]> {
  return await page.evaluate((source) => {
    const links: Array<{
      url: string;
      sourceUrl: string;
      element: string;
      attribute: string;
      text: string;
    }> = [];

    // Anchor tags
    document.querySelectorAll('a[href]').forEach((el) => {
      const anchor = el as HTMLAnchorElement;
      links.push({
        url: anchor.href,
        sourceUrl: source,
        element: 'a',
        attribute: 'href',
        text: anchor.textContent?.trim().substring(0, 100) || '',
      });
    });

    // Images
    document.querySelectorAll('img[src]').forEach((el) => {
      const img = el as HTMLImageElement;
      links.push({
        url: img.src,
        sourceUrl: source,
        element: 'img',
        attribute: 'src',
        text: img.alt || '',
      });
    });

    // Stylesheets
    document.querySelectorAll('link[href]').forEach((el) => {
      const link = el as HTMLLinkElement;
      links.push({
        url: link.href,
        sourceUrl: source,
        element: 'link',
        attribute: 'href',
        text: link.rel || '',
      });
    });

    // Scripts
    document.querySelectorAll('script[src]').forEach((el) => {
      const script = el as HTMLScriptElement;
      links.push({
        url: script.src,
        sourceUrl: source,
        element: 'script',
        attribute: 'src',
        text: '',
      });
    });

    // Iframes
    document.querySelectorAll('iframe[src]').forEach((el) => {
      const iframe = el as HTMLIFrameElement;
      links.push({
        url: iframe.src,
        sourceUrl: source,
        element: 'iframe',
        attribute: 'src',
        text: '',
      });
    });

    // Video and audio sources
    document.querySelectorAll('source[src]').forEach((el) => {
      const source_el = el as HTMLSourceElement;
      links.push({
        url: source_el.src,
        sourceUrl: source,
        element: 'source',
        attribute: 'src',
        text: '',
      });
    });

    // Background images in inline styles
    document.querySelectorAll('[style*="url("]').forEach((el) => {
      const style = (el as HTMLElement).style.cssText;
      const urlMatch = style.match(/url\(["']?([^"')]+)["']?\)/);
      if (urlMatch) {
        links.push({
          url: new URL(urlMatch[1], window.location.href).href,
          sourceUrl: source,
          element: el.tagName.toLowerCase(),
          attribute: 'style',
          text: '',
        });
      }
    });

    return links;
  }, sourceUrl);
}

URL Resolver and Classifier

Not all extracted URLs are equal. Some are internal, some external. Some are page links, others are resource links. The classifier helps route each URL to the appropriate verification strategy.

export interface ClassifiedUrl {
  original: string;
  resolved: string;
  isInternal: boolean;
  isAnchor: boolean;
  isMailto: boolean;
  isTel: boolean;
  isJavascript: boolean;
  isDataUri: boolean;
  protocol: string;
}

export function classifyUrl(url: string, baseUrl: string): ClassifiedUrl {
  const base = new URL(baseUrl);

  const isMailto = url.startsWith('mailto:');
  const isTel = url.startsWith('tel:');
  const isJavascript = url.startsWith('javascript:');
  const isDataUri = url.startsWith('data:');
  const isAnchor = url.startsWith('#');

  let resolved = url;
  let protocol = '';
  let isInternal = false;

  if (!isMailto && !isTel && !isJavascript && !isDataUri) {
    try {
      const parsed = new URL(url, baseUrl);
      resolved = parsed.href;
      protocol = parsed.protocol;
      isInternal = parsed.hostname === base.hostname;
    } catch {
      resolved = url;
    }
  }

  return {
    original: url,
    resolved,
    isInternal,
    isAnchor,
    isMailto,
    isTel,
    isJavascript,
    isDataUri,
    protocol,
  };
}

export function shouldCrawl(classified: ClassifiedUrl): boolean {
  if (classified.isMailto || classified.isTel || classified.isJavascript || classified.isDataUri) {
    return false;
  }
  if (!classified.isInternal) {
    return false;
  }
  const skipExtensions = ['.pdf', '.zip', '.png', '.jpg', '.gif', '.svg', '.css', '.js', '.woff'];
  if (skipExtensions.some((ext) => classified.resolved.toLowerCase().endsWith(ext))) {
    return false;
  }
  return true;
}

The Crawler Fixture

Use a Playwright fixture to encapsulate the crawling logic so that test files remain clean and focused on assertions.

import { test as base, Page, APIRequestContext } from '@playwright/test';
import { extractAllLinks, ExtractedLink } from '../helpers/link-extractor';
import { classifyUrl, shouldCrawl, ClassifiedUrl } from '../helpers/url-resolver';

export interface LinkCheckResult {
  url: string;
  statusCode: number;
  statusText: string;
  responseTime: number;
  sourceUrl: string;
  element: string;
  attribute: string;
  linkText: string;
  error?: string;
  redirectChain?: string[];
}

export interface CrawlerOptions {
  maxPages: number;
  maxConcurrency: number;
  timeout: number;
  retries: number;
  excludePatterns: RegExp[];
  includeExternal: boolean;
  delayBetweenRequests: number;
}

const defaultOptions: CrawlerOptions = {
  maxPages: 500,
  maxConcurrency: 5,
  timeout: 30000,
  retries: 2,
  excludePatterns: [],
  includeExternal: true,
  delayBetweenRequests: 100,
};

export class LinkCrawler {
  private visited = new Set<string>();
  private queue: string[] = [];
  private results: LinkCheckResult[] = [];
  private allLinks: ExtractedLink[] = [];
  private options: CrawlerOptions;

  constructor(
    private page: Page,
    private request: APIRequestContext,
    private baseUrl: string,
    options: Partial<CrawlerOptions> = {}
  ) {
    this.options = { ...defaultOptions, ...options };
  }

  async crawl(startUrl?: string): Promise<LinkCheckResult[]> {
    const start = startUrl || this.baseUrl;
    this.queue.push(start);

    while (this.queue.length > 0 && this.visited.size < this.options.maxPages) {
      const url = this.queue.shift()!;
      if (this.visited.has(url)) continue;
      if (this.options.excludePatterns.some((p) => p.test(url))) continue;

      this.visited.add(url);

      try {
        await this.page.goto(url, {
          waitUntil: 'networkidle',
          timeout: this.options.timeout,
        });

        const links = await extractAllLinks(this.page, url);
        this.allLinks.push(...links);

        for (const link of links) {
          const classified = classifyUrl(link.url, this.baseUrl);

          if (shouldCrawl(classified) && !this.visited.has(classified.resolved)) {
            this.queue.push(classified.resolved);
          }
        }

        if (this.options.delayBetweenRequests > 0) {
          await this.page.waitForTimeout(this.options.delayBetweenRequests);
        }
      } catch (error) {
        this.results.push({
          url,
          statusCode: 0,
          statusText: 'Navigation Error',
          responseTime: 0,
          sourceUrl: 'crawler',
          element: 'page',
          attribute: 'navigation',
          linkText: '',
          error: error instanceof Error ? error.message : String(error),
        });
      }
    }

    await this.verifyAllLinks();
    return this.results;
  }

  private async verifyAllLinks(): Promise<void> {
    const uniqueLinks = new Map<string, ExtractedLink>();
    for (const link of this.allLinks) {
      const classified = classifyUrl(link.url, this.baseUrl);
      if (classified.isMailto || classified.isTel || classified.isJavascript || classified.isDataUri) {
        continue;
      }
      if (!this.options.includeExternal && !classified.isInternal) {
        continue;
      }
      if (!uniqueLinks.has(classified.resolved)) {
        uniqueLinks.set(classified.resolved, link);
      }
    }

    const entries = Array.from(uniqueLinks.entries());
    for (let i = 0; i < entries.length; i += this.options.maxConcurrency) {
      const batch = entries.slice(i, i + this.options.maxConcurrency);
      const results = await Promise.allSettled(
        batch.map(([url, link]) => this.checkLink(url, link))
      );
      for (const result of results) {
        if (result.status === 'fulfilled') {
          this.results.push(result.value);
        }
      }
    }
  }

  private async checkLink(url: string, link: ExtractedLink): Promise<LinkCheckResult> {
    const start = Date.now();
    let lastError = '';

    for (let attempt = 0; attempt <= this.options.retries; attempt++) {
      try {
        const response = await this.request.head(url, {
          timeout: this.options.timeout,
          ignoreHTTPSErrors: true,
        });

        return {
          url,
          statusCode: response.status(),
          statusText: response.statusText(),
          responseTime: Date.now() - start,
          sourceUrl: link.sourceUrl,
          element: link.element,
          attribute: link.attribute,
          linkText: link.text,
        };
      } catch (error) {
        lastError = error instanceof Error ? error.message : String(error);
        if (attempt < this.options.retries) {
          await new Promise((r) => setTimeout(r, 1000 * (attempt + 1)));
        }
      }
    }

    // Fallback to GET if HEAD fails
    try {
      const response = await this.request.get(url, {
        timeout: this.options.timeout,
        ignoreHTTPSErrors: true,
      });
      return {
        url,
        statusCode: response.status(),
        statusText: response.statusText(),
        responseTime: Date.now() - start,
        sourceUrl: link.sourceUrl,
        element: link.element,
        attribute: link.attribute,
        linkText: link.text,
      };
    } catch {
      return {
        url,
        statusCode: 0,
        statusText: 'Request Failed',
        responseTime: Date.now() - start,
        sourceUrl: link.sourceUrl,
        element: link.element,
        attribute: link.attribute,
        linkText: link.text,
        error: lastError,
      };
    }
  }

  getResults(): LinkCheckResult[] {
    return this.results;
  }

  getBrokenLinks(): LinkCheckResult[] {
    return this.results.filter(
      (r) => r.statusCode === 0 || r.statusCode >= 400
    );
  }

  getRedirects(): LinkCheckResult[] {
    return this.results.filter(
      (r) => r.statusCode >= 300 && r.statusCode < 400
    );
  }
}

export const test = base.extend<{ crawler: LinkCrawler }>({
  crawler: async ({ page, request }, use) => {
    const baseUrl = process.env.BASE_URL || 'http://localhost:3000';
    const crawler = new LinkCrawler(page, request, baseUrl);
    await use(crawler);
  },
});

export { expect } from '@playwright/test';

Writing the Tests

With the fixture and helpers in place, the actual test files are concise and declarative.

// tests/link-checker/crawl-all-links.spec.ts
import { test, expect } from '../fixtures/crawler.fixture';

test.describe('Dead Link Detection - Full Site Crawl', () => {
  test('should have zero broken internal links', async ({ crawler }) => {
    const results = await crawler.crawl();
    const broken = results.filter(
      (r) => (r.statusCode === 0 || r.statusCode >= 400) && r.url.includes('localhost')
    );

    if (broken.length > 0) {
      const report = broken.map(
        (b) =>
          `  [${b.statusCode}] ${b.url}\n    Found on: ${b.sourceUrl}\n    Element: <${b.element} ${b.attribute}="${b.url}">\n    Text: "${b.linkText}"\n    Error: ${b.error || 'HTTP ' + b.statusCode}`
      );
      console.error(`Found ${broken.length} broken internal links:\n${report.join('\n')}`);
    }

    expect(broken).toHaveLength(0);
  });

  test('should have no server errors (5xx) on any page', async ({ crawler }) => {
    const results = await crawler.crawl();
    const serverErrors = results.filter(
      (r) => r.statusCode >= 500 && r.statusCode < 600
    );

    expect(serverErrors).toHaveLength(0);
  });

  test('should not have excessive redirects', async ({ crawler }) => {
    const results = await crawler.crawl();
    const redirects = crawler.getRedirects();

    // Redirects are not errors but excessive redirects indicate problems
    const redirectRatio = redirects.length / results.length;
    expect(redirectRatio).toBeLessThan(0.3); // Less than 30% of links redirect
  });
});

Handling Anchor Links and Hash Fragments

Anchor links (hash fragments) require special treatment because they reference elements within a page rather than separate resources. The server returns 200 for the page, but the target element might not exist.

// tests/link-checker/check-anchor-links.spec.ts
import { test, expect } from '@playwright/test';
import { extractAllLinks } from '../helpers/link-extractor';

test.describe('Anchor Link Validation', () => {
  test('all hash links should reference existing elements', async ({ page }) => {
    const baseUrl = process.env.BASE_URL || 'http://localhost:3000';
    const pagesToCheck = ['/', '/docs', '/about', '/faq'];
    const brokenAnchors: Array<{ page: string; hash: string }> = [];

    for (const path of pagesToCheck) {
      await page.goto(`${baseUrl}${path}`, { waitUntil: 'networkidle' });

      const anchorLinks = await page.evaluate(() => {
        const links: string[] = [];
        document.querySelectorAll('a[href*="#"]').forEach((el) => {
          const href = (el as HTMLAnchorElement).getAttribute('href');
          if (href && href.includes('#')) {
            const hash = href.split('#')[1];
            if (hash) links.push(hash);
          }
        });
        return links;
      });

      for (const hash of anchorLinks) {
        const exists = await page.evaluate((id) => {
          return !!(document.getElementById(id) || document.querySelector(`[name="${id}"]`));
        }, hash);

        if (!exists) {
          brokenAnchors.push({ page: path, hash });
        }
      }
    }

    if (brokenAnchors.length > 0) {
      console.error(
        'Broken anchor links:\n' +
          brokenAnchors.map((a) => `  ${a.page}#${a.hash}`).join('\n')
      );
    }

    expect(brokenAnchors).toHaveLength(0);
  });
});

Checking Images and Media Resources

Broken images degrade user experience significantly. This test specifically validates all image sources, including srcset attributes for responsive images, open graph images, and favicon references.

// tests/link-checker/check-images.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Image and Media Resource Validation', () => {
  test('all images should load successfully', async ({ page, request }) => {
    const baseUrl = process.env.BASE_URL || 'http://localhost:3000';
    await page.goto(baseUrl, { waitUntil: 'networkidle' });

    const imageUrls = await page.evaluate(() => {
      const urls: string[] = [];

      // Standard img src
      document.querySelectorAll('img[src]').forEach((el) => {
        urls.push((el as HTMLImageElement).src);
      });

      // srcset for responsive images
      document.querySelectorAll('img[srcset], source[srcset]').forEach((el) => {
        const srcset = el.getAttribute('srcset') || '';
        srcset.split(',').forEach((entry) => {
          const url = entry.trim().split(/\s+/)[0];
          if (url) {
            try {
              urls.push(new URL(url, window.location.href).href);
            } catch {
              // skip invalid URLs
            }
          }
        });
      });

      // Open Graph and meta images
      document
        .querySelectorAll('meta[property="og:image"], meta[name="twitter:image"]')
        .forEach((el) => {
          const content = el.getAttribute('content');
          if (content) {
            try {
              urls.push(new URL(content, window.location.href).href);
            } catch {
              // skip
            }
          }
        });

      // Favicon
      document.querySelectorAll('link[rel*="icon"]').forEach((el) => {
        const href = (el as HTMLLinkElement).href;
        if (href) urls.push(href);
      });

      return [...new Set(urls)];
    });

    const brokenImages: Array<{ url: string; status: number }> = [];

    for (const url of imageUrls) {
      try {
        const response = await request.head(url, { timeout: 15000 });
        if (response.status() >= 400) {
          brokenImages.push({ url, status: response.status() });
        }
      } catch {
        brokenImages.push({ url, status: 0 });
      }
    }

    if (brokenImages.length > 0) {
      console.error(
        `Found ${brokenImages.length} broken images:\n` +
          brokenImages.map((b) => `  [${b.status}] ${b.url}`).join('\n')
      );
    }

    expect(brokenImages).toHaveLength(0);
  });
});

Report Generation

Generating structured reports is essential for CI integration and historical tracking. Below is a report generator that produces both JSON and HTML output.

// tests/helpers/report-generator.ts
import * as fs from 'fs';
import * as path from 'path';
import { LinkCheckResult } from '../fixtures/crawler.fixture';

export interface LinkReport {
  timestamp: string;
  baseUrl: string;
  totalLinksChecked: number;
  brokenLinks: number;
  redirects: number;
  healthyLinks: number;
  results: LinkCheckResult[];
  summary: {
    byStatusCode: Record<number, number>;
    byElement: Record<string, number>;
    bySourcePage: Record<string, number>;
    averageResponseTime: number;
    slowestLinks: LinkCheckResult[];
  };
}

export function generateReport(
  results: LinkCheckResult[],
  baseUrl: string
): LinkReport {
  const byStatusCode: Record<number, number> = {};
  const byElement: Record<string, number> = {};
  const bySourcePage: Record<string, number> = {};
  let totalResponseTime = 0;

  for (const r of results) {
    byStatusCode[r.statusCode] = (byStatusCode[r.statusCode] || 0) + 1;
    byElement[r.element] = (byElement[r.element] || 0) + 1;

    const brokenOnly = r.statusCode === 0 || r.statusCode >= 400;
    if (brokenOnly) {
      bySourcePage[r.sourceUrl] = (bySourcePage[r.sourceUrl] || 0) + 1;
    }
    totalResponseTime += r.responseTime;
  }

  const broken = results.filter((r) => r.statusCode === 0 || r.statusCode >= 400);
  const redirects = results.filter((r) => r.statusCode >= 300 && r.statusCode < 400);
  const slowest = [...results].sort((a, b) => b.responseTime - a.responseTime).slice(0, 10);

  return {
    timestamp: new Date().toISOString(),
    baseUrl,
    totalLinksChecked: results.length,
    brokenLinks: broken.length,
    redirects: redirects.length,
    healthyLinks: results.length - broken.length - redirects.length,
    results,
    summary: {
      byStatusCode,
      byElement,
      bySourcePage,
      averageResponseTime: results.length > 0 ? totalResponseTime / results.length : 0,
      slowestLinks: slowest,
    },
  };
}

export function saveReport(report: LinkReport, outputDir: string): void {
  fs.mkdirSync(outputDir, { recursive: true });

  // JSON report
  const jsonPath = path.join(outputDir, 'broken-links.json');
  fs.writeFileSync(jsonPath, JSON.stringify(report, null, 2));

  // HTML report
  const htmlPath = path.join(outputDir, 'broken-links.html');
  const html = buildHtmlReport(report);
  fs.writeFileSync(htmlPath, html);

  console.log(`Reports saved to:\n  JSON: ${jsonPath}\n  HTML: ${htmlPath}`);
}

function buildHtmlReport(report: LinkReport): string {
  const brokenRows = report.results
    .filter((r) => r.statusCode === 0 || r.statusCode >= 400)
    .map(
      (r) => `
      <tr>
        <td>${r.statusCode}</td>
        <td><a href="${r.url}">${r.url}</a></td>
        <td>${r.sourceUrl}</td>
        <td>&lt;${r.element}&gt;</td>
        <td>${r.linkText}</td>
        <td>${r.error || ''}</td>
      </tr>`
    )
    .join('');