Bright Data - Structured Data Feeds

Extract structured data from major websites with automatic parsing. No scraping logic needed - just provide a URL and get clean JSON data.

Setup

Environment Variables (Required)

export BRIGHTDATA_API_KEY="your-api-key"

Optional

export BRIGHTDATA_POLLING_TIMEOUT=600  # Max seconds to wait (default: 600)

Get your API key from Bright Data Dashboard.

Usage

bash scripts/datasets.sh <dataset_type> <url> [additional_params...]

Available Datasets

E-Commerce

Dataset	Command	Description
Amazon Product	datasets.sh amazon_product <url>	Product details, pricing, ratings
Amazon Reviews	datasets.sh amazon_product_reviews <url>	Customer reviews for a product
Amazon Search	datasets.sh amazon_product_search <keyword> <domain_url>	Search results
Walmart Product	datasets.sh walmart_product <url>	Product details from Walmart
Walmart Seller	datasets.sh walmart_seller <url>	Seller information
eBay Product	datasets.sh ebay_product <url>	eBay listing details
Home Depot	datasets.sh homedepot_products <url>	Home Depot product data
Zara	datasets.sh zara_products <url>	Zara product details
Etsy	datasets.sh etsy_products <url>	Etsy listing data
Best Buy	datasets.sh bestbuy_products <url>	Best Buy product info

Professional Networks

Dataset	Command	Description
LinkedIn Person	datasets.sh linkedin_person_profile <url>	Profile data (experience, skills)
LinkedIn Company	datasets.sh linkedin_company_profile <url>	Company page data
LinkedIn Jobs	datasets.sh linkedin_job_listings <url>	Job posting details
LinkedIn Posts	datasets.sh linkedin_posts <url>	Post content and engagement
LinkedIn Search	datasets.sh linkedin_people_search <url> <first> <last>	Find people
Crunchbase	datasets.sh crunchbase_company <url>	Company funding, employees
ZoomInfo	datasets.sh zoominfo_company_profile <url>	Company profile data

Instagram

Dataset	Command	Description
Profiles	datasets.sh instagram_profiles <url>	Bio, followers, following
Posts	datasets.sh instagram_posts <url>	Post details, likes, captions
Reels	datasets.sh instagram_reels <url>	Reel data and metrics
Comments	datasets.sh instagram_comments <url>	Post comments

Facebook

Dataset	Command	Description
Posts	datasets.sh facebook_posts <url>	Post content and reactions
Marketplace	datasets.sh facebook_marketplace_listings <url>	Listing details
Reviews	datasets.sh facebook_company_reviews <url> [num]	Company reviews
Events	datasets.sh facebook_events <url>	Event details

TikTok

Dataset	Command	Description
Profiles	datasets.sh tiktok_profiles <url>	Creator profile data
Posts	datasets.sh tiktok_posts <url>	Video details and metrics
Shop	datasets.sh tiktok_shop <url>	TikTok Shop product data
Comments	datasets.sh tiktok_comments <url>	Video comments

YouTube

Dataset	Command	Description
Profiles	datasets.sh youtube_profiles <url>	Channel data
Videos	datasets.sh youtube_videos <url>	Video details and stats
Comments	datasets.sh youtube_comments <url> [num]	Video comments (default: 10)

Other Social

Dataset	Command	Description
X (Twitter)	datasets.sh x_posts <url>	Tweet data
Reddit	datasets.sh reddit_posts <url>	Post and comment data

Google Services

Dataset	Command	Description
Maps Reviews	datasets.sh google_maps_reviews <url> [days]	Business reviews (default: 3 days)
Shopping	datasets.sh google_shopping <url>	Product comparison data
Play Store	datasets.sh google_play_store <url>	App details and reviews

Other

Dataset	Command	Description
Apple App Store	datasets.sh apple_app_store <url>	iOS app data
Reuters News	datasets.sh reuter_news <url>	News article content
GitHub	datasets.sh github_repository_file <url>	Repository file data
Yahoo Finance	datasets.sh yahoo_finance_business <url>	Stock and company data
Zillow	datasets.sh zillow_properties_listing <url>	Property listing details
Booking.com	datasets.sh booking_hotel_listings <url>	Hotel listing data

Examples

Get LinkedIn Profile

bash scripts/datasets.sh linkedin_person_profile "https://www.linkedin.com/in/satyanadella/"

Get Amazon Product

bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB"

Get Instagram Profile

bash scripts/datasets.sh instagram_profiles "https://www.instagram.com/natgeo/"

Get YouTube Comments

bash scripts/datasets.sh youtube_comments "https://www.youtube.com/watch?v=dQw4w9WgXcQ" 20

Search Amazon

bash scripts/datasets.sh amazon_product_search "wireless headphones" "https://www.amazon.com"

Output Format

Returns structured JSON with website-specific fields. Example for LinkedIn profile:

{
  "name": "Satya Nadella",
  "headline": "Chairman and CEO at Microsoft",
  "location": "Greater Seattle Area",
  "connections": "500+",
  "experience": [...],
  "education": [...],
  "skills": [...]
}

How It Works

1. Trigger: Sends URL to Bright Data's Web Data API
2. Poll: Waits for data collection to complete (checks every second)
3. Return: Outputs structured JSON when ready

The polling mechanism handles rate limits and ensures data quality by waiting for full extraction.

Advanced: Direct Fetch

For custom dataset IDs or advanced use cases:

bash scripts/fetch.sh <dataset_id> '<json_input>'

Example:

bash scripts/fetch.sh gd_l1viktl72bvl7bjuj0 '{"url":"https://linkedin.com/in/someone"}'