LinkedIn Scraper with ScrapeGraphAI: Profiles to JSON

TL;DR

LinkedIn data extraction works best when you ask for a narrow public data schema, run small batches, validate every response, and send the final JSON into your CRM, recruiting database, or market research workflow.

ScrapeGraphAI helps because you can describe the fields you need in natural language instead of maintaining brittle selectors for names, locations, headlines, experience blocks, and education sections.

LinkedIn pages are useful for sales, recruiting, and market research, but they are awkward to extract from with hand-written scripts. The markup changes, content loads dynamically, and profile pages often have missing fields. A selector-based scraper usually turns into a maintenance job.

ScrapeGraphAI's extract endpoint gives you a different workflow. You provide a public LinkedIn URL and a clear extraction prompt, then receive structured JSON that your application can validate and store. That makes it a better fit for teams that need repeatable LinkedIn data extraction without building a dedicated parser for every page layout.

If your goal is sales sourcing, pair this workflow with the LinkedIn lead generation guide.

What a LinkedIn Scraper Should Return

Start by deciding which fields your downstream workflow actually needs. Asking for everything creates messy output and raises the chance that missing sections break your pipeline.

A practical profile extraction schema usually includes:

• name
• headline
• location
• followers
• current_company
• current_title
• experience
• education
• profile_url

Recruiting teams may also need skills, certifications, job history, and education. Sales teams often care more about current role, company, seniority, geography, and profile URL. Market research teams may aggregate job titles, industries, and hiring signals across many public profiles.

The safest prompt is specific about field names, output shape, and what to do when a field is missing. For example:

Extract public LinkedIn profile data as JSON.
 
Return:
- name
- headline
- location
- followers
- current_title
- current_company
- experience: array of { title, company, duration }
- education: array of { school, degree, dates }
 
Use null for missing fields. Do not infer private information.

That prompt gives the model enough structure to be useful while keeping the result honest.

Basic Python Example

Here is a small profile extraction script using the ScrapeGraphAI Python SDK:

import json
from scrapegraph_py import ScrapeGraphAI
 
sgai = ScrapeGraphAI()  # uses SGAI_API_KEY from your environment
 
profiles = [
    "https://www.linkedin.com/in/williamhgates/",
    "https://www.linkedin.com/in/jenhsunhuang/",
]
 
prompt = """
Extract public LinkedIn profile data as JSON.
 
Return name, headline, location, followers, current title,
current company, experience, education, and profile URL.
Use null for missing fields.
"""
 
for profile in profiles:
    response = sgai.extract(prompt, url=profile)
 
    if response.status == "success":
        print(json.dumps(response.data.json_data, indent=2))
    else:
        print(f"Extraction failed for {profile}")

This example keeps the prompt narrow and treats each profile as a separate job. In production, you would normally write each result to a queue, database, or enrichment table instead of printing it.

Example Output

The exact result depends on the public page and the fields visible when the request runs. A clean output shape might look like this:

{
  "name": "Example Person",
  "headline": "Founder and CEO at Example Company",
  "location": "San Francisco Bay Area",
  "followers": "120,000",
  "current_title": "Founder and CEO",
  "current_company": "Example Company",
  "experience": [
    {
      "title": "Founder and CEO",
      "company": "Example Company",
      "duration": "2020 - Present"
    }
  ],
  "education": [
    {
      "school": "Example University",
      "degree": "Computer Science",
      "dates": "2012 - 2016"
    }
  ],
  "profile_url": "https://www.linkedin.com/in/example/"
}

Notice the shape is stable even when some values are missing. That matters more than extracting every possible detail. Stable JSON lets your enrichment, matching, and scoring code stay simple.

Use a Schema Before Saving Data

Do not send raw extraction output directly into your CRM or recruiting system. Validate it first. A small schema catches missing fields, unexpected arrays, and values that need cleanup.

from pydantic import BaseModel, Field
 
 
class Experience(BaseModel):
    title: str | None = None
    company: str | None = None
    duration: str | None = None
 
 
class LinkedInProfile(BaseModel):
    name: str | None = None
    headline: str | None = None
    location: str | None = None
    followers: str | None = None
    current_title: str | None = None
    current_company: str | None = None
    experience: list[Experience] = Field(default_factory=list)
    profile_url: str

Then validate each extraction:

data = response.data.json_data
profile = LinkedInProfile.model_validate(data)

If validation fails, route the profile to a review queue or rerun it with a tighter prompt. This is much easier than debugging bad CRM records later.

Batch Workflow for Lead Generation

A real LinkedIn extraction workflow usually has five steps.

1. Collect candidate URLs from a source you are allowed to use.
2. Deduplicate URLs before sending requests.
3. Extract only the fields needed for your campaign or recruiting workflow.
4. Validate the JSON and normalize company names, titles, and locations.
5. Save the result with source URL, extraction date, and status.

For lead generation, combine public profile fields with company pages, directories, or website data. The lead generation scraping guide covers that broader pipeline. LinkedIn can be one signal, but it should not be the only source of truth for outreach or hiring decisions.

Data Quality Checklist

LinkedIn extraction quality is mostly decided before the request runs. If the input URLs are noisy, the output will be noisy too. Keep a small checklist for every batch.

First, normalize URLs. Remove tracking parameters, lowercase the host, and store one canonical URL per person. If the same profile appears from a search result, a spreadsheet, and a CRM export, deduplicate it before extraction. Duplicate profile requests waste credits and can create conflicting records.

Second, separate required fields from optional fields. For a sales workflow, name, current_company, current_title, location, and profile_url may be enough to enrich an account. For recruiting, you may need experience, education, and skills. Optional fields should never block the whole record.

Third, keep a confidence field in your own database. You can mark a profile as complete, partial, or needs_review based on the fields returned. A partial profile can still be useful when it has the current company and role, but it should not be treated like a fully enriched contact.

Fourth, store the prompt version with each result. When you improve the prompt later, you can compare records by prompt version and decide whether older records need a rerun.

{
  "profile_url": "https://www.linkedin.com/in/example/",
  "status": "partial",
  "prompt_version": "linkedin-profile-v3",
  "extracted_at": "2026-07-02",
  "missing_fields": ["education"]
}

That small audit trail makes the workflow easier to debug when sales or recruiting teams ask why a field is blank.

Prompt Patterns That Work

Use direct prompts that define the result. These are good starting points:

Extract the person's name, headline, current company, current role, location,
and profile URL. Return JSON only. Use null for missing fields.

Extract work experience from this public LinkedIn profile.
Return an array of jobs with title, company, duration, and description.
If a section is not visible, return an empty array.

Extract recruiting-relevant public profile data:
name, headline, location, current role, current company, past companies,
education, and skills. Do not infer email addresses or private details.

Avoid vague prompts like "get everything." They produce inconsistent results and make validation harder. A narrow prompt is easier to test and easier to improve.

Error Handling

LinkedIn pages can be unavailable, sparse, redirected, or missing the sections you expected. Treat those cases as normal.

def extract_profile(profile: str) -> dict:
    try:
        response = sgai.extract(prompt, url=profile)
    except Exception as error:
        return {
            "profile_url": profile,
            "status": "failed",
            "reason": str(error),
        }
 
    if response.status != "success":
        return {
            "profile_url": profile,
            "status": "failed",
            "reason": "extract request did not complete",
        }
 
    return {
        "profile_url": profile,
        "status": "completed",
        "data": response.data.json_data,
    }

Store failures separately from empty profiles. A failed request means the extraction did not complete. An empty field means the page did not expose that data or the prompt did not ask for it clearly enough.

Sending LinkedIn Data to a CRM

Once a profile validates, map it into the system that owns the next step. For sales teams, that is usually a CRM or enrichment table. For recruiting teams, it may be an applicant tracking system or a sourcing database.

Use a stable mapping instead of passing raw JSON through the whole stack:

def to_crm_record(profile: LinkedInProfile) -> dict:
    return {
        "source": "linkedin",
        "source_url": profile.profile_url,
        "full_name": profile.name,
        "job_title": profile.current_title,
        "company": profile.current_company,
        "location": profile.location,
        "headline": profile.headline,
    }

Before creating a new CRM contact, check whether the profile URL already exists. If it does, update the existing record with a new extraction date instead of creating a duplicate. If your CRM supports custom fields, keep source_url, extracted_at, and extraction_status visible to the team.

Avoid turning extracted public profile data into guessed contact details. If you need emails or phone numbers, use a separate enrichment process with its own consent, compliance, and validation rules.

Responsible LinkedIn Extraction

LinkedIn profile data can include personal data. Keep the workflow narrow and document why each field is needed. Respect LinkedIn's terms, privacy rules, and applicable laws for your jurisdiction.

Good practice includes:

• extract public fields only
• avoid private profiles and authenticated-only data
• keep source URLs and extraction dates
• remove fields that are not needed for the stated purpose
• give users a way to correct or remove stored data when required
• rate-limit jobs and avoid aggressive crawling

For legal context around scraping, read Is Web Scraping Legal?. It is a practical overview, not legal advice.

When to Use ScrapeGraphAI

Use ScrapeGraphAI for LinkedIn data extraction when you need structured output from changing public pages and want to avoid maintaining page-specific selectors. It is especially useful when:

• the required fields are easy to describe in natural language
• pages have similar intent but different layouts
• your team wants JSON output for a CRM, ATS, enrichment table, or research workflow
• you need a repeatable extraction step inside a larger data pipeline

Use a dedicated vendor or official API path when you need authenticated account data, message data, private profile access, or contractual guarantees around LinkedIn-specific coverage.

Testing the Workflow Before Scaling

Run a small test batch before any large extraction job. Pick 20 to 50 public profiles that represent the pages your workflow will see: complete profiles, sparse profiles, founders, recruiters, engineers, sales leaders, and profiles from different regions. That sample gives you better prompt feedback than testing one famous profile.

Track these metrics during the test:

• completed requests
• failed requests
• records that pass schema validation
• records missing required fields
• average fields populated per profile
• duplicate URLs removed before extraction

Review a handful of records manually. The goal is not perfect extraction from every page. The goal is a predictable workflow where missing data is explicit, errors are routed correctly, and the final JSON is safe to use in the next system.

If the test finds frequent missing fields, tighten the prompt before changing the pipeline. If the schema catches many malformed records, adjust the schema only when the new shape is genuinely useful for the business workflow. Do not loosen validation just to make a bad batch look successful.

Common Mistakes

The most common mistake is treating LinkedIn extraction as a one-step scrape. The extraction request is only one part of the workflow. You still need URL cleanup, schema validation, deduplication, status tracking, and a review path for partial records.

Another mistake is mixing different intents in one prompt. Profile extraction, company extraction, and job-post extraction should be separate prompts with separate schemas. That keeps the output predictable and makes failures easier to diagnose.

Related Articles

• LinkedIn Lead Generation
• Lead Generation Scraping
• Is Web Scraping Legal?
• Building Intelligent Agents