TL;DR
I extracted public LinkedIn pages with ScrapeGraphAI. Gates, Jensen Huang, and Microsoft's company page returned JSON at 5 credits with stealth off. Satya Nadella returned HTTP 502 twice, including with stealth.
The companion Colab runs the same schema extract. Paste your own key. Experience stays empty on the public profile because the login modal hides the work history.
Send a public profile or company URL to ScrapeGraphAI extract and pass a Pydantic schema into the call. I ran that against live pages while writing this guide. Header fields came back (name, location, followers, current company). The full work history did not. LinkedIn's logged-out view puts a sign-in modal over the experience block, so experience was [] on every person who succeeded.
I would not treat a public LinkedIn profile as a full resume dump. If you need the job list, you are looking at a logged-in page or a different source. The companion Google Colab is the runnable version of the calls below. Paste your own key. Nothing in the notebook stores one.
Sales workflows that start from public pages sit next to this in How to Generate Leads Using ScrapeGraphAI from LinkedIn Data. Job listings are a separate extract and a separate article: 6 Best Tools for Scraping Job Postings in 2026.
What a public page actually shows
I opened https://www.linkedin.com/in/williamhgates/ logged out. The header is still there: Bill Gates, Seattle, 41M followers, and the headline that names the Gates Foundation and Microsoft. A white modal sits on the rest of the page ("Browse anonymously, connect when ready"). You cannot read a usable experience list in that state.
Extract sees that same page. If the prompt demands a ten-row work history, you get an empty array. If it asks for the header, you often get the header.
The Microsoft company page uses the same modal trick and leaves more text around it. The overlay wants you to sign in to see who you already know. Behind it I could still read Software Development, Redmond, Washington, 28.9M followers, the About paragraph, and https://news.microsoft.com/.
I keep person pages and company pages on different schemas. A profile model that also waits for industry and specialties just fills those with nulls. A company model that asks for education does the reverse.
Working extract: pass the schema in
This is the Gates call. The schema is an argument to extract, not a cleanup step after you print JSON. extract and the Python SDK both take schema. Current scrapegraph-py wants Python 3.12+.
import json
from getpass import getpass
from pydantic import BaseModel, Field
from scrapegraph_py import ScrapeGraphAI
class Experience(BaseModel):
title: str | None = Field(default=None, description="Job title as shown")
company: str | None = Field(default=None, description="Company name as shown")
duration: str | None = Field(default=None, description="Duration text as shown")
class Education(BaseModel):
school: str | None = Field(default=None, description="School name as shown")
degree: str | None = Field(default=None, description="Degree or field as shown")
dates: str | None = Field(default=None, description="Attendance dates as shown")
class LinkedInProfile(BaseModel):
name: str | None = Field(default=None, description="Full name as shown on the public page")
headline: str | None = Field(default=None, description="Headline text under the name")
location: str | None = Field(default=None, description="Location text if visible")
followers: str | None = Field(default=None, description="Follower count text if visible")
current_title: str | None = Field(default=None, description="Current role title if visible")
current_company: str | None = Field(default=None, description="Current company if visible")
experience: list[Experience] = Field(default_factory=list, description="Visible experience rows")
education: list[Education] = Field(default_factory=list, description="Visible education rows")
profile_url: str | None = Field(default=None, description="Canonical public profile URL")
api_key = getpass("ScrapeGraphAI API key: ")
sgai = ScrapeGraphAI(api_key=api_key)
profile_url = "https://www.linkedin.com/in/williamhgates/"
profile_prompt = (
"This is a public LinkedIn profile page. Extract only fields visible without login: "
"name, headline, location, followers, current_title, current_company, "
"experience as an array of {title, company, duration}, "
"education as an array of {school, degree, dates}, and profile_url. "
"Use null for missing fields. Use an empty array when a section is not visible. "
"Do not infer email, phone, or any private information. "
"If the page is a login wall, captcha, or empty shell, return all fields as null "
"and empty arrays."
)
response = sgai.extract(
profile_prompt,
url=profile_url,
schema=LinkedInProfile.model_json_schema(),
)
if response.status != "success":
raise RuntimeError(response.error)
profile = LinkedInProfile.model_validate(response.data.json_data)
print(json.dumps(profile.model_dump(), indent=2))Pass the schema into extract. I still validate the response. That does not replace telling the service the shape you want.
Create a key in the dashboard. Keep it out of the notebook file.
What the extract returned
Gates, stealth off, 28.04 seconds, 5 credits:
{
"name": "Bill Gates",
"headline": "Chair of the Gates Foundation. Founder of Breakthrough Energy. Co-founder of Microsoft…",
"location": "Seattle",
"followers": "40602208",
"current_title": null,
"current_company": "Gates Foundation",
"experience": [],
"education": [{"school": "Harvard University", "degree": null, "dates": "1973-1975"}],
"profile_url": "https://www.linkedin.com/in/williamhgates/"
}The page badge said 41M followers. Extract returned "40602208". Same neighborhood, different formatting. current_title stayed null even though the headline names the chair role. I left it null. Copying the headline into job_title is a mapping decision I want to see in my code, not a guess inside extract.
Jensen Huang (https://www.linkedin.com/in/jenhsunhuang/) took 44.4 seconds, also 5 credits, also stealth off. Name came through. Location was Los Altos, California, United States. Followers: 1,381,584. Company: NVIDIA. Headline was null. Experience was empty again. Education had Stanford, 1990-1992, and a second row with school: null and dates 1980-1984. I kept the empty school. Deleting the row would have looked cleaner and would have been a lie about what the model returned.
Satya Nadella returned HTTP 502 with and without stealth. Plain extract: 17.8 seconds, 502. Stealth: 12.45 seconds, still 502. There is no success JSON for https://www.linkedin.com/in/satyanadella/ in this test, and I am not going to invent one.
Microsoft's company page was the easiest fetch of the day: 10.4 seconds, 5 credits, stealth off.
{
"name": "Microsoft",
"industry": "Software Development",
"headquarters": "Redmond, Washington",
"followers": "28957657",
"about": "Every company has a mission. What's ours? To empower every person and every organization to achieve more. We believe technology can and should be a force for good and that meaningful innovation contributes to a brighter world in the future and today. Our culture doesn't just encourage curiosity; it embraces it. Each day we make progress together by showing up as our authentic selves. We show up with a learn-it-all mentality, we show up cheering on others, knowing their success doesn't diminish our own. We show up every day open to learning our own biases, changing our behavior, and inviting in differences. Because impact matters. Microsoft operates in 190 countries and is made up of approximately 228,000 passionate employees worldwide.",
"website": "https://news.microsoft.com/",
"employee_count": "232851",
"specialties": ["Business Software", "Developer Tools", "Home & Educational Software", "Tablets", "Search", "Advertising", "Servers", "Windows Operating System", "Windows Applications & Platforms", "Smartphones", "Cloud Computing", "Quantum Computing", "Future of Work", "Productivity", "AI", "Artificial Intelligence", "Machine Learning", "Laptops", "Mixed Reality", "Virtual Reality", "Gaming", "Developers", "and IT Professional"],
"company_url": "https://www.linkedin.com/company/microsoft"
}The public page showed 28,957,650 followers and "10,001+ employees". Extract returned "28957657" and "232851". I stored both as strings, the way they arrived. The last specialty came through as "and IT Professional", which is the page's own trailing "and", not something I cleaned up.
Company pages returned more usable fields than profiles. On Microsoft I stored industry, headquarters, About, website, employee count, and a long specialties list. On Gates I stored a name, a headline, a city, a follower string, a company, and one education row. That gap is the login wall, not a bad prompt.
The Microsoft call used the same client and the same schema= argument. Store the Gates prompt as profile_prompt if you want the batch loop later to run as written.
class LinkedInCompany(BaseModel):
name: str | None = Field(default=None, description="Company name as shown")
industry: str | None = Field(default=None, description="Industry text if visible")
headquarters: str | None = Field(default=None, description="Headquarters location if visible")
followers: str | None = Field(default=None, description="Follower count text if visible")
about: str | None = Field(default=None, description="About / overview text if visible")
website: str | None = Field(default=None, description="Company website if visible")
employee_count: str | None = Field(default=None, description="Employee count range if visible")
specialties: list[str] = Field(default_factory=list, description="Specialties if listed")
company_url: str | None = Field(default=None, description="Canonical public company URL")
company_url = "https://www.linkedin.com/company/microsoft/"
company_prompt = (
"This is a public LinkedIn company page. Extract only fields visible without login: "
"name, industry, headquarters, followers, about, website, employee_count, "
"specialties, and company_url. Use null for missing fields. "
"Do not infer private information."
)
company_response = sgai.extract(
company_prompt,
url=company_url,
schema=LinkedInCompany.model_json_schema(),
)
if company_response.status != "success":
raise RuntimeError(company_response.error)
company = LinkedInCompany.model_validate(company_response.data.json_data)Profiles, company pages, and jobs are different extracts
I run three prompts on purpose.
Profile fields: name, headline, location, followers, current title, current company, experience, education, profile URL.
Company fields: name, industry, headquarters, followers, about, website, employee count, specialties, company URL.
Job fields: title, company, location, listing URL, pointed at a jobs search or a company careers board.
Keep profile, company, and job extracts as separate schemas. A blended model is how you end up ignoring half the keys on every row. Job listings belong in the job board scraping guide. That post is where LinkedIn jobs search was tested. This post is profiles and company pages.
If the next step is outbound from public pages, AI-Powered Web Scraping for Lead Generation at Scale covers the wider pipeline. LinkedIn can be one input. It should not invent an email.
Normalize the URL, then batch
I would not fire extract at a raw spreadsheet of LinkedIn links. Tracking parameters, linkedin.com versus www.linkedin.com, and the same person pasted twice will spend credits and split records.
Paste this, drop your list in the string, run it.
from urllib.parse import urlparse, urlunparse
raw = """
https://www.linkedin.com/in/williamhgates/?trk=public_profile
https://linkedin.com/in/williamhgates
https://www.linkedin.com/in/jenhsunhuang/
https://www.linkedin.com/company/microsoft/
https://www.linkedin.com/in/satyanadella/
"""
def canonicalize(url: str) -> str:
parsed = urlparse(url.strip())
host = "www.linkedin.com"
path = parsed.path.rstrip("/") + "/"
return urlunparse(("https", host, path, "", "", ""))
urls = []
seen = set()
for line in raw.splitlines():
line = line.strip()
if not line:
continue
canon = canonicalize(line)
if canon in seen:
continue
seen.add(canon)
urls.append(canon)
print(urls)That collapses the two Gates variants, keeps Huang, Microsoft, and Satya, and drops the query string. Then loop with the prompts from the extract section (reuse profile_prompt / company_prompt and the two models):
from datetime import datetime, timezone
rows = []
for url in urls:
kind = "company" if "/company/" in url else "profile"
schema = LinkedInCompany if kind == "company" else LinkedInProfile
prompt = company_prompt if kind == "company" else profile_prompt
response = sgai.extract(prompt, url=url, schema=schema.model_json_schema())
collected_at = datetime.now(timezone.utc).isoformat()
if response.status != "success":
rows.append({
"source_url": url,
"kind": kind,
"status": "failed",
"reason": str(response.error),
"collected_at": collected_at,
"data": None,
})
continue
rows.append({
"source_url": url,
"kind": kind,
"status": "completed",
"reason": None,
"collected_at": collected_at,
"data": schema.model_validate(response.data.json_data).model_dump(),
})Satya lands in failed with HTTP 502. Leave it there. Do not retry it in a tight loop.
Validate, then map to a CRM
The extract already used the schema. Validate again on the way into your store so a drifted field fails in your process, not in HubSpot.
profile = LinkedInProfile.model_validate(response.data.json_data)
def to_crm_record(profile: LinkedInProfile) -> dict:
return {
"source": "linkedin_public",
"source_url": profile.profile_url or profile_url,
"full_name": profile.name,
"job_title": profile.current_title,
"company": profile.current_company,
"location": profile.location,
"headline": profile.headline,
"followers": profile.followers,
}Match on source_url before you create a contact. Update the existing row and stamp collected_at. When current_title is null and the headline is not, I copy the headline into headline or notes. I do not silently promote it to job title.
Do not ask extract for email or phone. Public LinkedIn pages do not hand those over. If you need them, that is a separate enrichment path with its own consent rules.
Error handling, including HTTP 502
A failed request and an empty experience array are different events. Gates finished and the page hid the section. Satya never produced JSON.
def extract_one(url: str, prompt: str, schema):
try:
response = sgai.extract(prompt, url=url, schema=schema.model_json_schema())
except Exception as error:
return {"source_url": url, "status": "failed", "reason": str(error)}
if response.status != "success":
return {
"source_url": url,
"status": "failed",
"reason": str(response.error or "extract request did not complete"),
}
data = schema.model_validate(response.data.json_data)
missing = [
name for name, value in data.model_dump().items()
if value in (None, "", [])
]
status = "partial" if missing else "completed"
return {
"source_url": url,
"status": status,
"missing_fields": missing,
"data": data.model_dump(),
}On this test set, Gates and Huang are partial (empty experience, some nulls). Microsoft company is close to completed. Satya is failed / HTTP 502.
I retry a 502 once, later, and I leave stealth off on that first retry. If the second plain call 502s, I log it and go to the next URL. Flipping stealth on immediately is how you spend 10 credits to watch the same error twice. I did that once for Satya so I could write it down. I would not make it the default.
Credits
Extract costs 5 credits. Stealth is +5 and did not save that URL.
Gates, Huang, and Microsoft were 5 credits each. The Satya failures logged 0 credits on the success-path counter I keep; I still would not treat a 502 as free. Stealth adds 5 on a successful fetch. Plan math is on ScrapeGraphAI Pricing: Plans and Credits Guide and the pricing page. Starter is $20 for 10,000 credits a month, which is 2,000 plain extracts if you are not also scraping something else.
Leave stealth off until a page comes back empty or blocked. I turned it on for Satya because the first call 502'd. The second call 502'd too.
Markdown scrape is 1 credit if an agent only needs readable text. You will not get a typed profile out of it.
Public pages only
This workflow is for pages that load without an account. No private profiles, no InMail, no emails pulled from a logged-in session, no stepping through the sign-in modal.
LinkedIn's user agreement restricts unauthenticated scraping. A screenshot of public HTML is not a license to crawl the site. Rate-limit your own jobs. Keep source_url and collected_at. Drop fields you do not need. Give people a way to correct or delete a stored record when the law you are under says you must.
I am not your lawyer. A practical overview is Is Web Scraping Legal? Legality Guide and Best Practices. ScrapeGraphAI fetches the URL you send. It does not decide whether your use is permitted.
Run it in Google Colab
The companion notebook follows the same steps as this post: install, profile extract on Gates, company extract on Microsoft, URL cleanup, then one CSV. Paste your own key. The notebook has no key and no executed outputs.
Open it, paste your own key, run top to bottom. The notebook has no key and no executed outputs.
If you later wire the same extract into an agent, Enhance AI Agents with LangChain Web Scraping is the adjacent writeup. The public-page pattern on Instagram is Instagram Scraping with ScrapeGraphAI Extract.
Related Articles
- How to Generate Leads Using ScrapeGraphAI from LinkedIn Data
- AI-Powered Web Scraping for Lead Generation at Scale
- Instagram Scraping with ScrapeGraphAI Extract
- 6 Best Tools for Scraping Job Postings in 2026
- ScrapeGraphAI Pricing: Plans and Credits Guide
- Is Web Scraping Legal? Legality Guide and Best Practices
- Enhance AI Agents with LangChain Web Scraping