ScrapeGraphAI x Hivemind: Enrich AI Agent Skills With Web Scraping

TL;DR

A copy-paste Claude skill that takes the focused skills Hivemind captures from your sessions and layers real web research from ScrapeGraphAI on top.

• The setup: Hivemind, from Activeloop, watches your Claude Code sessions and mines them into reusable skill files. Each one captures a real lesson your team learned, distilled down to the essentials.
• The add-on: A skill called enrich-hivemind-skill. It reads one of Hivemind's skills, researches the topic with ScrapeGraphAI in one call, and expands the file with gotchas, code examples, and references.
• How to use it: Drop the skill in ~/.claude/skills/, then tell your agent "enrich the X skill." It does the rest.
• The result: Hivemind's focused 44-line skill grows into a 260-line playbook, its original lesson intact and every added section traceable to a real source.

Here is the pipeline in one line: Hivemind captures and codifies your sessions into skills; ScrapeGraphAI enriches those skills with what the open web already documents.

Hivemind is an agent memory tool from Activeloop. It watches your Claude Code sessions, spots recurring patterns, and writes them down as reusable SKILL.md files so the next agent on your team starts smarter. It runs a capture -> codify -> propagate loop, and the codify step uses a small model to turn raw session traces into a skill. It already does the hard part: turning a real session into a reusable lesson that travels across your team.

So we built a skill that takes those skills further. You hand it one of Hivemind's skills, it researches the topic against the open web with ScrapeGraphAI, and expands the file with everything the docs and the community already know. Hivemind supplies the hard-won lesson; ScrapeGraphAI adds the surrounding depth. First, some quick context on Hivemind itself; then what the skill does, how to use it, and the full skill to copy.

What Is Hivemind

Hivemind is the shared memory layer for teams that run multiple AI agents and need them to share context without manual wiring. It captures what happens across your sessions, codifies the lessons, and propagates them so every agent on your team starts smarter. Everything your agents need to remember, in one place:

• AI session summaries — A background worker writes a wiki-style summary of every session (key decisions, code changes, next steps) and indexes it for recall.
• Org-level access control — Invite team members with ADMIN, WRITE, or READ permissions, and switch between orgs and workspaces on the fly.
• Virtual filesystem — Memory looks like a folder under ~/.deeplake/memory/. Use cat, ls, grep — the same commands you already know. No new APIs to learn.
• Real-time sync — All writes go to Deeplake Cloud and are instantly available to every agent in the org, with sub-second propagation.
• Zero infrastructure — No databases to run, no servers to maintain. Deeplake Cloud handles storage, indexing, and search as a managed service.
• Privacy controls — Opt out of capture anytime with HIVEMIND_CAPTURE=false. Credentials stay local with strict file permissions (0600 / 0700).

Installing Hivemind

One command, every agent on your machine. The unified installer detects every supported assistant, wires up the hooks, and opens your browser once for SSO. Restart your assistants and they all share the same brain.

Recommended — all assistants:

npm install -g @deeplake/hivemind && hivemind install

That auto-detects Claude Code, OpenClaw, Codex, Cursor, Hermes, and pi.

Prefer to wire up a single assistant? After npm install -g, target one instead of all of them:

hivemind claude install   # Claude Code
hivemind claw install     # OpenClaw
hivemind codex install    # Codex
hivemind cursor install   # Cursor
hivemind hermes install   # Hermes
hivemind pi install       # pi

What Is ScrapeGraphAI

ScrapeGraphAI is the AI web-scraping engine on the other side of this pipeline. You hand it a natural-language prompt and a URL, and it returns clean, structured JSON — no CSS selectors to maintain, no brittle markup to babysit. It runs the full loop in a single call: search the web, scrape a page, extract structured data against a schema, crawl a whole site, and even monitor pages for changes. It ships as an API and SDK, as MCP tools your agent can call directly, and as a drop-in CLI skill. That is exactly what the enrichment skill leans on: one search call that researches a topic across the open web and comes back as structured data, ready to merge into a SKILL.md.

Installing the just-scrape skill

The fastest way to put ScrapeGraphAI in your agent's hands is the just-scrape skill — search, scrape, crawl, extract structured JSON, and monitor page changes from the command line. Add it with one command:

npx skills add https://github.com/ScrapeGraphAI/just-scrape --skill just-scrape

Then confirm it is installed and authenticated:

which just-scrape || npm install -g just-scrape@latest
just-scrape validate
just-scrape credits

Grab an API key from the dashboard if validate asks for one — the free tier covers plenty of runs.

What ScrapeGraphAI Adds To Hivemind

Picture the skill Hivemind just codified after a Next.js debugging session:

---
name: nextjs-server-client-boundary-verification
description: "Catch all server/client boundary violations in Next.js before declaring fixes done by systematically checking data flow through component trees."
source_sessions:
  - 177626fa-90c1-4d83-b130-8aad087c3b5c
version: 1
created_by_agent: claude_code
---
 
## When to Use
After fixing any Next.js server→client prop passing issue, before marking work complete.
 
## Workflow
1. Identify the error component (e.g., PermitsTable, CorridorsTable)
2. Trace the component tree, verify directives and props
3. For each prop: function? data? serializable?
4. Run dev server and check for residual errors before closing the task

Forty-four lines, and every one earns its place. This is Hivemind doing exactly what it is great at: distilling a real debugging session down to the problem your team actually hit and the workflow to catch it again — the high-signal core you only get from real work, captured automatically and ready for the next agent. That focused lesson is the ideal foundation. The enrichment skill keeps every word of it and builds on top with implementation detail sourced from the live web:

• An Error → Cause → Fix lookup table for the 9 most common boundary errors (Only plain objects can be passed..., Event handlers cannot be passed..., Hydration mismatch, etc.)
• Wrong-vs-right code pairs for the actual fixes (Mongoose .lean(), server-action "use server" pragma, arrow-wrap on onClick)
• Serialization rules spelling out what crosses the boundary and what does not
• A tooling row with next lint, next-safe-action, React DevTools
• References to official Next.js, React, and Vercel docs

Here is a real slice of the actual enriched output for that same skill. The whole file is 263 lines now; what follows is roughly the first third so you can see the shape.

First the frontmatter, which carries the enrichment provenance:

---
name: nextjs-server-client-boundary-verification
version: 2
enriched_by: ScrapeGraphAI
enriched_at: 2026-05-26T15:58:00.000Z
enrichment_sources:
  - https://nextjs.org/docs/app/getting-started/server-and-client-components
  - https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions-and-mutations
  - https://react.dev/reference/react/use-server#serializable-parameters-and-return-values
  - https://stackoverflow.com/questions/77091418/warning-only-plain-objects-can-be-passed-to-client-components-from-server-compo
  - https://github.com/vercel/next.js/discussions/46795
  - https://upsun.com/blog/avoid-common-mistakes-with-next-js-app-router/
  - https://www.propelauth.com/post/5-common-pitfalls-with-server-components-in-next13-with-examples
---

The original body is untouched above it. Below it, new sections appear, each suffixed (enriched) so it stays distinct from what Hivemind wrote.

The first new section is the error lookup. The agent can grep this the moment it sees a stack trace:

## Error → Cause → Fix lookup table (enriched)
 
| Error Message | Root Cause | Fix |
|---|---|---|
| Only plain objects can be passed to Client Components from Server Components | Server passing class instance, Mongoose doc, Map, Set, or object with prototype | JSON.parse(JSON.stringify(obj)) OR manually extract primitives |
| Event handlers cannot be passed to Client Component props | Server passing a function (onClick, formatter) to a client component | Add "use client" to the receiving component, or define the handler inside the client component |
| Objects with toJSON methods are not supported | Object has custom toJSON (common with ORMs) | Strip the method via destructuring or JSON.parse(JSON.stringify(obj)) |
| Hydration mismatch: Server rendered markup does not match client rendered markup | Prop changed between server/client render (Date, random ID, Date.now()) | Move the dynamic value generation into the client component via useState(() => ...) or useEffect |
| Passing a server function reference to an onClick handler is not allowed | onClick={serverAction} forwards the event object to the server action | Wrap in arrow: onClick={() => serverAction()} |
| Mongoose document cannot be serialized when passed to a client component | Mongoose docs contain ObjectId, Date, internal methods | Use .lean() on the query, then _id: doc._id.toString() |
| Missing "use server" directive in a server action | File exporting server actions without the pragma | Add "use server"; at the top of the file or function |
| Fetch response returned directly to a client component | Returned raw Response object from fetch | return await response.json() instead of return response |
| API route returns 404 / unexpected behavior | API file named page.ts instead of route.ts | Rename to route.ts |

Then the serialization rules so the agent stops guessing:

## Serialization rules — what CAN cross the boundary (enriched)
 
Allowed (serializable):
- Primitives: string, number, bigint, boolean, undefined, null, globally-registered Symbols
- Plain objects (object literals) with only serializable properties
- Arrays, Map, Set, TypedArray, ArrayBuffer containing serializable values
- Date and FormData instances (Next.js handles these specifically)
- Promise (resolved values must be serializable)
- Server Actions (functions marked with "use server")
 
NOT allowed (will throw):
- Functions (except Server Actions)
- React elements / JSX
- Class instances (Mongoose documents, Prisma models, custom classes)
- Objects with custom prototypes
- Objects with custom toJSON methods
- RegExp, raw Request/Response, Node.js streams
- Symbols NOT registered globally
- Circular references
 
Hidden traps:
- undefined values are silently dropped during serialization, don't rely on them surviving
- ORM result objects look plain but have hidden prototype methods, always .lean() (Mongoose) or destructure (Prisma)

Then a wrong-vs-right code pair for each common fix. Here are two of the eight:

// Pattern 1 — Server action as onClick handler (forward-event bug)
 
// ❌ Wrong — forwards the event object to the server, breaks
<button onClick={revalidateEvents}>Refresh</button>
 
// ✅ Right — wrap in arrow, drop the event arg
<button onClick={() => revalidateEvents()}>Refresh</button>

// Pattern 2 — Mongoose document leak
 
// ❌ Wrong — full Mongoose doc has ObjectId + methods
const user = await UserModel.findById(id);
return <UserCard user={user} />;
 
// ✅ Right — .lean() returns plain obj, stringify _id
const user = await UserModel.findById(id).lean();
user._id = user._id.toString();
return <UserCard user={user} />;

A tooling row so the agent knows which lint/runtime tools catch this class of bug:

## Tooling (enriched)
 
| Tool | Purpose |
|------|---------|
| next lint (built-in) | Runs ESLint with Next.js config, catches missing directives |
| @next/eslint-plugin-next | Server/client boundary checks, illegal imports |
| typescript-plugin-next | TS helpers surfacing server/client contract violations in your editor |
| React DevTools | Inspect component hierarchy and actual props received post-boundary |
| next-safe-action | Library enforcing serializable inputs to server actions, with runtime type checks |

And after these sections come six more: the full set of eight wrong-vs-right patterns, a "use client" decision list, a debugging workflow, a pre-completion checklist, and seven authoritative references. Same skill, now a full playbook: Hivemind's original 44 lines anchor all 263, and every new line traces back to a source listed in the frontmatter. The agent that picks it up next has both the team's lesson and the reference material in one place.

The key idea: Hivemind gives you the tacit knowledge (what your team learned the hard way in a real session), and ScrapeGraphAI adds the explicit knowledge (what the docs and community already document). The enrichment skill merges them in one file.

How To Use It

No infrastructure, no build step. The enricher is itself just a skill file.

1. Connect ScrapeGraphAI as an MCP server. Grab a key from the dashboard (the free tier covers plenty of runs) and wire it into Claude Code. Your agent now has search, scrape, and extract as tools.

2. Drop the skill in place. Save the file below as ~/.claude/skills/enrich-hivemind-skill/SKILL.md.

3. Ask for it. In any Claude Code session, say:

enrich the nextjs-server-client-boundary-verification skill

The agent reads the skill, runs one ScrapeGraphAI search with a topic-shaped schema, dedupes the results, and rewrites the file. It bumps the version, marks every new section (enriched) so it stays distinct from the original, and lists the sources it used.

That is the whole workflow. Point it at any Hivemind skill, and its lesson grows into a full playbook.

The Skill

Copy this into ~/.claude/skills/enrich-hivemind-skill/SKILL.md:

---
name: enrich-hivemind-skill
description: Enrich concise Hivemind-codified skills with real-world web research using ScrapeGraphAI MCP tools. Use when the user asks to "enrich", "deepen", "research-boost", or "improve" a SKILL.md, OR when you encounter a SKILL.md under ~/.claude/skills/ that has Hivemind provenance frontmatter (source_sessions, created_by_agent) but lacks an `enriched_by: ScrapeGraphAI` field and has room to grow (under ~50 lines, no Code Examples section yet, no References section yet). The skill uses mcp__ScrapeGraphAI-mcp__search with a topic-adapted output_schema to research the skill's topic on the open web in a single call, then rewrites the SKILL.md with added Known Gotchas, Code Examples, Best Practices, and References while preserving the original pattern and frontmatter.
version: 2
---
 
# Enrich Hivemind Skill with ScrapeGraphAI
 
You are the enrichment loop between a memory tool (which codifies concise skills
from agent traces) and the open web. Read a concise SKILL.md, research its topic
with ScrapeGraphAI, and expand it with richer, sourced content.
 
## When to fire
- A SKILL.md has provenance frontmatter (source_sessions, created_by_agent)
- It does NOT already have `enriched_by: ScrapeGraphAI`
- The topic is researchable on the open web (a library, tool, or pattern, not a
  generic principle like "verify before claiming")
 
## Pipeline
1. Read the target SKILL.md. Hold the original body in memory verbatim.
2. Write a SPECIFIC research query. Not "Next.js bugs" but "Next.js 15 App
   Router server client component boundary errors serialization".
3. Pick an output_schema shaped to the topic. Always include a floor of
   gotchas, code_patterns, best_practices, tools_and_libraries, key_references.
   Add topic-specific fields:
   - error-debugging skills: error_messages, wrong_vs_right_patterns
   - detection skills: detection_heuristics, diagnostic_workflow
   - API skills: auth_requirements, rate_limits, endpoint_patterns
4. Call mcp__ScrapeGraphAI-mcp__search with user_prompt + prompt + output_schema.
   One call does search + scrape + extract together.
5. If the response is too large for context, it is auto-saved to disk. Pull
   sections out with jq instead of reading the whole file.
6. Dedupe: tools by name, references by url, gotchas by meaning, code by
   language+purpose. Keep the most specific version. Aim to cut 30-50%.
7. Compose the new file:
   - Preserve the original body and frontmatter verbatim.
   - Bump version, add enriched_by: ScrapeGraphAI, enriched_at, enrichment_sources.
   - Append new sections, each suffixed "(enriched)" so they stay distinct.
   - Order by what the reader reaches for first (errors -> lookup table first,
     detection -> heuristics first, API -> endpoints first).
   - Tables for 3+ column data, bullets for flat lists, fenced+language-tagged
     code blocks for every snippet.
8. Quality gate. Refuse to write if the content is generic ("be careful"), the
   code looks hallucinated, all references share one domain, or the extract is
   nearly empty.
9. Atomic write: write SKILL.md.new, then mv it over SKILL.md.
10. Report one line: enriched <name>: +N gotchas, +M code, +K refs, vX->vY.
 
## Non-goals
- Do NOT enrich hand-written skills (no provenance frontmatter).
- Do NOT call ScrapeGraphAI for topics that need no web research.
- Do NOT modify the original body or the source_sessions / created_by_agent fields.

The skill is intentionally model-driven. It tells the agent what to do and trusts it to handle each topic, rather than hard-coding a rigid script. That is why one skill works across scraping, API usage, and framework-debugging topics without changes.

Honest Notes

A few things worth saying plainly, because the tradeoffs are real.

Not every enriched section is equally useful. Code examples are gold — the agent can copy them straight into a fix. But some research comes back written for humans, not agents: "open Chrome DevTools and check the Network tab" is a step a person follows, not one an agent can run. The skill biases its extraction prompt toward executable patterns, which helps, but expect a few inert lines in any given run.

Quality tracks the source pool. Topics with strong official docs (Next.js, Playwright, Mapbox) enrich beautifully. Topics dominated by SEO content farms come back thinner and more generic. The dedupe step and the source-diversity check in the quality gate catch a lot, but garbage in is still garbage out — if the open web doesn't document your topic well, neither will the enriched skill.

Verify the code before you trust it. Enrichment pulls from real sources, but it is still an LLM extracting and reshaping them. A snippet can be subtly outdated, mix two framework versions, or describe an API that changed last month. The frontmatter lists every source for exactly this reason: when a pattern matters, follow the link and confirm it. Treat the enriched skill as a strong first draft, not a verified spec.

It is non-deterministic, and it costs a call. Each enrichment is one ScrapeGraphAI search (and the credits that go with it), and running it twice on the same skill will not return identical text — the search results and the extraction both vary. That is fine for a one-shot upgrade, but it means enrichment is not a build step you can pin and reproduce. Re-run it when the underlying topic actually moves, not on every commit.

Longer is not automatically better. The win is the specific gotchas and the runnable code, not the line count. A 44-line skill that grew to 263 is only an improvement if those extra 219 lines earn their place. The quality gate exists precisely to stop the skill from padding a file with generic filler — and if a run comes back mostly filler, the right move is to throw it away, not commit it.

Try It

If your agents already produce skill files, this layers on top without changing anything else. Connect ScrapeGraphAI, drop the skill in ~/.claude/skills/, and run it against one of your Hivemind skills. See if the deeper version helps your next session. If it does, run it across the rest.

Related Articles

• AI Agents for Web Scraping: A Practical Tutorial - Build agents that scrape and extract structured data
• AI Agent Scraping Tool - How AI agents handle real-world scraping tasks
• The AI Agent Revolution - Where autonomous agents are headed and why memory matters
• AI-Powered Data Extraction - Turning raw web pages into structured insight