Scraping Amazon for Competitive Insights with ScrapeGraphAI

Amazon is not only the world’s largest e-commerce marketplace—it’s also a goldmine of data. By scraping Amazon product listings, businesses can uncover trends, monitor pricing, and gain a competitive edge. In this guide, we explain why scraping Amazon is valuable and show you how to extract data like keyboard names and prices using ScrapeGraphAI. We also provide examples in Python, JavaScript, and via a cURL request.

Why Scrape Amazon?

Scraping Amazon offers numerous strategic benefits:

• Price Monitoring: Track pricing trends over time to stay competitive.
• Competitive Analysis: Discover insights into competitors’ product offerings and pricing strategies.
• Market Research: Learn about consumer preferences by analyzing product names, descriptions, and prices.
• Data-Driven Decisions: Use the extracted data to refine marketing strategies, optimize inventory, and improve product offerings.

Automating data extraction saves valuable time and lets you focus on strategic analysis.

How ScrapeGraphAI Simplifies Amazon Scraping

ScrapeGraphAI streamlines the process by combining advanced scraping techniques with schema validation, ensuring accurate and structured data extraction.

Python Example

python
from scrapegraph_py import Client
from scrapegraph_py.logger import sgai_logger
from pydantic import BaseModel, Field

# Set logging to INFO for detailed output
sgai_logger.set_logging(level="INFO")

# Initialize the client with your API key
sgai_client = Client(api_key="sgai-********************")

# Define the schema for the product listing
class ProductListingSchema(BaseModel):
    KeyboardName: str = Field(..., description="The name of the keyboard")
    Price: float = Field(..., description="The price of the keyboard")

# Send a SmartScraper request to Amazon's search page for keyboards
response = sgai_client.smartscraper(
    website_url="https://www.amazon.it/s?k=keyboards&__mk_it_IT=ÅMÅŽÕÑ&crid=MMF6T1GWDMO6&sprefix=keyboars%2Caps%2C119&ref=nb_sb_noss_2",
    user_prompt="Extract the KeyboardName and Price from the product listings",
    output_schema=ProductListingSchema
)

# Print the response details
print(f"Request ID: {response['request_id']}")
print(f"Result: {response['result']}")

sgai_client.close()

JavaScript Example with Zod

javascript
import { Client } from 'scrapegraph-js';
import { z } from "zod";

// Define the schema using Zod
const productListingSchema = z.object({
  /** The name of the keyboard */
  KeyboardName: z.string(),
  /** The price of the keyboard */
  Price: z.number(),
});

type ProductListingSchema = z.infer<typeof productListingSchema>;

// Initialize the client
const sgai_client = new Client("sgai-********************");

try {
  const response = await sgai_client.smartscraper({
    websiteUrl: "https://www.amazon.it/s?k=keyboards&__mk_it_IT=ÅMÅŽÕÑ&crid=MMF6T1GWDMO6&sprefix=keyboars%2Caps%2C119&ref=nb_sb_noss_2",
    userPrompt: "Extract the KeyboardName and Price from the product listings",
    outputSchema: productListingSchema
  });

  console.log('Request ID:', response.requestId);
  console.log('Result:', response.result);
} catch (error) {
  console.error(error);
} finally {
  sgai_client.close();
}

cURL Example

bash
curl -X 'POST'   'https://api.scrapegraphai.com/v1/smartscraper'   -H 'accept: application/json'   -H 'SGAI-APIKEY: sgai-********************'   -H 'Content-Type: application/json'   -d '{
  "website_url": "https://www.amazon.it/s?k=keyboards&__mk_it_IT=ÅMÅŽÕÑ&crid=MMF6T1GWDMO6&sprefix=keyboars%2Caps%2C119&ref=nb_sb_noss_2",
  "user_prompt": "Extract the KeyboardName and Price from the product listings",
  "output_schema": {
    "type": "object",
    "title": "ProductListingSchema",
    "properties": {
      "KeyboardName": {
        "type": "string",
        "title": "KeyboardName",
        "description": "The name of the keyboard"
      },
      "Price": {
        "type": "number",
        "title": "Price",
        "description": "The price of the keyboard"
      }
    },
    "required": ["KeyboardName", "Price"],
    "$defs": {}
  }
}'

Example Response

Here’s an example of the JSON response you might receive:

json
{
  "KeyboardName": [
    "Logitech MX Keys S",
    "Razer Ornata V3",
    "EPOMAKER x Aula F75",
    "Dierya DK63",
    "8Bitdo Retro Mechanical Keyboard"
  ],
  "Price": [
    91.1,
    32.78,
    24.58,
    32.78,
    81.96
  ]
}

Conclusion

Scraping Amazon with ScrapeGraphAI enables you to harness vast amounts of data for competitive analysis, market research, and strategic decision-making. Whether you choose Python, JavaScript, or a simple cURL request, the process is streamlined and efficient—empowering you to extract structured data that drives actionable insights.

Happy scraping, and may your data fuel your next breakthrough!