Unable to replicate ajax request for scraping

2139 views python

I'm trying to scrape the data from a website but the content is dynamically added. I'm using Python and requests library. Here's the link to the example https://www.goat.com/sneakers/off-white-x-air-presto-aa3830-002

url = 'https://www.goat.com/web-api/graphql'  # the API endpoint

# replicate the header in Chrome debugger tool of the successful request
headers = {  
        ':authority': 'www.goat.com',
        ':method': 'POST',
        ':path': '/web-api/graphql',
        ':scheme': 'https',
        "accept": "application/json",
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "vi-VN,vi;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6,en;q=0.5",
        "content-length": "1153",
        "content-type": "application/json",
        "origin": "https://www.goat.com",
        "referer": "https://www.goat.com/sneakers/off-white-x-air-presto-aa3830-002",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36",

# the request payload from Chrome debugger tool
data = {"query":'{\n    viewer {\n      productTemplate(slug: \"off-white-x-air-presto-aa3830-002\") {\n        id\n        internal_shot\n        details\n        name\n        original_picture_url\n        sku\n        slug\n        story\n        is_active\n        release_date\n        color\n        special_type\n        upper_material\n        lowest_price_cents\n        new_lowest_price_cents\n        can_return\n        brand {\n          id\n          name\n        }\n        size_brand\n        midsole\n        designer\n        nickname\n        silhouette\n        gender\n        formatted_available_sizes_new_v2 {\n          size\n          price_cents\n          box_condition\n          shoe_condition\n        }\n        product_template_additional_pictures {\n          attribution_url\n          original_picture_url\n          source_url\n        }\n        selling_count\n        used_for_sale_count\n        used_lowest_price_cents\n        goat_clean_for_sale_count\n        new_lowest_price_cents\n        with_defect_for_sale_count\n        category\n      }\n    }\n  }',

r = session.post(url, headers=headers, data=data)

The POST request sent to the server returned Internal Server Error (500), however, the browser executes it normally and the data is retrieved.

I feel like I'm missing something, especially in the part query payload. Please tell me why this happens. I'm very much appreciated!

answered question

1 Answer


I finally managed to figure it out. So here's the code that's working:


r = session.post(url, headers=headers, data=data)


r = session.post(url, headers=headers, json=data)

And it works like a charm!

posted this

Have an answer?


Please login first before posting an answer.