Mydin Outlets Web Scraping – How To Get Hidden Data API Embedded in Google Map

mydin outlets on google map

This time, I would like to get Mydin Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://www.mydin.com.my/stores/store-locator

2) Hover to “Find a store near” panel and right click then click “Inspect”

mydin outlets location inspect
mydin outlets location inspect

3) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

mydin outlets inspect network empty
mydin outlets inspect network empty

4) Refresh your browser and you will see, it is populated with files and their types.

mydin outlets xhr
mydin outlets xhr

5) Sort by type and look for “xhr” type

mydin outlets api prettified
mydin outlets api prettified

Look for ProcessAjaxRequest

XHR is XMLHttpRequest (XHR) is an API in the form of an object whose methods transfer data between a web browser and a web server. It supports XML or JSON data format.

6) You can double click to see full view in the browser and see its full URL

mydin outlets api full view
mydin outlets api full view

7) Write a code that read the API and parse the JSON data

I’m using NodeJS and save it as CSV file.

import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
let records = await rp("https://www.mydin.com.my/base/ajax/ProcessAjaxRequest?action=getAllStores&_=1590477828561");
records = JSON.parse(records);
records = records.Response.Stores;

const csvWriter = createCsvWriter({
    path: '/path-to-save/mydin.csv',
    header: [
        {id: 'StoreExtId', title: 'ID'},
        {id: 'StoreName', title: 'Name'},
        {id: 'AddressLine1', title: 'Address 1'},
        {id: 'AddressLine2', title: 'Address 2'},
        {id: 'City', title: 'City'},
        {id: 'State', title: 'State'},
        {id: 'Latitude', title: 'Latitude'},
        {id: 'Longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

8) Sample CSV Output

mydin outlets csv
mydin outlets csv

References:

XHR Explanation