Mydin Outlets Web Scraping – How To Get Hidden Data API Embedded in Google Map

This time, I would like to get Mydin Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://www.mydin.com.my/stores/store-locator

2) Hover to “Find a store near” panel and right click then click “Inspect”

mydin outlets location inspect
mydin outlets location inspect

3) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

mydin outlets inspect network empty
mydin outlets inspect network empty

4) Refresh your browser and you will see, it is populated with files and their types.

mydin outlets xhr
mydin outlets xhr

5) Sort by type and look for “xhr” type

mydin outlets api prettified
mydin outlets api prettified

Look for ProcessAjaxRequest

XHR is XMLHttpRequest (XHR) is an API in the form of an object whose methods transfer data between a web browser and a web server. It supports XML or JSON data format.

6) You can double click to see full view in the browser and see its full URL

mydin outlets api full view
mydin outlets api full view

7) Write a code that read the API and parse the JSON data

I’m using NodeJS and save it as CSV file.

import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
let records = await rp("https://www.mydin.com.my/base/ajax/ProcessAjaxRequest?action=getAllStores&_=1590477828561");
records = JSON.parse(records);
records = records.Response.Stores;

const csvWriter = createCsvWriter({
    path: '/path-to-save/mydin.csv',
    header: [
        {id: 'StoreExtId', title: 'ID'},
        {id: 'StoreName', title: 'Name'},
        {id: 'AddressLine1', title: 'Address 1'},
        {id: 'AddressLine2', title: 'Address 2'},
        {id: 'City', title: 'City'},
        {id: 'State', title: 'State'},
        {id: 'Latitude', title: 'Latitude'},
        {id: 'Longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

8) Sample CSV Output

mydin outlets csv
mydin outlets csv

References:

XHR Explanation

Web Scraping – How To Get Hidden Data API that is Embedded in Google Map

I would like to get KFC Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://kfc.com.my/find-a-kfc/

2) Click on the outlets to view its details information.

kfc outlets location
kfc outlets location

3) Right click on the outlet’s information box and click Inspect.

kfc outlets inspect
kfc outlets inspect

 

4) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

kfc inspect network blank
kfc inspect network blank

5) Refresh your browser and you will see, it is populated with files and their types.

6) Sort by type and look for “fetch” type

identify data api by looking for fetch type
identify data api by looking for fetch type

7) Click on the file link and new window will appear. Click on “Response” tab.

Find until you see JSON format with outlet information. In KFC case, I found the file name is store?xxxxxxxx (xx denotes numbers)

kfc outlets api response
kfc outlets api response

8) You can also click “Preview” tab to see prettified JSON format.

kfc outlets api preview
kfc outlets api preview

9) After found the file that provides the outlets data, right click on the file and Copy -> Copy link address

get the outlets api by copying link address
get the outlets api by copying link address

10) Paste the link to a new browser tab and you can see the link

  • In KFC case it is, https://kfc.com.my/api/v2/store?1588173941864
  • You will see the response as below
kfc hidden outlets api response
kfc hidden outlets api response

11) Write a code that read the API and parse the JSON data

  • I’m using NodeJS and save it as CSV file.
import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
const records = await rp("https://kfc.com.my/api/v2/store?1588173941864");
const rows = JSON.parse(records);
const csvWriter = createCsvWriter({
    path: '/path-to-save/kfc.csv',
    header: [
        {id: 'id', title: 'ID'},
        {id: 'name', title: 'Name'},
        {id: 'address', title: 'Address'},
        {id: 'phone', title: 'Phone'},
        {id: 'open24h', title: 'Open 24 Hr'},
        {id: 'hasbreakfast', title: 'Has Breakfast'},
        {id: 'hasdrivethru', title: 'Has Drive Thru'},
        {id: 'delivery', title: 'Delivery'},
        {id: 'selfcollect', title: 'Self Collect'},
        {id: 'haswifi', title: 'Has Wifi'},
        {id: 'weekdayopen', title: 'Weekday Open'},
        {id: 'weekdayclose', title: 'Weekday Close'},
        {id: 'weekendopen', title: 'Weekend Open'},
        {id: 'weekendclose', title: 'Weekend Close'},
        {id: 'latitude', title: 'Latitude'},
        {id: 'longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

11) Sample CSV Output

kfc sample csv output
kfc sample csv output
  • From the CSV, KFC Malaysia has 712 outlets as of 29/04/2020.