This time, I would like to get Mydin Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.
So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.
Steps
1) Go to https://www.mydin.com.my/stores/store-locator
2) Hover to “Find a store near” panel and right click then click “Inspect”
3) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty
4) Refresh your browser and you will see, it is populated with files and their types.
5) Sort by type and look for “xhr” type
Look for ProcessAjaxRequest
XHR is XMLHttpRequest (XHR) is an API in the form of an object whose methods transfer data between a web browser and a web server. It supports XML or JSON data format.
6) You can double click to see full view in the browser and see its full URL
7) Write a code that read the API and parse the JSON data
I’m using NodeJS and save it as CSV file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import rp from 'request-promise'; const createCsvWriter = require('csv-writer').createObjectCsvWriter; (async function() { let records = await rp("https://www.mydin.com.my/base/ajax/ProcessAjaxRequest?action=getAllStores&_=1590477828561"); records = JSON.parse(records); records = records.Response.Stores; const csvWriter = createCsvWriter({ path: '/path-to-save/mydin.csv', header: [ {id: 'StoreExtId', title: 'ID'}, {id: 'StoreName', title: 'Name'}, {id: 'AddressLine1', title: 'Address 1'}, {id: 'AddressLine2', title: 'Address 2'}, {id: 'City', title: 'City'}, {id: 'State', title: 'State'}, {id: 'Latitude', title: 'Latitude'}, {id: 'Longitude', title: 'Longitude'} ] }); await csvWriter.writeRecords(rows); }()); |