Web Scraping – How To Get Hidden Data API that is Embedded in Google Map

kfc outlets location

I would like to get KFC Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://kfc.com.my/find-a-kfc/

2) Click on the outlets to view its details information.

kfc outlets location
kfc outlets location

3) Right click on the outlet’s information box and click Inspect.

kfc outlets inspect
kfc outlets inspect

 

4) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

kfc inspect network blank
kfc inspect network blank

5) Refresh your browser and you will see, it is populated with files and their types.

6) Sort by type and look for “fetch” type

identify data api by looking for fetch type
identify data api by looking for fetch type

7) Click on the file link and new window will appear. Click on “Response” tab.

Find until you see JSON format with outlet information. In KFC case, I found the file name is store?xxxxxxxx (xx denotes numbers)

kfc outlets api response
kfc outlets api response

8) You can also click “Preview” tab to see prettified JSON format.

kfc outlets api preview
kfc outlets api preview

9) After found the file that provides the outlets data, right click on the file and Copy -> Copy link address

get the outlets api by copying link address
get the outlets api by copying link address

10) Paste the link to a new browser tab and you can see the link

  • In KFC case it is, https://kfc.com.my/api/v2/store?1588173941864
  • You will see the response as below
kfc hidden outlets api response
kfc hidden outlets api response

11) Write a code that read the API and parse the JSON data

  • I’m using NodeJS and save it as CSV file.
import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
const records = await rp("https://kfc.com.my/api/v2/store?1588173941864");
const rows = JSON.parse(records);
const csvWriter = createCsvWriter({
    path: '/path-to-save/kfc.csv',
    header: [
        {id: 'id', title: 'ID'},
        {id: 'name', title: 'Name'},
        {id: 'address', title: 'Address'},
        {id: 'phone', title: 'Phone'},
        {id: 'open24h', title: 'Open 24 Hr'},
        {id: 'hasbreakfast', title: 'Has Breakfast'},
        {id: 'hasdrivethru', title: 'Has Drive Thru'},
        {id: 'delivery', title: 'Delivery'},
        {id: 'selfcollect', title: 'Self Collect'},
        {id: 'haswifi', title: 'Has Wifi'},
        {id: 'weekdayopen', title: 'Weekday Open'},
        {id: 'weekdayclose', title: 'Weekday Close'},
        {id: 'weekendopen', title: 'Weekend Open'},
        {id: 'weekendclose', title: 'Weekend Close'},
        {id: 'latitude', title: 'Latitude'},
        {id: 'longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

11) Sample CSV Output

kfc sample csv output
kfc sample csv output
  • From the CSV, KFC Malaysia has 712 outlets as of 29/04/2020.