Web Scraping – How To Get Hidden Data API that is Embedded in Google Map

I would like to get KFC Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://kfc.com.my/find-a-kfc/

2) Click on the outlets to view its details information.

kfc outlets location
kfc outlets location

3) Right click on the outlet’s information box and click Inspect.

kfc outlets inspect
kfc outlets inspect

 

4) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

kfc inspect network blank
kfc inspect network blank

5) Refresh your browser and you will see, it is populated with files and their types.

6) Sort by type and look for “fetch” type

identify data api by looking for fetch type
identify data api by looking for fetch type

7) Click on the file link and new window will appear. Click on “Response” tab.

Find until you see JSON format with outlet information. In KFC case, I found the file name is store?xxxxxxxx (xx denotes numbers)

kfc outlets api response
kfc outlets api response

8) You can also click “Preview” tab to see prettified JSON format.

kfc outlets api preview
kfc outlets api preview

9) After found the file that provides the outlets data, right click on the file and Copy -> Copy link address

get the outlets api by copying link address
get the outlets api by copying link address

10) Paste the link to a new browser tab and you can see the link

  • In KFC case it is, https://kfc.com.my/api/v2/store?1588173941864
  • You will see the response as below
kfc hidden outlets api response
kfc hidden outlets api response

11) Write a code that read the API and parse the JSON data

  • I’m using NodeJS and save it as CSV file.
import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
const records = await rp("https://kfc.com.my/api/v2/store?1588173941864");
const rows = JSON.parse(records);
const csvWriter = createCsvWriter({
    path: '/path-to-save/kfc.csv',
    header: [
        {id: 'id', title: 'ID'},
        {id: 'name', title: 'Name'},
        {id: 'address', title: 'Address'},
        {id: 'phone', title: 'Phone'},
        {id: 'open24h', title: 'Open 24 Hr'},
        {id: 'hasbreakfast', title: 'Has Breakfast'},
        {id: 'hasdrivethru', title: 'Has Drive Thru'},
        {id: 'delivery', title: 'Delivery'},
        {id: 'selfcollect', title: 'Self Collect'},
        {id: 'haswifi', title: 'Has Wifi'},
        {id: 'weekdayopen', title: 'Weekday Open'},
        {id: 'weekdayclose', title: 'Weekday Close'},
        {id: 'weekendopen', title: 'Weekend Open'},
        {id: 'weekendclose', title: 'Weekend Close'},
        {id: 'latitude', title: 'Latitude'},
        {id: 'longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

11) Sample CSV Output

kfc sample csv output
kfc sample csv output
  • From the CSV, KFC Malaysia has 712 outlets as of 29/04/2020.

QGIS How To Create Hexagon Grid Based on Selangor State Boundary

GIS hexagon grid can be created using QGIS. In this case I’m using QGIS version 2.18.15.

First of all, you must have a boundary layer. I’m using Selangor parliamentary boundary.

QGIS Selangor Parliamentary Boundary
QGIS Selangor Parliamentary Boundary

Steps to Create Hexagon Grid

1) Enable the Processing Toolbox

View -> Panel -> Toolbox

QGIS Enable Toolbox
QGIS Enable Toolbox

2) From the Processing Toolbox, select Create Grid

QGIS Processing Toolbox
QGIS Processing Toolbox

3) After setting the configuration as follows, click run:

  • Grid type: Hexagon
  • Grid extent:  Select canvas and choose Selangor boundary or your boundary layer.
  • QGIS Use Extent Selangor Parliamentary
    QGIS Use Extent Selangor Parliamentary
  • Horizontal / Vertical spacing: 0.025 (means 0.025km)
  • Grid CRS: EPSG 4326 (or popularly known as WGS 84 where Google Map use this CRS)
QGIS Hexagon Grid Processing
QGIS Hexagon Grid Processing

4) After click Run, a hexagon grid layer will appear as follows:

QGIS Hexagon Grid Layer
QGIS Hexagon Grid Layer

5) Select Vector -> Research Tools -> Select by Location

QGIS Select By Location
QGIS Select By Location

6) Input the Configuratoin for Select by Location as follows, then click Run:

  • Layer to select: Your Hexagon Grid Layer
  • Additional layer (intersection layer): Your boundary layer in my case Selangor parliamentary
  • Geometric predicate: choose intersects, contains, touches, overlaps
QGIS Select By Location Configuration
QGIS Select By Location Configuration

7)Then click Select Features using an expression -> Invert Feature Selection

QGIS Invert Feature Selection
QGIS Invert Feature Selection

8)After inversion, your hexagon grid would look like below:

QGIS After inversion
QGIS After inversion

9) Click Toggle Editing (pencil icon) then Delete (trash icon)

QGIS Toggle Editing & Trash
QGIS Toggle Editing & Trash

10) Final output would look like this

QGIS Hexagon Grid
QGIS Hexagon Grid

How To Reduce Al Rajhi Housing Loan Principal

If you have extra money and want to reduce your housing loan principal with Al Rajhi bank, below are the steps.

Steps to Reduce Housing Loan Principal

  1. Send an official letter to Al Rajhi,
    • state the amount to reduce principal
    • state your saving account number with Al Rajhi
    • state your housing loan account number
    • authorize Al Rajhi to deduct the amount from your saving account with Al Rajhi
    • sign the letter
  2. Ensure the amount is in your Al Rajhi bank account
  3. Email the soft copy letter to customersupport@alrajhibank.com.my
  4. Ensure Al Rajhi provide you with acknowledgement
  • There is no minimum amount to reduce the principal
  • It takes 7 working days to process
Notes

This information is verified as of 31st March 2020, when I made a call to Al Rajhi call centre. Better to double confirm with Al Rajhi regarding your housing loan.