Selenium Webdriver How To Increase Time Out for Page Load, Script Load or Element Locator

I couldn’t find at Stackoverflow on how to increase the timeout and the Selenium API not that easy to find on how to use it.

For Selenium WebDriver using NodeJS below is how to increase its time outs. The number is in milliseconds.

 let driver = await new Builder().forBrowser('chrome')
                .setChromeOptions(new chrome.Options().headless())
                .setFirefoxOptions(new firefox.Options().headless())
                .build();

 driver.manage().setTimeouts({implicit: 120000, script: 120000, pageLoad:120000});

From Selenium Documentation

The following timeouts are supported (all timeouts are specified in milliseconds):

  • implicit specifies the maximum amount of time to wait for an element locator to succeed when locating elements on the page. Defaults to 0 milliseconds.
  • pageLoad specifies the maximum amount of time to wait for a page to finishing loading. Defaults to 300000 milliseconds.
  • script specifies the maximum amount of time to wait for an evaluated script to run. If set to null, the script timeout will be indefinite. Defaults to 30000 milliseconds.

Reference:
https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/lib/webdriver_exports_Options.html

Selenium User Agent vs Normal Browser User Agent

I noticed lots of websites now implement anti robot from website scraping. Most of them using fingerprintjs.com technology where it uses browser users agent as part to detect whether it is a robot or human.

I use Selenium Webdriver to scrap few websites. When I browse the website, it shows the content but if using Selenium it shows empty content.

You can test the fingerprintjs here to check whether it detects human or robot.

If using Selenium for Chrome and Firefox , fingerprintjs will detect it as robots. However if using Selenium Safari, it detects Selenium as human.

fingerprint detect selenium for safari as human
fingerprint detect selenium for safari as human

Safari when launch by default is in incognito mode so maybe harder for fingerprintjs.com detect it.

Below are comparison between Selenium for Chrome, Firefox and Safari browsers user agent.

Legend:

Yellow colored rows means there are differences between Selenium and normal browsers user agent.

Chrome Browser

NoChrome - Normal Browser User AgentChrome - Selenium User Agent
1Host: localhostHost: localhost
2Connection: keep-aliveConnection: keep-alive
3Cache-Control: max-age=0
4Upgrade-Insecure-Requests: 1Upgrade-Insecure-Requests: 1
5User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
6Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
7Sec-Fetch-Site: noneSec-Fetch-Site: none
8Sec-Fetch-Mode: navigateSec-Fetch-Mode: navigate
9Sec-Fetch-User: ?1Sec-Fetch-User: ?1
10Sec-Fetch-Dest: documentSec-Fetch-Dest: document
11Accept-Encoding: gzip, deflate, brAccept-Encoding: gzip, deflate, br
12Accept-Language: en-US,en;q=0.9,id;q=0.8,ms;q=0.7,fr;q=0.6Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
13Cookie: chat_uuid=1458159213.1598025760; gr_session=17411c1fd31-2125f7fe5bbd95cd; gr_reco=17411c1fd70-baa36285123101b1; utag_main=v_id:017411c1fb650017fec4094270bb03079001a07100942$_sn:4$_ss:1$_st:1598096705870$_pn:1%3Bexp-session$ses_id:1598094905870%3Bexp-session

Firefox Browser

NoFirefox - Normal Browser User AgentFirefox - Selenium User Agent
1Host: localhostHost: localhost
2User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko/20100101 Firefox/79.0User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko/20100101 Firefox/79.0
3Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
4Accept-Language: en-US,en;q=0.5Accept-Language: en-US,en;q=0.5
5Accept-Encoding: gzip, deflateAccept-Encoding: gzip, deflate
6Connection: keep-aliveConnection: keep-alive
7Cookie: _ga=GA1.1.1739912853.1548032065; gr_reco=1688e9dab95-b5d644815084ad40; utag_main=v_id:01688e9dad18001ff21f0bc7b4a800052009100f0093c$_sn:7$_ss:1$_st:1594889778490$_pn:1%3Bexp-session$ses_id:1594887978490%3Bexp-session; uuid=092deff09bc14cec8f32915260380536; cto_lwid=3ff13ed0-ccf0-46a0-b3d7-8a2e1cadcb3f; cto_idcpy=752176f4-9133-4eb6-a23d-8aacb880eaee; __gads=ID=258eba7044613b5a-22a30c3f84c20074:T=1594886702:S=ALNI_MYCzBeDMjB7doBDh_F5BZztWvpLvw; chat_uuid=407415995.1568209940; cto_bundle=aJ5M6V9JZCUyRjlLVVRRMG5ucUg0RUMlMkJtYk53UXpDJTJGY0Z3b09EU1RTZHpyY0xER1o0ektySUZZdU9iQ3d5UWJsOXFqUFZ4UDlZaGF4WmR4eThta2Q2S2o1NTVFd2Ruc1lBcVNJVW1IQ1ZlN1hYWTZ1VGpCZXJIZjRnRnhnZ0lEc3VuTkg3dCUyQmtTc0ZkajQ2WnJ6JTJGR0w5T3ElMkJEa1NlbjNCZlZzc2k0eVJYVU5oOEtPTjAlM0Q; _ga_SDQJRXDGN4=GS1.1.1596669834.9.0.1596669837.0; __atuvc=0%7C28%2C1%7C29%2C0%7C30%2C0%7C31%2C1%7C32; _fbp=fb.0.1586054074356.121762323; mudahHash=ec9f3f6655e4ccc0814a616dd3ee38f75d7ce494; cto_bidid=wxF3T19jb2hMYzJ1V0dITEZHRjRxMURWMWZrSTNmaDIlMkZ3RzBxTllBczB0amZMMWFOTDg2JTJGdU8xMTQzWDNzMm9TRzh0cmJHUEZFRTR0aUJWWDZQbHA1ZFhUd042cDMlMkIxdkdpOEt6S3ZqWlZ6YUNySSUzRA; _gcl_au=1.1.919768330.1594886698; gr_session=17356a7b66b-a9389a87a3e2a370
8Upgrade-Insecure-Requests: 1 Upgrade-Insecure-Requests: 1

Safari Browser

NoSafari - Normal Browser User AgentSafari - Selenium User Agent
1Host: localhost Host: localhost
2Upgrade-Insecure-Requests: 1 Upgrade-Insecure-Requests: 1
3Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
4User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15
5Accept-Language: en-us Accept-Language: en-us
6Accept-Encoding: gzip, deflate Accept-Encoding: gzip, deflate
7Connection: keep-alive Connection: keep-alive

Conclusion

Most number of differences found in Chrome and followed by Firefox. Safari doesn’t show difference between Selenium and normal browser user agent.

Does it means, it is much easier to trick anti robot using Safari?

ChromeFirefoxSafari
310

How To Convert Javascript to TypeScript

Previously I use Babel Javascript to convert my javascript to ECMA2015. I like to program in object oriented because I learned my first programming language in C/C++ and Java.

Doing procedural language like C was much easier vs object oriented Java. But in the long run, I noticed that coding in object oriented is much easier to maintain.

Steps to Convert Javascript to TypeScript

Install TypeScript

npm install -g typescript

-g to make it global so all your JavaScript programs can be migrated to TypeScript.

Create tsconfig.build.json, tsconfig.json and tslint.json

At your root project folder run the below command.

tsc --init

I edited my tsconfig.json as follows:

{
  "compilerOptions": {
    "target": "es2017",
    "module": "commonjs",
    "lib": [ "dom", "dom.iterable", "esnext"],
    "rootDir": ".",
    "moduleResolution": "node",
    "resolveJsonModule": true,

    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "types": ["node", "mocha", "chai"],
    "typeRoots": ["node_modules/@types", "src/types"],
    "jsx": "react"    
  },
  "exclude": ["node_modules", "dist", "src/admin"],

  "include": [ "src/**/*.ts", "src/**/*.json"]
}

I added manually tsconfig.build.json and tslint.json

{
  "extends": "./tsconfig.json",
  "exclude": ["node_modules", "test", "dist", "**/*spec.ts"]
}

tslint.json

{
  "defaultSeverity": "error",
  "extends": [
    "tslint:recommended"
  ],
  "jsRules": {
    "no-unused-expression": true
  },
  "rules": {
    "quotemark": [
      true,
      "double"
    ],
    "member-access": [
      false
    ],
    "ordered-imports": [
      false
    ],
    "max-line-length": [
      true,
      150
    ],
    "member-ordering": [
      false
    ],
    "interface-name": [
      false
    ],
    "arrow-parens": false,
    "object-literal-sort-keys": false
  },
  "rulesDirectory": []
}

Change Your Existing Files Extension from .js to .ts

typescript change all js to ts extension
typescript change all js to ts extension
typescript change all js to ts extension
typescript change all js to ts extension

Amend Coding T0 Follows TypeScript

Below are few examples TypeScript coding standard that you must follow.

#1 - require to import
const Humanoid = require("./humanoid/index.js");

//change require to import & remove file extension
import {Humanoid} from "./humanoid/index";

#2 - add export in front of class and remove module.exports
export class Humanoid extends HumanoidReqHandler

//remove all line that have module.exports
module.exports = Humanoid; //delete this line

#3 - declare all global variables inside a class as class variables
this.taskName = "something";

//then can call this.taskName inside your class after declare the variable name
protected taskName: string;

//#4 - declare abstract method by adding keyword abstract
//example of javascript - abstract method
if(this._execute === undefined){
	throw new Error("Abstract method _execute()");
}

//put abstract in front of class keyword
abstract class Task{
    constructor(){}
    abstract _execute() : any;
}

//#5 - use import * as something from 'somewhere'
//if error class name .default is not a function decla
//example
import m from 'moment';
TypeError: moment_1.default is not a function

//change to
import * as m from 'moment';

If you Find below error, add return true at main async function.

typescript error promise contructor
typescript error promise contructor
(async function() {
  console.log('start program');
  console.log('end program');
  return true; //add return true to solve error
}());

Besides, you can use TypeScript linter @ tslint (that was set up early) to fix any coding that doesn’t follow TypeScript standard. TypeScript standard is more strict vs JavaScript.

Example of tslint verbose message:

typescript tslint
typescript tslint

Run Unit Test

I use mocha to do unit test.

npm install --save-dev @types/mocha

At package.json, add new script command

"scripts":{
... //other commands
"test-filename": "mocha -r ts-node/register"
},

Example if you want to run one individual unit test.

npm run test-filename ./-- -g "./src/lib/utils/date-timestamp.test.ts"

If you found Error Cannot find module ‘ts-node/register’

Solution: save ts-node locally as develepment dependency.

npm install ts-node --save-dev

Run TypeScript Application

At the terminal or console, type

ts-node path-to-your-main-file

//example
ts-node ./src/main.ts

Conclusion

Even though it takes time to convert from babel javascript into typescript but in the long run it is easier to understand the code and easier to maintain the code.

Besides, you just need to install ts-node and its linter is very good in displaying possible errors.

Benefits of using TypeScript

  • Easy debugging – Typescript shows directly line that throws an error
  • Strong type – any mismatch of variable will be highlighted by ts-lint
  • Less transpilation time – no need to compile like babel. If you have lots of .js files, it takes time to transpile it
  • Support better object oriented – it supports abstract, protected, private methods and variables.

 

Mydin Outlets Web Scraping – How To Get Hidden Data API Embedded in Google Map

This time, I would like to get Mydin Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://www.mydin.com.my/stores/store-locator

2) Hover to “Find a store near” panel and right click then click “Inspect”

mydin outlets location inspect
mydin outlets location inspect

3) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

mydin outlets inspect network empty
mydin outlets inspect network empty

4) Refresh your browser and you will see, it is populated with files and their types.

mydin outlets xhr
mydin outlets xhr

5) Sort by type and look for “xhr” type

mydin outlets api prettified
mydin outlets api prettified

Look for ProcessAjaxRequest

XHR is XMLHttpRequest (XHR) is an API in the form of an object whose methods transfer data between a web browser and a web server. It supports XML or JSON data format.

6) You can double click to see full view in the browser and see its full URL

mydin outlets api full view
mydin outlets api full view

7) Write a code that read the API and parse the JSON data

I’m using NodeJS and save it as CSV file.

import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
let records = await rp("https://www.mydin.com.my/base/ajax/ProcessAjaxRequest?action=getAllStores&_=1590477828561");
records = JSON.parse(records);
records = records.Response.Stores;

const csvWriter = createCsvWriter({
    path: '/path-to-save/mydin.csv',
    header: [
        {id: 'StoreExtId', title: 'ID'},
        {id: 'StoreName', title: 'Name'},
        {id: 'AddressLine1', title: 'Address 1'},
        {id: 'AddressLine2', title: 'Address 2'},
        {id: 'City', title: 'City'},
        {id: 'State', title: 'State'},
        {id: 'Latitude', title: 'Latitude'},
        {id: 'Longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

8) Sample CSV Output

mydin outlets csv
mydin outlets csv

References:

XHR Explanation

NodeJS Reading and Writing to WordPress PHP via Rest API

I developed NodeJS program that can read and write to WordPress Rest API. I’m having problem of reading or writing properly the data in NodeJS and WordPress and vice versa.

NodeJS Writes JSON Objects to WordPress Rest API

Any JSON String[] or Object[], will be saved into serialized PHP string by WordPress.

Surprisingly, when NodeJS read the data save in serialize string, NodeJS can read it as JSON object without needs to call JSON.parse().

However JSON Object, will be saved as string by WordPress but still follows JSON string format.

As the JSON Object is saved as string, when NodeJS read the data via WordPress Rest API, NodeJS has to call JSON.parse() to convert it back into JSON Object.

Examples of JSON Object saved by WordPress Rest API

NodeJS JSON String[]: 
[ '019-3312 493' ]

Saved in Wordress PHP MySQL
as PHP Serialize string: a:1:{i:0;s:12:"019-3312 493";}

NodeJS JSON Object[]: 
[ { day: 'monday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'tuesday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'wednesday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'thursday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'friday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'saturday', open: '8:00 AM', close: '10:00 PM' },
  { day: 'sunday', open: '8:00 AM', close: '10:00 PM' } ]

Saved as string that follows JSON Object format in WordPress PHP MySQL

 a:7:{i:0;a:3:{s:3:"day";s:6:"monday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:1;a:3:{s:3:"day";s:7:"tuesday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:2;a:3:{s:3:"day";s:9:"wednesday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:3;a:3:{s:3:"day";s:8:"thursday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:4;a:3:{s:3:"day";s:6:"friday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:5;a:3:{s:3:"day";s:8:"saturday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}i:6;a:3:{s:3:"day";s:6:"sunday";s:4:"open";s:7:"8:00 AM";s:5:"close";s:8:"10:00 PM";}}


NodeJS JSON Object:
  { open24h: true,
    hasbreakfast: true,
    hasdrivethru: false,
    delivery: true,
    selfcollect: true,
    haswifi: true }

Saved as string that follows JSON Object format in WordPress PHP MySQL
{"open24h":true,"hasbreakfast":true,"hasdrivethru":false,"delivery":true,"selfcollect":true,"haswifi":true}

How To Have Consistency?

By having JSON arrays save as PHP serialize string and JSON object as string in PHP, I need to have do extra checking or conversion by calling JSON.parse()

Another possible solution is to call JSON.stringify() to JSON arrays so it is saved as string at PHP. However, it is still extra step that we need to do and it doesn’t solve the problem stated above where there is no consistency.

So, in short there is no possible solution to have consistency, we still need to do extra step as mentioned above.

PHP Serialize String and JSON Object

I’m not really understand difference between PHP serialize string and JSON object.

I notice PHP when stored data inside database is in PHP serialize string format. Serialize string is only available in PHP, hence it is not interoperable with NodeJS.

Hence, I did simple comparison for better understanding in order to make my NodeJS can read and write PHP data and vice versa.

PHP Array Conversion to PHP Serialize String and JSON Object

//convert PHP array to serialize string and JSON object
$array = array( '1' => 'elem 1', '2'=> 'elem 2', '3'=>' elem 3');


$serialized = serialize($array);
print_r($serialized);
//expected output - serialize string: a:3:{i:1;s:6:"elem 1";i:2;s:6:"elem 2";i:3;s:7:" elem 3";}


$json = json_encode($array);
print_r($json);
//expected output - JSON object: {"1":"elem 1","2":"elem 2","3":" elem 3"}

//convert back to PHP array

$unserialized = unserialize($serialized);
print_r($unserialized);
//expected output - unserialized: Array ( [1] => elem 1 [2] => elem 2 [3] => elem 3 )

$decoded1 = json_decode($json);
print_r($decoded1);
//expected output - decoded into PHP object: stdClass Object ( [1] => elem 1 [2] => elem 2 [3] => elem 3 )


$decoded2 = json_decode($json, true);
print_r($decoded2);
//expected output - decoded into PHP array: Array ( [1] => elem 1 [2] => elem 2 [3] => elem 3 )

PHP Object Conversion to PHP Serialize String and JSON Object

$object = new stdClass();
$object->name = 'Here we go';
$object->message = 'Hello world';


$serialized = serialize($object);
print_r($serialized);
//expected output - serialize string: O:8:"stdClass":2:{s:4:"name";s:10:"Here we go";s:7:"message";s:11:"Hello world";}
 

$json = json_encode($object);
print_r($json);
//expected output - json object: {"name":"Here we go","message":"Hello world"}

$unserialized = unserialize($serialized);
print_r($unserialized);
//expected output - unserialized: stdClass Object ( [name] => Here we go [message] => Hello world )

$decoded = json_decode($json);
print_r($decoded);
//expected output - decoded JSON: stdClass Object ( [name] => Here we go [message] => Hello world )

$decoded2 = json_decode($json, true);
print_r($decoded2);
//expected output - decoded JSON: Array ( [name] => Here we go [message] => Hello world )

PHP Array of Object Conversion to PHP Serialize String and JSON Object

$object1 = new stdClass();
$object1->color = 'blue';
$object1->type = 'suv';

$object2 = new stdClass();
$object2->color = 'white';
$object2->type = 'mpv';

$arrObjects = array($object1, $object2);


$serialized = serialize($arrObjects);
print_r($serialized);
//expected output - serialize: a:2:{i:0;O:8:"stdClass":2:{s:5:"color";s:4:"blue";s:4:"type";s:3:"suv";}i:1;O:8:"stdClass":2:{s:5:"color";s:5:"white";s:4:"type";s:3:"mpv";}}


$json = json_encode($arrObjects);
print_r($json);
//expected output - JSON: [{"color":"blue","type":"suv"},{"color":"white","type":"mpv"}]
 

$unserialized = unserialize($serialized);
print_r($unserialized);
//expected output - unserialize: Array ( [0] => stdClass Object ( [color] => blue [type] => suv ) [1] => stdClass Object ( [color] => white [type] => mpv ) )


$decoded = json_decode($json);
print_r($decoded);
//expected output - decoded: Array ( [0] => stdClass Object ( [color] => blue [type] => suv ) [1] => stdClass Object ( [color] => white [type] => mpv ) )


$decoded2 = json_decode($json, true);
print_r($decoded2);
//expected output - decoded with true: Array ( [0] => Array ( [color] => blue [type] => suv ) [1] => Array ( [color] => white [type] => mpv ) )

Web Scraping – How To Get Hidden Data API that is Embedded in Google Map

I would like to get KFC Malaysia outlets that are located on top of Google Map. I tried using a scraper tool but it always gives me empty result.

So I have to find the hidden outlets API to retrieve it. I’m using Google Chrome to find the hidden data API.

Steps

1) Go to https://kfc.com.my/find-a-kfc/

2) Click on the outlets to view its details information.

kfc outlets location
kfc outlets location

3) Right click on the outlet’s information box and click Inspect.

kfc outlets inspect
kfc outlets inspect

 

4) You will see new window at bottom or right panel of your Chrome browser. Then click Network tab. The tab will be empty

kfc inspect network blank
kfc inspect network blank

5) Refresh your browser and you will see, it is populated with files and their types.

6) Sort by type and look for “fetch” type

identify data api by looking for fetch type
identify data api by looking for fetch type

7) Click on the file link and new window will appear. Click on “Response” tab.

Find until you see JSON format with outlet information. In KFC case, I found the file name is store?xxxxxxxx (xx denotes numbers)

kfc outlets api response
kfc outlets api response

8) You can also click “Preview” tab to see prettified JSON format.

kfc outlets api preview
kfc outlets api preview

9) After found the file that provides the outlets data, right click on the file and Copy -> Copy link address

get the outlets api by copying link address
get the outlets api by copying link address

10) Paste the link to a new browser tab and you can see the link

  • In KFC case it is, https://kfc.com.my/api/v2/store?1588173941864
  • You will see the response as below
kfc hidden outlets api response
kfc hidden outlets api response

11) Write a code that read the API and parse the JSON data

  • I’m using NodeJS and save it as CSV file.
import rp from 'request-promise';
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

(async function() {
const records = await rp("https://kfc.com.my/api/v2/store?1588173941864");
const rows = JSON.parse(records);
const csvWriter = createCsvWriter({
    path: '/path-to-save/kfc.csv',
    header: [
        {id: 'id', title: 'ID'},
        {id: 'name', title: 'Name'},
        {id: 'address', title: 'Address'},
        {id: 'phone', title: 'Phone'},
        {id: 'open24h', title: 'Open 24 Hr'},
        {id: 'hasbreakfast', title: 'Has Breakfast'},
        {id: 'hasdrivethru', title: 'Has Drive Thru'},
        {id: 'delivery', title: 'Delivery'},
        {id: 'selfcollect', title: 'Self Collect'},
        {id: 'haswifi', title: 'Has Wifi'},
        {id: 'weekdayopen', title: 'Weekday Open'},
        {id: 'weekdayclose', title: 'Weekday Close'},
        {id: 'weekendopen', title: 'Weekend Open'},
        {id: 'weekendclose', title: 'Weekend Close'},
        {id: 'latitude', title: 'Latitude'},
        {id: 'longitude', title: 'Longitude'}
    ]
});

  await csvWriter.writeRecords(rows);
}());

11) Sample CSV Output

kfc sample csv output
kfc sample csv output
  • From the CSV, KFC Malaysia has 712 outlets as of 29/04/2020.

php.ini File Location at MacOS Catalina

Disable PHP Warning Message

I wanted to disable my warning message from my php installation. To do so

I need to configure my php.ini file. I searched on the internet but couldn’t find a direct answers where to change my php.ini on MacOS Catalina.

php warning appears
php warning appears

Location of php.ini File at MacOS Catalina

For MacOS Catalina the location of php.ini is located at /etc/

You can verify it by printing phpinfo() and look for “Loaded Configuration File (php.ini) path”

phpinfo location of php.ini
phpinfo location of php.ini

For my MacOS Catalina, the location of php.ini is located at /etc/. As you can see there is no php.ini is loaded (None).

Load Your Own php.ini File

To load your own php.ini, copy the default file into your new php.ini

sudo cp /etc/php.ini.default /etc/php.ini

Open the php.ini, in this case I’m using atom to open it.

sudo atom /etc/php.ini

Then change php error reporting into

error_reporting = E_ERROR

Save it then restart apache.

sudo apachectl restart

Then check the php.ini file is loaded correctly by printing the phpinfo. It will shows /etc/php.ini at “Loaded Configuration File” row.

phpinfo loaded php ini at macos catalina
phpinfo loaded php ini at macos catalina

or you can run php –ini to see the loaded configuration file.

macos php ini command
macos php ini command

When you refresh the website, the warning php message will disappear.

php warning message disappear
php warning message disappear

What is Python Numpy Array Dimension or Axis?

I’m beginner in Python & Numpy. Most tutorials I found seems for expert without really explaining the basic of it.

Even understanding what axis represents in Numpy array is difficult.

I have to read few tutorials and try it out myself before really understand it.

I will update it along with my growing knowledge.

1. Numpy Array Properties

1.1 Dimension

Important to know dimension because when to do concatenation, it will use axis or array dimension.

python array and axis - source oreilly
python array and axis – source oreilly

Row – in Numpy it is called axis 0

Columns – in Numpy it is called axis 1

Depth – in Numpy it is called axis 2

Python Example

import numpy as np

# Array with 1 dimension
A = np.array([1])
B = np.array([1,2])

print("A: ", A)
print("A dimensions: ", A.ndim)

print("B: ", B)
print("B dimensions: ", B.ndim)

# Array with 2 dimensions

C = np.array([[1,2], [3,4], [5,6]])

print("C: ", C)
print("C dimensions: ", C.ndim)

# Array with 3 dimensions

D = np.array([[[1,2], [3,4], [5,6]]])
print("D: ", D)
print("D dimensions: ", D.ndim)

Output

A:  [1]
A dimensions:  1
B:  [1 2]
B dimensions:  1
C:  [[1 2]
 [3 4]
 [5 6]]
C dimensions:  2
D:  [[[1 2]
  [3 4]
  [5 6]]]
D dimensions:  3

Snippet

References

https://www.datacamp.com/community/tutorials/python-numpy-tutorial
https://www.oreilly.com/library/view/elegant-scipy/9781491922927/ch01.html

WebDriverIO Version 5 vs Version 4 Differences

I used webdriverIO library in my programming. Recently I upgraded from version 4 to version 5. To my horror so many breaking changes and not much explanation on the internet. So I put some difference on it.

WebDriveIO – Version 4

  const options = {
    desiredCapabilities: {
        browserName: 'firefox'
    }
  };

  const browser = webdriverio.remote(options);
  await browser.init();

  const select = "a";
  const attribute = "href";

  let results = await browser.getAttribute(selector, attribute);
  results = await browser.getHTML(selector);

WebDriverIO – Version 5

const options = {
    capabilities: {
        browserName: 'firefox'
    }
  };

  const browser = await webdriverio.remote(options);


  const select = "a";
  const attribute = "href";

  let elements = await browser.$(selector);
  let results = await elements.getAttribute(attribute);

  elements = await browser.$(selector);
  results = await elements.getHTML(selector);

The Difference

  1.  In version 5, options desiredCapabilities change to capabilities
  2.  In version 5, need to put await in remote and no need to declare init() anymore as it is deprecated.
  3. In version 5, use $(selector) to select elements before getting elements content or attribute.

Reference

Breaking Change WebDriverIO from Version 4 to Version 5

WebDriver version 5 Release Announcement