Programming – My Awesome Moments

What is Python Numpy Array Dimension or Axis?

I’m beginner in Python & Numpy. Most tutorials I found seems for expert without really explaining the basic of it.

Even understanding what axis represents in Numpy array is difficult.

I have to read few tutorials and try it out myself before really understand it.

I will update it along with my growing knowledge.

1. Numpy Array Properties

1.1 Dimension

Important to know dimension because when to do concatenation, it will use axis or array dimension.

python array and axis - source oreilly — python array and axis – source oreilly

Row – in Numpy it is called axis 0

Columns – in Numpy it is called axis 1

Depth – in Numpy it is called axis 2

Python Example

import numpy as np

# Array with 1 dimension
A = np.array([1])
B = np.array([1,2])

print("A: ", A)
print("A dimensions: ", A.ndim)

print("B: ", B)
print("B dimensions: ", B.ndim)

# Array with 2 dimensions

C = np.array([[1,2], [3,4], [5,6]])

print("C: ", C)
print("C dimensions: ", C.ndim)

# Array with 3 dimensions

D = np.array([[[1,2], [3,4], [5,6]]])
print("D: ", D)
print("D dimensions: ", D.ndim)

Output

A:  [1]
A dimensions:  1
B:  [1 2]
B dimensions:  1
C:  [[1 2]
 [3 4]
 [5 6]]
C dimensions:  2
D:  [[[1 2]
  [3 4]
  [5 6]]]
D dimensions:  3

Snippet

References

https://www.datacamp.com/community/tutorials/python-numpy-tutorial
https://www.oreilly.com/library/view/elegant-scipy/9781491922927/ch01.html

WebDriverIO Version 5 vs Version 4 Differences

I used webdriverIO library in my programming. Recently I upgraded from version 4 to version 5. To my horror so many breaking changes and not much explanation on the internet. So I put some difference on it.

WebDriveIO – Version 4

  const options = {
    desiredCapabilities: {
        browserName: 'firefox'
    }
  };

  const browser = webdriverio.remote(options);
  await browser.init();

  const select = "a";
  const attribute = "href";

  let results = await browser.getAttribute(selector, attribute);
  results = await browser.getHTML(selector);

WebDriverIO – Version 5

const options = {
    capabilities: {
        browserName: 'firefox'
    }
  };

  const browser = await webdriverio.remote(options);


  const select = "a";
  const attribute = "href";

  let elements = await browser.$(selector);
  let results = await elements.getAttribute(attribute);

  elements = await browser.$(selector);
  results = await elements.getHTML(selector);

The Difference

In version 5, options desiredCapabilities change to capabilities
In version 5, need to put await in remote and no need to declare init() anymore as it is deprecated.
In version 5, use $(selector) to select elements before getting elements content or attribute.

Reference

Breaking Change WebDriverIO from Version 4 to Version 5

WebDriver version 5 Release Announcement

Node Red Set POST Parameters for HTTP Request

It is easy to set POST parameters for HTTP request.

The Flow Diagram

node red set post parameters for http request flow diagram

Steps To Set Get Parameters

1) Inject Node

– Payload – can set to anything. In my case I use timestamp

2) Function Node

– Set POST parameters value here
– msg.headers must set the content type.
– All the POST parameters are set inside msg.payload and the parameter names must correspond to RESTFUL API that you query.

msg.headers={ 
    'Content-Type': 'application/x-www-form-urlencoded'
};
msg.payload = {};
msg.payload={ 
    'ID': 1159
};

node red - function node setting post parameters for http request — node red – function node setting post parameters for http request

3) HTTP Request Node

– Choose method: POST
– Set the correspondent RESTFUL endpoint.

node red - http request node properties — node red – http request node properties

4) Debug Node

– Use to display the output

Notes:

Node Red version used – v0.20.7

Node Red Pass GET Parameters for HTTP Request

It is easy to set GET parameters for HTTP request. You need to use the mustache brackets inside the URL itself.

The Flow Diagram

Steps To Set Get Parameters

1) Inject Node

– Payload – can set to anything. In my case I use timestamp

2) Function Node

– Set parameter GET value here
– In my case I use parameter name “path” and “limit”, set as msg.path and msg.limit
– You can use any name for your parameter name but it be the same as in HTTP Request Node

node red function node set get http request parameters

3) HTTP Request Node

– Choose method: GET
– In the URL, use mustache brackets to declare your variable {{{path}}}
– The parameter name must be same as in the Function Node

4) Debug Node

– Use to display the output

node red debug output with GET parameter value

Notes:

Node Red version used – v0.20.7

Node Red Set Parameters HTTP Request GET – Flow File Example

Detecting Similar Images

Use Case

I want to know programmatically whether listings posted at Mudah.My are similar or not even though it is posted by different persons and different date.

To do this, I think the best way is to detect whether the photos are similar or not.

Why To Know Same Property Is Advertised Over Period of Time?

I would like to know whether the property
1) price change over time, signalling time to purchase it
2) possibility owner becomes desperate to let it go if advertised for quite some time. So I can get better price

Same Property But Advertised by Different Agents and Different Date

All listings refer to same property by evaluating using naked eyes.
So now, I want to detect programmatically that all the listings are referring to same property by detecting those photos are similar.

Listing	Agent	Posted Date
Listing 1 URL Price: RM260,000	Dilla	05/07/2019
Listing 2 URL Price: RM260,000	Aiman	05/07/2019
Listing 3 URL Price:RM260,000	Norhayati	05/07/2019
Listing 4 URL Price:RM260,000	Fahana	05/07/2019
Listing 5 URL Price:RM260,000	Fazri	22/07/2019
Listing 6: URL Price:RM270,000	Marina	12/07/2019

Kitchen Photos

Seroja Apartment Listing 1 - Kitchen — Seroja Apartment Listing 1 – Kitchen

Seroja Apartment Listing 2 - Kitchen — Seroja Apartment Listing 2 – Kitchen

Seroja Apartment Listing 3 - Kitchen — Seroja Apartment Listing 3 – Kitchen

Seroja Apartment Listing 4 - Kitchen — Seroja Apartment Listing 4 – Kitchen

Seroja Apartment Listing 6 - Kitchen — Seroja Apartment Listing 6 – Kitchen

Bedroom Images

Seroja Apartment Listing 2 - Bedroom — Seroja Apartment Listing 2 – Bedroom

Seroja Apartment Listing 5 - Bedroom — Seroja Apartment Listing 5 – Bedroom

Technique Used

1. Step 1: Fingerprinting the Photos

Fingerprinting the photos is using image hashing technique. In this case, DHash will be used.

1. Step 2: Compare the Photos

After fingerprinting, image hash will be compared among listings. If the image hash are same or Levenshtein distance is less or equal to 2, then we can consider as the listings are referring to same property.

Results

Photos	Size	Noticeable Features	Image Hash (DHash)
Listing 1 - Kitchen	18KB 480x480	Kitchen - listing 1 ,listing 3 and listing 6 are similar	cc6c7a727e7c3e77
Listing 2 - Kitchen	19KB 640x480	Has door and wall fan	3f37333333333339
Listing 3 - Kitchen	18KB 480x480		cc6c7a727e7c3e77
Listing 4 - Kitchen	17KB 480x480	Kitchen - Listing 4 color is lighter compared to listing 1, 3 & 6	cc6e7a727e7c7e77
Listing 6 - Kitchen	18KB 480x480		cc6c7a727e7c3e77
Listing 2 - Bedroom	22KB 640x480	similar with bedroom listing 5 only size is different.	e1b90d0c8ccc84c1
Listing 5 - Bedroom	10KB 320x240		e1b52d0c8ccc84c1

Kitchen Photos

If we take Listing 1 as a base, we can say easily that it has same image hash with Listing 3 & Listing 6.

Listing 1 has Levenshtein distance of 2 with Listing 4.
Listing 1 has Levenshtein distance of 15 with Listing 5.

Bedroom Photos

Listing 2 and Listing 5 has Levenshtein distance of 2.

Conclusion

We can conclude photos are similar if their image hash is the same or their Levenshtein distance is less or equal to 2.

False Positive

What happens if different properties use same photos such as signage or building block? The algorithm will detect it as same property even though it is not.

To avoid this, we should establish a database of signage or building block to remove this false positive.

Photos Example

References:

Fingerprinting Images For Near Duplicate Detection
Python Code & Images Used
DHash Algorithm
Listing 1
Listing 2
Listing 3
Listing 4
Listing 5
Listing 6