zwayoriginPackets.json, zwayparsedPackets.json

Discussions about Z-Way software and Z-Wave technology in general
Post Reply
piet66
Posts: 91
Joined: 04 Feb 2017 17:00

zwayoriginPackets.json, zwayparsedPackets.json

Post by piet66 » 19 Nov 2019 13:56

There are these 2 files containing the received and sent z-wave messages.
They are suitable for analyzing telegram traffic.
Unfortunately the contents is not always correct: often messages are missing, which were received and processed (I know that the entries aren't stored at once). And also it happens, that messages are included several times.

I recognized this when looking at Security Message Encapsulation telegrams. Here it's easy to compare log file and zwayparsedPackets.json.

The value of the files would be much larger if they were complete.
Raspberry Pi 3 Model B Rev 1.2
Raspbian GNU/Linux 9 (stretch)
ZMEERAZ 2016, SDK Version: 6.51.03, Serial API Version: 05.00
Z-Way version v3.0.0 from 2019-07-18 02:13:01 +0300 (build 146fa9b44605907737b17032131987cab573da1f)

User avatar
PoltoS
Posts: 5091
Joined: 26 Jan 2011 19:36

Re: zwayoriginPackets.json, zwayparsedPackets.json

Post by PoltoS » 22 Nov 2019 03:32

This is a know architecture flow and very hard to fix. This is because of high load of your system

piet66
Posts: 91
Joined: 04 Feb 2017 17:00

Re: zwayoriginPackets.json, zwayparsedPackets.json

Post by piet66 » 23 Nov 2019 14:43

This is because of high load of your system
Does this mean, that on a faster hardware the files would contain all entries ?

But why are many entries several times in the json files? I counted for one special entry 6 times.
Raspberry Pi 3 Model B Rev 1.2
Raspbian GNU/Linux 9 (stretch)
ZMEERAZ 2016, SDK Version: 6.51.03, Serial API Version: 05.00
Z-Way version v3.0.0 from 2019-07-18 02:13:01 +0300 (build 146fa9b44605907737b17032131987cab573da1f)

piet66
Posts: 91
Joined: 04 Feb 2017 17:00

Re: zwayoriginPackets.json, zwayparsedPackets.json

Post by piet66 » 28 Nov 2019 14:39

For illustration here an example:
Messages selected from log:

Code: Select all

[2019-11-23 08:23:51.092] [I] [zway] Node 18:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
>> entry missing in zwayoriginPackets.json, zwayparsedPackets.json

[2019-11-23 08:25:06.635] [I] [zway] Node 22:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
>> entry missing in zwayoriginPackets.json, zwayparsedPackets.json

[2019-11-23 08:46:43.906] [I] [zway] Node 20:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 08:53:39.120] [I] [zway] Node 23:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 08:54:09.316] [I] [zway] Node 22:0 CC Security: passing decrypted packet to application level: [ 71 05 00 00 00 ff 06 16 00 ]
[2019-11-23 09:01:32.319] [I] [zway] Node 22:0 CC Security: passing decrypted packet to application level: [ 71 05 00 00 00 ff 06 17 00 ]
[2019-11-23 09:21:59.078] [I] [zway] Node 18:0 CC Security: passing decrypted packet to application level: [ 84 07 ]

[2019-11-23 09:25:28.190] [I] [zway] Node 22:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
>> entry 2 times in zwayoriginPackets.json, zwayparsedPackets.json

[2019-11-23 09:43:02.564] [I] [zway] Node 20:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 09:50:21.186] [I] [zway] Node 23:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 10:20:27.684] [I] [zway] Node 18:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 10:22:48.239] [I] [zway] Node 22:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 10:39:20.827] [I] [zway] Node 20:0 CC Security: passing decrypted packet to application level: [ 84 07 ]
[2019-11-23 10:47:39.762] [I] [zway] Node 23:0 CC Security: passing decrypted packet to application level: [ 84 07 ]

[2019-11-23 10:47:40.254] [W] [zway] Node 23:0 CC Security: can not decrypt packet due to nonce lookup problem
>> entry missing in zwayoriginPackets.json, zwayparsedPackets.json

[2019-11-23 10:47:42.679] [W] [zway] Node 23:0 CC Security: can not decrypt packet due to nonce lookup problem
>> entry 4 times in zwayoriginPackets.json, zwayparsedPackets.json

There are a lot more multiple entries in these files.
Today one entry I just found 26 times, a "38=Command Class Switch Multilevel, 2=Switch Multilevel Get". The all were delivered=true.

Code: Select all

{

    "updateTime": 1574936730,
    "delivered": true,
    "deliveryTime": 60,
    "packetLength": 5,According to my estimate about 10% of the entries are double or multiple.
    "nodeId": 53,
    "returnRSSI": null,
    "hops": null,
    "returnChannel": null,
    "txChannel": null,
    "speed": null,
    "schemeState": null,
    "tries": null,
    "lastFailPath": null,
    "value": "[0,19,53,2,38,2,37]"

},
{

    "updateTime": 1574936730,
    "delivered": true,
    "deliveryTime": 60,
    "packetLength": 5,
    "nodeId": 53,
    "returnRSSI": null,
    "hops": null,
    "returnChannel": null,
    "txChannel": null,
    "speed": null,
    "schemeState": null,
    "tries": null,
    "lastFailPath": null,
    "value": "[0,19,53,2,38,2,37]"

},
{

    "updateTime": 1574936730,
    "delivered": true,
    "deliveryTime": 60,
    "packetLength": 5,
    "nodeId": 53,
    "returnRSSI": null,
    "hops": null,
    "returnChannel": null,
    "txChannel": null,
    "speed": null,
    "schemeState": null,
    "tries": null,
    "lastFailPath": null,
    "value": "[0,19,53,2,38,2,37]"

}
I don't believe, that zway sends 26 Multilevel Gets to one node within one second. There must be a loop anywhere.
About 1300 of 5000 entries are double or multiple. (I can't know which of them are really existing doubles). Of course I don't know how many are missing.
This considerably reduces the benefits of PacketLog and Zniffer.
Raspberry Pi 3 Model B Rev 1.2
Raspbian GNU/Linux 9 (stretch)
ZMEERAZ 2016, SDK Version: 6.51.03, Serial API Version: 05.00
Z-Way version v3.0.0 from 2019-07-18 02:13:01 +0300 (build 146fa9b44605907737b17032131987cab573da1f)

User avatar
PoltoS
Posts: 5091
Joined: 26 Jan 2011 19:36

Re: zwayoriginPackets.json, zwayparsedPackets.json

Post by PoltoS » 02 Dec 2019 03:28

This sounds pretty strange for me.

In brief, when JS callback for new packet arrives, it reads C level variable that might in theory change by that time. But this is very rare since C level code being fast still gives quite some quantum to the JS code. So this is really rare.

Either this is real data or you see some effect of high load of Z-Way.

The best is to compare Z-Way log (C level log) with the JSON file.

Post Reply