z-Way Server stops working

Discussions about RaZberry - Z-Wave board for Raspberry computer
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

Looks like my watchdog didn't work well as I thought, still somewhere a crash overnight.

originPacketsjson can became large as well.

My system is pretty simple:
I have two z-wave aeotec multisensor 6.
I have additional process that queries every 60 seconds from z-way-server by using these from IP: 127.0.0.1:8083

const char *baseSidQuery = "curl --connect-timeout 10 --max-time 600 -d '{\"password\":\"%s\",\"login\":\"%s\",\"rememberme\":false}' -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -X POST %s/ZAutomation/api/v1/login";
const char *baseDevicesQuery = "curl --connect-timeout 10 --max-time 600 -X GET -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -H \"ZWAYSession: %s\" -H \"Cookie: ZWAYSession=%s\" %s/ZAutomation/api/v1/devices";
const char *allDevicesQuery = "curl --connect-timeout 10 --max-time 600 -X GET -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -H \"ZWAYSession: %s\" -H \"Cookie: ZWAYSession=%s\" %s/ZWaveAPI/Run/devices";

Perhaps I should start using earlier version. I think, they were more stable.
User avatar
PoltoS
Posts: 7562
Joined: 26 Jan 2011 19:36

Re: z-Way Server stops working

Post by PoltoS »

Can you give use more info?

Is it crashing or hanging? If crashing, can we get the gdb trace?
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

it's hanging and then it seems whole raspberry pi can crash.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
434 root 20 0 114220 38720 15736 S 100.2 4.1 0:44.20 z-way-server

-> CPU usage have been all the way 100% long time.
-> It "probably" have something to do with those curl calls I did put on previous post. I could be wrong.
---
---- and so on, that seems to be continuation forever now in this process 434
---- weirdly, I cannot even kill this process 434 :)

This version was: v2.3.1-5-g4292f78

It can be also that it was writing something big and something went wrong.
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

There is another bug as well.

My 2x aeotec multisensor 6 should send data to razberry (1) in every 10 seconds. but z-way-server seems to stop recognizing these multisensors after some time - sometimes it may take few hours, sometimes more, sometimes less.

But after I reset

/etc/init.d/z-way-server stop
/etc/init.d/z-way-server start

It's all fine.
---
I do have a watchdog for these too. These have nothing to do curl urls before.

Sounds like there is no proper stress testing in z-way-server. if you stress-test this one-two days, it should be easy to get crash/hangup/or just behaving buggy.
User avatar
PoltoS
Posts: 7562
Joined: 26 Jan 2011 19:36

Re: z-Way Server stops working

Post by PoltoS »

Sounds really strange. Log is needed. I believe there is a rational explanation behind
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

Yes, the reason seems that large number of data is coming from Multisensors, and this causes multiple issues for raspberry pi & zway server.

* I have those multisensors updating data in every 10 seconds.
-> Sometimes when filesystem is writing alot data at the same time -> /dev/ttyAMA0 just stops receiving data, it seems this have something to do with ext filesystem. as we all know, in ext filesystems data is not fully saved, even when the functions would return that all data is saved.
-> now when there is overflow of saving on same file(s), eventually filesystem goes stuck and this may cause system crash.
-> Another case for these problems, if filesystems get stuck, /dev/ttyAMA0 may just stop receiving data until z-way-server is re-started. This may have something to do with z-way-server - perhaps reading or saving eventually gets stuck. and that's why there is no more "visible" streaming from /dev/ttyAMA0.

Anyways, I sort of have workaround for this saving mess:
* on /opt/z-way-server/config.xml
I did set: <log-level>5</log-level> (Silent, no logging).
-> I noted that /var/log/z-way-server.log gets extreme large in quick time with these multisensors, in few hours it was 18M size.

Then on /opt/z-way-server/automation/main.js
I "disabled" function: saveObject = function(name, object) {....

saveObject = function(name, object) {
/* if (__storageContent.indexOf(name) === -1 && !!name) {
__storageContent.push(name);
__saveObject("__storageContent", __storageContent);
}
__saveObject(name, object);
*/
};
-> then there is no more extra large json files saving. and now it's been quite stable.

I am slightly worried about events, z-way-server shows large number of events still, I wish I know how to disable that collection of events history.
Basically that's: http://xxx.xxx.xxx.xxx:8083/smarthome/#/events
on z-way-server
----
But so far, this is much more stable now. But perhaps it might crash later.
----
So I hope that soon there will be way to disable (and set limit) into that automation/storage json, they seems to not be required. and also disable (and set limits) into events. and if there is anything else large saving, then disable (and set limits) for that too.

I have
* 2x Aeotec Multisensor 6, I will configure these to push data slower, perhaps once in 120 seconds. But 10 seconds is good stress test for start.
* Raspberry Pi 3
* Image: Jessie Lite + basic UI addons
* Razberry (1)
* At the moment: z-way-server 2.3.1, but I believe, that latest z-way-server is ok as well, as long I remember to disable automation/storage and logging.
* Micro SD card write speed should be 70M/s, but due ext filesystem "features", it's not always the case.
---
I have programmed own little wall clock around this system, it shows
* Current date
* Current time
* Current temperature, humid, UV, Light from multisensors (indoor and outdoor)
* Weather forecast for next 5 days.
-> All on 7" display. that is on all a time. (Display have little slot for Raspberry Pi behind the display).
-> this is not correct thread but I upload the picture here anyways. It's all finnish, but you get the idea.
Attachments
workingweather.png
workingweather.png (742.16 KiB) Viewed 13496 times
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

Looks like my earlier modifications made z-way-server much more stable, but not fully.

* Before modifications z-way-server crashed or caused a system crash in few hours due stress test.
* Now z-way-server died after one week of running due stress test.
-> I didn't need to do boot on raspberry pi, just starting z-way-server over again and it's again running.

It's improvement, but not fully. Perhaps I should try newest version again and disable automation storage and logging again. and I think, disabling events handling should help as well.

Stress tests are good way to check if software is actually running well all a time. This kind of software requires 3-4 weeks full stress test because it's supposed to run all a time.
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

Actually I had to reboot the raspberry pi eventually. I will give one more try for z-way-server, and then try other things if it crash. This time events are "hidden".
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

I made logging enabled again, and put my logs here before it started to slow down alot.
* I also put my 2x multisensors push data only once in 2 minutes, but for some reason, data collection gets huge in 1.5 hours... seems some issues in data handling. Which is most likely reason for all these crash talk that people are telling on these forums.
Attachments
z-way-server-log.zip
(407.46 KiB) Downloaded 306 times
timohan
Posts: 17
Joined: 09 Feb 2017 14:26

Re: z-Way Server stops working

Post by timohan »

I have changed to homegenie and now everything seems to be stable.
-> homegenie have been up 11 days already and things are looking good for my "wall clock" - no crashes, all performance levels are good, CPU usage is low as well on my stress testing.
-> I may give a try for z-way-server later because homegenie doesn't give me UV from sensor + homegenie's UI is kinda bad - even I don't need homegenie's UI.
Post Reply