Page 4 of 5

Re: z-Way Server stops working

Posted: 16 Jun 2017 06:54
by timohan
Looks like my watchdog didn't work well as I thought, still somewhere a crash overnight.

originPacketsjson can became large as well.

My system is pretty simple:
I have two z-wave aeotec multisensor 6.
I have additional process that queries every 60 seconds from z-way-server by using these from IP:

const char *baseSidQuery = "curl --connect-timeout 10 --max-time 600 -d '{\"password\":\"%s\",\"login\":\"%s\",\"rememberme\":false}' -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -X POST %s/ZAutomation/api/v1/login";
const char *baseDevicesQuery = "curl --connect-timeout 10 --max-time 600 -X GET -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -H \"ZWAYSession: %s\" -H \"Cookie: ZWAYSession=%s\" %s/ZAutomation/api/v1/devices";
const char *allDevicesQuery = "curl --connect-timeout 10 --max-time 600 -X GET -H \"Content-Type: application/json;charset=utf-8\" -H \"Accept: application/json, text/plain, */*\" -H \"ZWAYSession: %s\" -H \"Cookie: ZWAYSession=%s\" %s/ZWaveAPI/Run/devices";

Perhaps I should start using earlier version. I think, they were more stable.

Re: z-Way Server stops working

Posted: 16 Jun 2017 10:52
by PoltoS
Can you give use more info?

Is it crashing or hanging? If crashing, can we get the gdb trace?

Re: z-Way Server stops working

Posted: 16 Jun 2017 16:15
by timohan
it's hanging and then it seems whole raspberry pi can crash.

434 root 20 0 114220 38720 15736 S 100.2 4.1 0:44.20 z-way-server

-> CPU usage have been all the way 100% long time.
-> It "probably" have something to do with those curl calls I did put on previous post. I could be wrong.
---- and so on, that seems to be continuation forever now in this process 434
---- weirdly, I cannot even kill this process 434 :)

This version was: v2.3.1-5-g4292f78

It can be also that it was writing something big and something went wrong.

Re: z-Way Server stops working

Posted: 17 Jun 2017 07:24
by timohan
There is another bug as well.

My 2x aeotec multisensor 6 should send data to razberry (1) in every 10 seconds. but z-way-server seems to stop recognizing these multisensors after some time - sometimes it may take few hours, sometimes more, sometimes less.

But after I reset

/etc/init.d/z-way-server stop
/etc/init.d/z-way-server start

It's all fine.
I do have a watchdog for these too. These have nothing to do curl urls before.

Sounds like there is no proper stress testing in z-way-server. if you stress-test this one-two days, it should be easy to get crash/hangup/or just behaving buggy.

Re: z-Way Server stops working

Posted: 17 Jun 2017 11:55
by PoltoS
Sounds really strange. Log is needed. I believe there is a rational explanation behind

Re: z-Way Server stops working

Posted: 18 Jun 2017 12:56
by timohan
Yes, the reason seems that large number of data is coming from Multisensors, and this causes multiple issues for raspberry pi & zway server.

* I have those multisensors updating data in every 10 seconds.
-> Sometimes when filesystem is writing alot data at the same time -> /dev/ttyAMA0 just stops receiving data, it seems this have something to do with ext filesystem. as we all know, in ext filesystems data is not fully saved, even when the functions would return that all data is saved.
-> now when there is overflow of saving on same file(s), eventually filesystem goes stuck and this may cause system crash.
-> Another case for these problems, if filesystems get stuck, /dev/ttyAMA0 may just stop receiving data until z-way-server is re-started. This may have something to do with z-way-server - perhaps reading or saving eventually gets stuck. and that's why there is no more "visible" streaming from /dev/ttyAMA0.

Anyways, I sort of have workaround for this saving mess:
* on /opt/z-way-server/config.xml
I did set: <log-level>5</log-level> (Silent, no logging).
-> I noted that /var/log/z-way-server.log gets extreme large in quick time with these multisensors, in few hours it was 18M size.

Then on /opt/z-way-server/automation/main.js
I "disabled" function: saveObject = function(name, object) {....

saveObject = function(name, object) {
/* if (__storageContent.indexOf(name) === -1 && !!name) {
__saveObject("__storageContent", __storageContent);
__saveObject(name, object);
-> then there is no more extra large json files saving. and now it's been quite stable.

I am slightly worried about events, z-way-server shows large number of events still, I wish I know how to disable that collection of events history.
Basically that's:
on z-way-server
But so far, this is much more stable now. But perhaps it might crash later.
So I hope that soon there will be way to disable (and set limit) into that automation/storage json, they seems to not be required. and also disable (and set limits) into events. and if there is anything else large saving, then disable (and set limits) for that too.

I have
* 2x Aeotec Multisensor 6, I will configure these to push data slower, perhaps once in 120 seconds. But 10 seconds is good stress test for start.
* Raspberry Pi 3
* Image: Jessie Lite + basic UI addons
* Razberry (1)
* At the moment: z-way-server 2.3.1, but I believe, that latest z-way-server is ok as well, as long I remember to disable automation/storage and logging.
* Micro SD card write speed should be 70M/s, but due ext filesystem "features", it's not always the case.
I have programmed own little wall clock around this system, it shows
* Current date
* Current time
* Current temperature, humid, UV, Light from multisensors (indoor and outdoor)
* Weather forecast for next 5 days.
-> All on 7" display. that is on all a time. (Display have little slot for Raspberry Pi behind the display).
-> this is not correct thread but I upload the picture here anyways. It's all finnish, but you get the idea.

Re: z-Way Server stops working

Posted: 25 Jun 2017 09:39
by timohan
Looks like my earlier modifications made z-way-server much more stable, but not fully.

* Before modifications z-way-server crashed or caused a system crash in few hours due stress test.
* Now z-way-server died after one week of running due stress test.
-> I didn't need to do boot on raspberry pi, just starting z-way-server over again and it's again running.

It's improvement, but not fully. Perhaps I should try newest version again and disable automation storage and logging again. and I think, disabling events handling should help as well.

Stress tests are good way to check if software is actually running well all a time. This kind of software requires 3-4 weeks full stress test because it's supposed to run all a time.

Re: z-Way Server stops working

Posted: 25 Jun 2017 21:34
by timohan
Actually I had to reboot the raspberry pi eventually. I will give one more try for z-way-server, and then try other things if it crash. This time events are "hidden".

Re: z-Way Server stops working

Posted: 26 Jun 2017 18:00
by timohan
I made logging enabled again, and put my logs here before it started to slow down alot.
* I also put my 2x multisensors push data only once in 2 minutes, but for some reason, data collection gets huge in 1.5 hours... seems some issues in data handling. Which is most likely reason for all these crash talk that people are telling on these forums.

Re: z-Way Server stops working

Posted: 12 Jul 2017 19:59
by timohan
I have changed to homegenie and now everything seems to be stable.
-> homegenie have been up 11 days already and things are looking good for my "wall clock" - no crashes, all performance levels are good, CPU usage is low as well on my stress testing.
-> I may give a try for z-way-server later because homegenie doesn't give me UV from sensor + homegenie's UI is kinda bad - even I don't need homegenie's UI.