Page 1 of 1

[TW] Frequent Lag Spikes

Posted: 21 Aug 2018 20:41
by Regent
I don't know if it's the new host or if it's memory leak issues in the core, but whatever the cause may be, it's causing an insane amount of lag spikes quite frequently - often at the worst possible times like a boss fight in a raid.
I, as well as several others, have checked tracerts to the server and the problem is not on our end.

It's hard enough to get raids together on TW realm now - the last thing it needs is these lag spikes to ruin gameplay and cause wipes and severe inconveniences.

Someone please look into this before it gets worse.
Regards.

Re: [TW] Frequent Lag Spikes

Posted: 22 Aug 2018 00:40
by IronSharona
If you're talking about some of the lag spike from today around 7:15 and 8:15 pm SVT, these were the result of the core compilations for weekly updates. I tried to make in game announcements around the time that the compiles were started so people would be aware the lag was likely coming. It seems that the new host is not as good at being able to handle the workload as our prior was. We have taken steps to reduce the load on the host during compiles, and Noxies has a plan to further optimize the processing of compiles but I don't know what his ETA on that would be.

As to what is causing the freezes/crashes on the weekends lately, we are not yet sure, but are actively looking into it. We've gathered a series of possible causes and they need to be evaluated.

In the meantime, do you have any suggestions for how we can help to lessen the impact of the update process for you and the other players, if that was your primary concern today? I'm happy to announce when each compile will be starting and ending and suggest anyone else who runs updates do the same, if knowing the timeline on it would ease your mind. Is there a different time of day or a different day of the week that would be preferable for compiles? (Keep in mind that compiles require an admin to be available, so any time adjustments to updates must be agreed upon just to ensure they are feasible for us.)

Re: [TW] Frequent Lag Spikes

Posted: 22 Aug 2018 03:46
by Dunkelstein
There is usually nothing going on from 07:00 to 15:00 server time (7am - 3pm GMT+2 Summertime). Most raids in general are from Wednesday to Sunday. Most arenas in general are on Tuesday nights.
If an Admin has to be present, you should prioritize for their needs, though. Development is the bottle neck on this server so they should be as comfortable as possible.

Re: [TW] Frequent Lag Spikes

Posted: 22 Aug 2018 12:30
by Jiranthos
In my opinion, we should automate compilation scripts as well, like a scheduled cronjob. This does seem to be problem with the host being less powerful now and we need to optimize and automate wherever possible. This would make it so that admin is not required to remember and trigger it BUT it would require some admin on pager duty - if something fails to compile or goes wrong - someone has to be around to troubleshoot, announce etc.

I will be looking into automating this and set up a clear timeline for compiles and restarts if everyone agrees (both staff and playerbase).
The schedule could be for example as follows (this is not yet decided on):

Evening of Tuesday - compile PTR and restart it to allow to catch compilation errors and bugs and developers to test their features.
8:00 of Wednesday morning - compile PW and restart it
9:00 of Wednesday morning - compile TW and restart it

All times ST.

Re: [TW] Frequent Lag Spikes

Posted: 22 Aug 2018 18:37
by Dunkelstein
Won't compiling for PTR on Tuesday nights also lag the other realms? Why not do it Tuesday or Monday morning, give testing a bit more time before weekly update, too?

Re: [TW] Frequent Lag Spikes

Posted: 22 Aug 2018 19:47
by Eronox
I've lowered cores used on compilation and also done a overhaul of cleanup and procedure of threading.
A beneficial effect should be noticed from now on.

Re: [TW] Frequent Lag Spikes

Posted: 23 Aug 2018 11:30
by Jiranthos
Dunkelstein wrote:
22 Aug 2018 18:37
Won't compiling for PTR on Tuesday nights also lag the other realms? Why not do it Tuesday or Monday morning, give testing a bit more time before weekly update, too?
No, we have not seen any lags during PTR compiles. But sure it can be compiled earlier but compiling it TOO early makes the development sprint/iteration (The time when devs can commit changes for the next update) smaller and forces them to develop things in 2-3 days and is a lot of stress and if they fail to do so - we will see restarts with very little updates per week, thus killing the purpose.

Re: [TW] Frequent Lag Spikes

Posted: 23 Aug 2018 11:30
by Jiranthos
Noxies wrote:
22 Aug 2018 19:47
I've lowered cores used on compilation and also done a overhaul of cleanup and procedure of threading.
A beneficial effect should be noticed from now on.
Thank you very much. <3

Re: [TW] Frequent Lag Spikes

Posted: 23 Aug 2018 21:10
by Nyeriah
Compilation is heavy on disk usage and that has always been the bottleneck we had. Even when we had a SSD in use we had issues during compile times in consequence to that, so it’s really expected for those peaks to happen during compiles now that we are on a hard drive, which is incredibly slower in comparison to SSD.

Re: [TW] Frequent Lag Spikes

Posted: 18 Sep 2018 22:56
by Jiranthos
I am working on a solution to this right now. Stay tuned.