[TW] Frequent Lag Spikes

Locked
User avatar
Regent
Tester
Posts: 375
Joined: 30 Apr 2014 16:11
Location: Darnassus, Teldrassil.
Contact:

[TW] Frequent Lag Spikes

#1 » Post by Regent » 21 Aug 2018 20:41

I don't know if it's the new host or if it's memory leak issues in the core, but whatever the cause may be, it's causing an insane amount of lag spikes quite frequently - often at the worst possible times like a boss fight in a raid.
I, as well as several others, have checked tracerts to the server and the problem is not on our end.

It's hard enough to get raids together on TW realm now - the last thing it needs is these lag spikes to ruin gameplay and cause wipes and severe inconveniences.

Someone please look into this before it gets worse.
Regards.
Rest in Peace,
Sarah Melody Jones.
May 20, 1989 - May 12, 2018.
Beloved Girlfriend & Mother-To-Be.
When I close my eyes, I see you.
When I open my eyes, I miss you.

ImageImageImageImageImageImageImageImage
Regent- - - - - - - - -Regentt- - - - - - - - -Regennt- - - - - - - - -Regency

User avatar
IronSharona
Former Staff
Posts: 1145
Joined: 02 Feb 2017 00:49
Location: Austin, Texas, USA

Re: [TW] Frequent Lag Spikes

#2 » Post by IronSharona » 22 Aug 2018 00:40

If you're talking about some of the lag spike from today around 7:15 and 8:15 pm SVT, these were the result of the core compilations for weekly updates. I tried to make in game announcements around the time that the compiles were started so people would be aware the lag was likely coming. It seems that the new host is not as good at being able to handle the workload as our prior was. We have taken steps to reduce the load on the host during compiles, and Noxies has a plan to further optimize the processing of compiles but I don't know what his ETA on that would be.

As to what is causing the freezes/crashes on the weekends lately, we are not yet sure, but are actively looking into it. We've gathered a series of possible causes and they need to be evaluated.

In the meantime, do you have any suggestions for how we can help to lessen the impact of the update process for you and the other players, if that was your primary concern today? I'm happy to announce when each compile will be starting and ending and suggest anyone else who runs updates do the same, if knowing the timeline on it would ease your mind. Is there a different time of day or a different day of the week that would be preferable for compiles? (Keep in mind that compiles require an admin to be available, so any time adjustments to updates must be agreed upon just to ensure they are feasible for us.)
Welcome to the new decade! Let's hope the Roaring 20's come with just as much jazz, but a bit less prohibition and Great Depression. 8-)

User avatar
Dunkelstein
Posts: 324
Joined: 11 Oct 2015 22:32

Re: [TW] Frequent Lag Spikes

#3 » Post by Dunkelstein » 22 Aug 2018 03:46

There is usually nothing going on from 07:00 to 15:00 server time (7am - 3pm GMT+2 Summertime). Most raids in general are from Wednesday to Sunday. Most arenas in general are on Tuesday nights.
If an Admin has to be present, you should prioritize for their needs, though. Development is the bottle neck on this server so they should be as comfortable as possible.
Aka Funkelstein

User avatar
Jiranthos
Admin
Posts: 1978
Joined: 23 Jun 2015 03:43
Location: Not on your bad side, hopefully

Re: [TW] Frequent Lag Spikes

#4 » Post by Jiranthos » 22 Aug 2018 12:30

In my opinion, we should automate compilation scripts as well, like a scheduled cronjob. This does seem to be problem with the host being less powerful now and we need to optimize and automate wherever possible. This would make it so that admin is not required to remember and trigger it BUT it would require some admin on pager duty - if something fails to compile or goes wrong - someone has to be around to troubleshoot, announce etc.

I will be looking into automating this and set up a clear timeline for compiles and restarts if everyone agrees (both staff and playerbase).
The schedule could be for example as follows (this is not yet decided on):

Evening of Tuesday - compile PTR and restart it to allow to catch compilation errors and bugs and developers to test their features.
8:00 of Wednesday morning - compile PW and restart it
9:00 of Wednesday morning - compile TW and restart it

All times ST.

Everybody knows that the best way to describe the ocean to a blind man is to push him in

User avatar
Dunkelstein
Posts: 324
Joined: 11 Oct 2015 22:32

Re: [TW] Frequent Lag Spikes

#5 » Post by Dunkelstein » 22 Aug 2018 18:37

Won't compiling for PTR on Tuesday nights also lag the other realms? Why not do it Tuesday or Monday morning, give testing a bit more time before weekly update, too?
Aka Funkelstein

User avatar
Eronox
MVP
Posts: 2331
Joined: 24 Apr 2016 11:03
Location: in ICC & Healing

Re: [TW] Frequent Lag Spikes

#6 » Post by Eronox » 22 Aug 2018 19:47

I've lowered cores used on compilation and also done a overhaul of cleanup and procedure of threading.
A beneficial effect should be noticed from now on.
.

There's things that never will be right I know, and things need changin' everywhere you go.
But 'til we start to make a move to make a few things right,
You'll never see me wear a suit of white.

- J.R Cash


User avatar
Jiranthos
Admin
Posts: 1978
Joined: 23 Jun 2015 03:43
Location: Not on your bad side, hopefully

Re: [TW] Frequent Lag Spikes

#7 » Post by Jiranthos » 23 Aug 2018 11:30

Dunkelstein wrote:
22 Aug 2018 18:37
Won't compiling for PTR on Tuesday nights also lag the other realms? Why not do it Tuesday or Monday morning, give testing a bit more time before weekly update, too?
No, we have not seen any lags during PTR compiles. But sure it can be compiled earlier but compiling it TOO early makes the development sprint/iteration (The time when devs can commit changes for the next update) smaller and forces them to develop things in 2-3 days and is a lot of stress and if they fail to do so - we will see restarts with very little updates per week, thus killing the purpose.

Everybody knows that the best way to describe the ocean to a blind man is to push him in

User avatar
Jiranthos
Admin
Posts: 1978
Joined: 23 Jun 2015 03:43
Location: Not on your bad side, hopefully

Re: [TW] Frequent Lag Spikes

#8 » Post by Jiranthos » 23 Aug 2018 11:30

Noxies wrote:
22 Aug 2018 19:47
I've lowered cores used on compilation and also done a overhaul of cleanup and procedure of threading.
A beneficial effect should be noticed from now on.
Thank you very much. <3

Everybody knows that the best way to describe the ocean to a blind man is to push him in

User avatar
Nyeriah
Posts: 658
Joined: 13 Sep 2015 17:10

Re: [TW] Frequent Lag Spikes

#9 » Post by Nyeriah » 23 Aug 2018 21:10

Compilation is heavy on disk usage and that has always been the bottleneck we had. Even when we had a SSD in use we had issues during compile times in consequence to that, so it’s really expected for those peaks to happen during compiles now that we are on a hard drive, which is incredibly slower in comparison to SSD.
“All your life, you will be faced with a choice. You can choose love or hate…I choose love.”
― Johnny R. Cash

:heart:

User avatar
Jiranthos
Admin
Posts: 1978
Joined: 23 Jun 2015 03:43
Location: Not on your bad side, hopefully

Re: [TW] Frequent Lag Spikes

#10 » Post by Jiranthos » 18 Sep 2018 22:56

I am working on a solution to this right now. Stay tuned.

Everybody knows that the best way to describe the ocean to a blind man is to push him in

Locked

Who is online

Users browsing this forum: No registered users and 3 guests