Year of the Dragon: Through April 23rd, claim the adventure pack Slice of Life for free! Speak to Xatheral in the Hall of Heroes.

Game mechanics • Newbie guide • In development • DDO Store • Social Media

Challenges • Classes • Collectables • Crafting • Enhancements • Epic Destinies • Favor • Feats

Glossary • Items • Maps • Monsters • Places • Quests • Races • Reincarnation • Skills • Spells

Please create an account or log in to build a reputation and unlock more editing privileges, and then visit DDO wiki's IRC Chat/Discord if you need any help!

Talk:Game performance issues after 2016 March data center move

From DDO wiki
Jump to navigation Jump to search
Forums: Index > Troubleshoot > Game performance issues after 2016 March data center move

anybody still out there? well, its kinda late, I haven't been playing much so it didn't affect me much but this now-a-month-long lag fiasco appears to be a big deal, really "doom"-worthy. I think we should be archiving whats happening with current "state of the game". some say same kinda performance issue happened for LotRO and some say it didnt and improved things greatly for them, anybody know the details there or a thread or two to look up on? yoko5000 (Contribs • Message) 01:49, April 6, 2016 (EDT)

Timeline and forum threads[edit]

Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

Data Center Migration update: So far things are progressing well, and we are on track with out expected uptime.

  • 03-08-2016 (official) Port Forwarding update and changes - Important
    • If you previously setup port forwarding (or port triggering) on your router and/or modem for ports 9000-9010 UDP, updating the range of ports to 9000-9058 is now recommended.
Quote:
Originally Posted by Cocomajobo Source

This work was done as a test case to attempt to isolate issues causing the lag. When we determine that we have found the root of the issues through these test cases we will, of course, implement the changes on all servers.

Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

1. What's being done to resolve these issues?

We are implementing new network connections and adjusting our main networking backbone. Additionally, we are troubleshooting server-side configuration changes.

2. Why is it taking so long?

As part of our move to the new data center we updated some architecture to modern operating systems and virtualization technologies. In some cases adjusting the old code to run in this new environment has added additional challenge.

3. What is happening in the next few days?

We will be migrating select networking components, and make additional configuration changes to servers. When these efforts require downtime, we'll let you know.

Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

...we would like to continue to get your experiences and reports. Please provide the following information:

{{#widget:YouTube|id=WKQswRWtrKw|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
{{#widget:YouTube|id=81iMfGoZDXU|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
{{#widget:YouTube|id=b1I2h5VLTu4|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

We are currently in the process of working on a hotfix, and will use the downtime to deploy the hotfix and make some adjustments at the data center. ETA incoming but not ready yet.

Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

The plan is to roll out a change made to Argonessen and Sarlona to all the worlds next week, along with a patch/hotfix that squashes several "stacking enhancement" issues. Stay tuned on an ETA.

Quote:
Originally Posted by Turbine representative's response

We are well aware that some players are continuing to experience issues as a result of our recent move to a new data center. We have had teams across the studio working non-stop to resolve outstanding issues. Moving data centers is a very complex task and while we have made resolved many of these issues, we clearly have more work to do.

Quote:
Originally Posted by Justin Olivetti, Massively Podcast 60 Source

Probably one of the most passive-aggressive comments we've seen a studio post!

{{#widget:YouTube|id=h4nNLDdi5CY|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
  • some details on tomorrow's fix
    • "some backend technical fixes that will replace some Microsoft functionality that was duplicated" ... "the games right now cannot switch CPUs in a virtualized environment"
    • "barbarian fixes, that are causing effects stacking, that will be all cleaned up"
    • "also working behind the scenes to shore up some of our networking lines, hang up on that as we negotiating with vendors".... "just a process we cannot turn around quickly because those contracts have to be rock solid"
    • "If you raid with barbarians, it should make a big difference"
    • "the reason we're doing this is, this is something in other properties we saw big change as we moved them up into these later operating systems, so I'm reasonably confident this will make a difference"
    • "until we shore up the networking anyone that's going through the alternate networking sentry link, I believe, we will see bits of lag just because of the network latency."
    • "but the thing is, this would clear up a lot of stuff so that evaluations are much cleaner."
  • 04-14-2016 (official) The latest on lag and game performance improvements ← current player report thread
    • we identified an improvement we will be making to our monitoring tools to lessen their impact on game performance, and will be working to bring that out to the game worlds in the relatively near future.
  • 04-20-2016 (official) Downtime Notice: Wednesday, April 20th 1:00 - 3:00 PM Eastern
    • ... making some specific changes to Argonnessen and Wayfinder in an effort to improve game performance and reduce lag. These changes involve;
      • various updates to 64-bit dll's
      • and implementing an alternate internal game timer to Argonnessen and Wayfinder.
    • We will be keeping an eye on the impact, and will consider the next step afterwards.
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

Lamannia is in the new data center.

  • 05-10-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

This past weekend had an above-normal amount of poor game performance, and is something we will be looking into. We are continuing to look into the various reported causes of lag, including renown gain. We would continue to caution that Defiler of the Just has issues outside of server-client latency, and is not a good test case for lag. We fully understand people's frustration in Defiler of the Just, and will be considering some options for that particular raid in the future.

  • 05-10-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

There was some work done at the data center today, so we're checking into that prior to reopening.

Quote:
Originally Posted by Release Notes
Maximum number of loops have been performed
Defiler of the Just
  • Reduced the number of reinforcements that spawn in the raid, and adjusted the remaining reinforcements' Melee and Ranged Power accordingly.
  • A delay has been added between the double reinforcement spawns.
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan)

This is a temporary test to gather information necessary to our investigation.

We expect the tech to have the following impact:

  • Haywire Foundry will not be able to be completed.
  • Moving platforms in the Cannith Manufactury will no longer move, which blocks access to Schemes of the Enemy, The Master Artificer, The Lord of Blades, and the Artificer Workshops and Research Facility.

It is possible there will be other impacts as well. This change will be in place for two hours tonight, after which point we will turn it off, gather the results, and use those results in our ongoing investigation. We apologize for the inconvenience to players on Argonnessen.

  • 05-13-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

I don't know that we can say for sure that "guild renown causes lag", but yes, there is definitely something going on that manifests in some folks (but not everyone) experiencing lag when renown is granted, and it affects everyone in the group. The reason I say we can't say for sure is it's not certain whether the renown lag is a symptom or a cause. This is something we are continuing to look into.

  • 05-13-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

Figured we should end the week with a bit of a status report:

We did work this week to improve game performance in Defiler of the Just, and at least initially the results seem quite positive. We will be able to say more definitively on both this and other improvements as time goes on; it's true that we rebooted the worlds yesterday, so we'll be monitoring how things change over time.

Yesterday, we ran some tech on Argonnessen that is being looked at to see if it delivers on smoothing out server spikes. The tech has some issues (like breaking Von4), but if the data shows a significant improvement, we'll do further work to remove these issues before putting the tech in place.

Overall, lag in parties and smaller raid groups is greatly improved over what it was, say, a month ago. We're continuing to work on identifying and resolving sources of lag, including an investigation into the so-called "guild renown lag".

{{#widget:YouTube|id=D6FZ3C4mBPk|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
{{#widget:YouTube|id=s8hoRRxTiiA|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
{{#widget:YouTube|id=c3uB-ezN4oA|t=0s|width=640|height=480|autohide=&autohide=2|autoplay=0|rel=0|modestbranding=1|annotation=1|showinfo=1|loop=0|}}
  • 05-17-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

We are in the process of getting out a patch that will hopefully include a fix to guild renown lag. Since gauging our success in fixing the issue is challenging outside of the real, live world environment, we don't want to say we've fixed it until we hear from you that it's fixed and seen it ourselves while running things like Legendary Shroud. That said, we - think - we've discovered and corrected the issue, which involved sluggish guild renown SQL communication in our virtualized environment.

If all goes according to plan, the patch will be out later this week, and will include a few other additions like the two new enhancements to the Fighter Kensei tree. We'll have more info, including a downtime announcement and release notes, as we get things confirmed in the next 24-48 hours.

Quote:
Originally Posted by Release Notes

Renown

  • Corrected an issue that could cause lag when guild renown is granted.
  • 05-19-2016
Quote:
Originally Posted by Cordovan (Contribs • Message • Cordovan) Source

We're certainly not basing any kind of long-term conclusion on lag in DDO by experience in the first couple of days following a server restart, but it is still valuable to know whether things are better or worse (or the same) than they were the last time we rebooted the worlds at this point in time. The amount of time since the last server restart was not a major factor in guild renown lag, at least according to folks on the forums and in-game who reported it to us regardless of time since the past reboot. Since we're looking to specifically get feedback on guild renown lag, we'd love to hear from folks who have run Legendary Shroud, Defiler of the Just, or any other group situation where they previously experienced guild renown lag, with their thoughts on how things behaved. We would love to know whether your experience changes in regard to guild renown lag over time as well, but please don't wait several weeks to tell us how things went tonight, ya know?

  • 06-21-2016 Update 31 Patch 3 Release Notes
    • The implementation of a 50 mail limit per character in the in-game mail system. Players with more than 50 mails will be unable to send mail or post/bid in the auction houses. This is being done to improve game performance. The improvement will be seen when players reduce the size of their inboxes. At this time we are not deleting mail.
    • Upcoming Change to In-Game Mail

Tips, resources and other forum threads[edit]

Retrieved from "https://ddowiki.com/index.php?title=Talk:Game_performance_issues_after_2016_March_data_center_move&oldid=364705"