2003-02-20: Sunsword on Age of Shadows

Sunsword on Age of Shadows

Feb 20 2003 1:10PM

This was originally posted to Development at uo.com.

The Age of Shadows launch has obviously been a hectic one. I’ve put together this State of the Service address to give you an idea of how we got here, what’s currently going on, and what the near future will bring. Our Online Community Relations team has been posting on individual items, so I may not cover everything explicitly but I do want you to know that we are working to fix things as quickly as possible.

The UO team knew that the AoS expansion was going to be a big one from very the beginning. The feature set was well received and everyone was really excited about the possibilities. What we didn’t know is how big it was going to be! We planned for increased memory usage on the hardware, and we upgraded almost all of our servers to their physical maximum for memory. We spent a lot of time upgrading the housing database and server applications to be able to handle the higher load. But it wasn’t until the weeks before launch that the real implication of this expansion’s success began to come to light – and even in the last week before launch, our numbers rose significantly higher. Our pre-orders soared to company record levels.

We looked at our capacity for handling logins and registration, and felt that, while stressed, it should be able to handle the load. As it turns out, we were wrong. It took us 2 or 3 days of working 24/7 to iron out all the capacity issues regarding registering and logging in after our North America launch. We built up the login servers beyond their normal maximum capacity, and for the next 4 or 5 days the login servers were slow, but many players were definitely getting through. I personally spent several hours touring different shards and checking out some of the customized houses.

Despite all the upgrading we’ve done, the launch of AoS in Japan has once again put us beyond our max capacity for the login servers, and we found some new bugs in the system that were not revealed until we hit such high usage. We’re still working on the problem, and we also expect the peak usage to drop off over the next few days.

Beyond the log-in servers, our biggest priority right now is shard stability. As our shards hit their peak loads, we found a problem with our shard memory usage, which can cause a server to become unstable after a few hours and then restart. We have partially addressed that problem already, and now the servers are quite a bit more stable, but we still have one major memory problem that we are trying to solve. We’ve isolated the area where the problem occurs, and are working right now to fix it. Again, these problems are related to the load exceeding our forecast, which is both a blessing and a curse.

We’re also fully aware that the memory issues on the shards are more than just an inconvenience – they’re also causing some people to have major problems getting through the long wait time in the GM queue. We’re working on this in two ways – one, we’re fixing the memory issues at the core of the problem. And two, we’re working on two of the biggest call generators right now, which are “black holes” (see below) and issues with the Hooded Shroud of Shadows. I understand that it is frustrating to wait in the GM queue for several hours, only to get kicked out when a server goes down. Again, stability is our top priority and we will fix this as soon as possible.

Next in line to the shard stability problem are “black holes” as they’ve been nicknamed. A “black hole” occurs when the server crosses a threshold of network traffic being sent to the client, usually related to a large number of items in the area. The player gets disconnected and when they try to reconnect, the same problem occurs. The result is that the player is stuck. Fertbert, one of our server programmers, is working on this problem full time (he managed the Japan launch yesterday). We have a temporary solution in the works that will “telestorm” any player that gets stuck in a black hole, but this is not a permanent fix and we’re still working on it.

There are definitely other issues that we are working to fix, especially those related to housing and vendors, but that’s not the point of this message. I want to make it clear that we are listening to you and paying attention. We have been working literally 24/7 for a few weeks on the game and will continue to do so until we have these primary issues resolved. We’ve said many times that the beauty of an online game is that we can continue to grow, and that’s what Age of Shadows will do for UO. Granted, there are bugs, and we’re fixing them. We’ve experienced problems keeping up with the traffic that’s hitting us right now, but these problems are temporary.

I know many of you are already playing and enjoying the new expansion, and our goal is to get all of you to that point. We won’t rest until then. After that, we are going to get back to the nitty-gritty of fixing long standing issues, and adding features iteratively for a while. I think the pattern of seeing nothing for a few months and then having a huge change dropped on the game is going to go away for a while. We’re going to be asking you what you want, talking to you about it, and then giving it to you. But that’s a little ways off. For now, we’re committed to Age of Shadows, and we’re working to make sure every one of you gets a chance to experience it its full glory for themselves.

Thanks for your time; I’m sure you have many additional questions and comments.

Anthony "SunSword" Castoro

Producer, Ultima Online

References: