2004-01-28: Brandon Williams Comments on the Evolution of UO

Brandon Williams Comments on the Evolution of UO

Jan 28 2004 4:46PM

My name is Brandon Williams. I am a Sr. Unix Administrator for Electronic Arts/OSI. I am considered somewhat of an old timer around here in the OSI office, as I have been directly involved with building and supporting Ultima Online production and OSI development environments for the majority of the past seven years. I have seen UO go from a "proof of concept game," that was hopefully capable of lasting a few months, to a staggering alternate universe successfully capturing the interest of millions of avid players for over six years now. The evolution of UO is truly a remarkable story. If you have a moment, and an inkling of interest, please feel free to read on as this relatively dry Sys Admin tries to spin a tale.

1996 was a time of change, both for me and for a relatively small project in development at OSI code named "Multima." It was around this time when public beta for the project was getting ready to start. I was not yet a member of Operations, partially because there was no Operations team in place at that time. I was working part-time at OSI and majoring in computer science at school. As an avid gamer and office peon desperately looking for ways to get closer to game development, I volunteered to manage the influx of beta tester applications for Ultima Online. Little did I know, this would mean I would be opening thirty thousand letters, removing the two dollar shipping fee that applicants were sending to cover the cost of shipping their beta CD, and manually entering return shipping addresses into a simple flat database! The public's interest in UO's open beta program was overwhelming at times, but also extremely exciting. Around this time, a member of OSI's local IT staff, Mark Rizzo, broke out of his everyday duties to focus on the infrastructure that was to support UO.

The initial plan was to run the entire service on a cluster of "powerful" dual processor Pentium Pro 200Mhz servers with 128MB of RAM--all from the "high tech" converted office/server room in our development studio! (Please excuse my facetiousness. It's just that looking back now; it's funny to think what we initially thought it would take to run the UO service.) With public beta ramping up, and a vast amount of work needing to be accomplished, Rizzo was able to justify the need for a junior-level sys admin position to help with the launch and support of UO. In hind site, I realize I am very fortunate to have jumped into this role full-time when I did. Rizzo was an exceptional mentor. I learned many aspects of sys admin work by working with him. Together we were discovering the trials and tribulations of launching and running an online service. Things were very different at that time. There was not an abundance of hardware venders in the market providing a state of the art X86 server platform. Instead, we were faced with building customized full tower case servers from scratch. ISPs were in an infant state. What is typically expected in a co-location center now was in the process of being defined when UO was getting ready to launch. Luckily for UO, big brass from the EA ranks were aware of the mammoth interest that was building for UO and had the foresight to open wide the checkbook and support the project as needed for an appropriate launch.

In the very late stages of development, it became painfully apparent that the X86 platform was not going to efficiently support the game, and that we were not going to be able to support all of UO's customers in one single instance of the online world. This is when the idea of "a shard" was created. Fiction was put together to support this need, and hence, we now know of "Mondain the wizard" and the gem that shattered creating hundreds of like instances of the Britannia universe. Within the first few frenzied months of UO's launch, the Pacific, Baja, Napa Valley, and Sonoma shards were opened on the US West Coast; Great Lakes and Lake Superior were centrally located in Chicago; and the Chesapeake, Atlantic, and Catskills shards were located on the East Coast. All of these shards were running on more powerful (yet more expensive) Sun Sparc servers.

With UO growing by leaps and bounds, EA and the UO support staff had to adapt quickly. It was not long before the Operations team began to hire on some of the industries best talent. Pretty soon, our operations team had software engineers specializing in login and billing systems, network architects specializing in Internet routing, web administrators mastering one of the most impressive dynamic web sites around, and, of course, more Unix administrators to support the growing platform needed by Ultima Online.

Soon, EA began to tap foreign markets as well. Interest in UO from Japan, Europe, South Korea, Taiwan, and Australia was and still is staggering. To this day, our non-USA customer base makes up about half of our active players. Throughout the life of the service, Operations has added many shards, in several regions around the world. UO has benefited from several milestones and improvements in its life. Many champions have done great things for this game, whether it was a programmer developing distributed backups, significantly reducing backup times, or an architect sponsoring major hardware upgrades at various stages of this game's life. It is truly amazing how many developers and support staff UO has had over the years making the service what it is today.

Looking forward, Operations knows there are many improvements to be made still yet on the infrastructure that supports UO. The day when shard crashes are a thing of the past is a goal of ours. Reworking the in game backup system to make reverts significantly rarer--and no longer than a few seconds when there's no way to stop them from occurring--is on the radar. Improving the server platform and our Internet connectivity to make latency a forgotten word is a utopia that our Operations team is working hard to achieve. Operations can not achieve all of these goals alone. Many other teams within the UO support structure and the Internet as a whole have a large say in these goals as well. They are common goals though, and they will be achieved.

So that is my tale. (At least that is all I have time to tell right now.) Thank you for your interest and, more importantly, thank you so much for your continued dedication to the amazing world of Ultima Online! EA, OSI, and everyone here who directly supports UO know we're lucky to have such a dedicated and truly creative player base, and that you are a major factor in the continued success of this online universe. For that, my colleagues and I are extremely grateful to you. Keep having fun, and keep driving your high expectations. Hopefully together we can one day enjoy UO's 20th anniversary and beyond.

Brandon Williams

Sr. Unix Administrator

Origin Systems