Melbourne 2006 Commonwealth Games
What is the Melbourne 2006 Commonwealth Games Data Network?
Systems Integration Services for the M2006 Games Data Network (GDN) is a large scale project spanning 16 months with numerous interfaces and dependencies to other M2006 Technology Programs. APCS is responsible for the full lifecycle phases of design, build, test, deploy, support and de-commissioning of the technology assets - and, of course, the overall governance of the project.
The GDN is the enabling infrastructure that inter-connects all Melbourne 2006 Commonwealth Games technology systems, applications and services and includes a highly distributed architecture of more than 90 high-powered servers, 20TB of data storage, 180 network switches and security appliances, 1800 desktop and notebook computers and more than 800 reprographics devices.
This distributed architecture is deployed upon an Internet Protocol (IP) wide-area network that is both high capacity and highly-available for linking the 35 competition and non-competition Commonwealth Games venues.
Engagement Approach
(throw away most of the standard rules, this is radically different……….)
Four simple principles have become the cornerstone of the APCS engagement approach for the GDN project:-
- We use simple, effective tools and processes.
- We continuously break-up work effort into smaller tasks with rapid review cycles.
- We use risk management tactics to mitigate against shifting or changing requirements, and
- We ensure all dependencies are researched, addressed and validated.
With any large, long duration project that has numerous absolutely time-critical milestones, the first step in undertaking the execution is to establish a Project Management Office (PMO). The PMO provides the systems, tools, processes, resources, roles, responsibilities, control points and review cycles that are essential for governing the project in accordance with our well established standards.
We actively measure our performance weekly – this enables us to continuously break down work activity into smaller and smaller tasks that can be commoditised, tactically executed and then compared back to planned schedules for optimisation. The importance of this approach can’t be understated given that 15-26 March 2006 are absolutely immovable dates!
Running in parallel with the GDN, and at varying stages of progress, are all the other M2006 programs. These are the consumers of the GDN and drive the requirements for range and reach of the network. There is no doubt that securing firm and committed requirements early (or as early as possible) in the planning cycles is the single biggest and most important challenge with organising a complex event such as the Commonwealth Games. The APCS team has needed to maintain a delicate balance of diplomacy, negotiation, risk consideration coupled with good, tight and effective controls over change management to minimise the re-work or the possibility of delivering a service that does not meet all requirements.
Other factors that contribute to our engagement style are the number of stakeholders and M2006 external providers who are contracted to contribute to the various areas of the technology landscape. This aspect in particular demands careful dependency planning, addressing, checking, testing and review to ensure that the supply chain works efficiently.
Does a Design ever stand still……
Officially, February, March and part of April 2005 was the period set aside for the development of the conceptual design of the Games Data Network, the creation of Security Policies and the progression and evolution of documented requirements and assumptions into the aggregation of detailed venue design documents.
The outputs from this phase were used to generate the specification and bill of materials for the various technology equipment assets used in the GDN solution. By consolidating requirements and standardising on core sets of technology, M2006 was able to initiate their technology procurement processes.
The GDN design had to take a number of factors into consideration. In particular, redundant componentry has been incorporated into critical areas such as at the field of play, local results rooms at venues, transmission of data to outside broadcast vans, core switching within the venues, central results, commentator information systems and games information systems.
The APCS design also needed to cater for the unique or special requirements of external providers that were delivering specific areas timing, scoring and results systems.
In a dynamic environment such as M2006, the most acute risk in putting down designs early in the overall planning cycle is the obvious opportunity for requirements to change. Generally this can be due to the rest of the organisation not reaching similar stages of completion in their own programs at the same time. Strange as this may seem, there can be valid reasons for this to occur, in particular due to the various factors and situations outside of M2006’s direct control.
To mitigate against these risks, we have enhanced our normal project delivery approach by developing firm strategies for the selection of technology, the minimisation of equipment types and the development of various prototype solution models that enable either consolidation or distribution deployment scenarios to be mixed and matched. In other words, we were able to “design in” a great deal of flexibility within a strict technology framework.
Quality, Consistency and Commoditisation – The “Maccas” Formula…….
From May through July 2005, technology equipment began arriving from M2006 suppliers. The APCS Build-Integrate-Test (BIT) Lab was established off-site to provide a large area that could be physically divided into various zones which were designated with venue names. This approach enabled our GDN team to completely build all server and switch infrastructure within the lab environment. Once done, we were able to inter-connect all the venue equipment to simulate the full breadth of the Melbourne 2006 Commonwealth Games Data Network.
Critical to the success of the Build-Integrate-Test cycle was our upfront investment in the development of automated practices for the majority of tasks, reducing the chance of error. This enabled us to commoditise the build processes in order to achieve the highest possible standards of efficiency and quality.
All phases were validated using a highly modular testing methodology that enabled the re-run of any test in the event of failure without a incurring a high time/cost exposure.
Governance of build-integrate-test involved the deployment of monitoring and management tools and separating the teams that checked and compared results. In addition to the tools were a range of manual processes that validated results and bar-code scanning of status at each phase to ensure consistency and completeness of every step. This enabled us to maintain an “at-a-glance” knowledge of current and forecast progress.
The build-integrate-test phase for all servers and switches continued from June through to early September 2005.
The equipment is now fully operational and awaiting the preparation of facilities within the venues prior to deployment.
Getting in early to beat the mad rush expected near the end………..
Each of the M2006 venues can contain a broad range of technology equipment including CATV, voice and data communications from telecommunications providers, fibre-optic cabling, copper cabling as well as specialised cabling and equipment for needed for timing systems and scoreboards. Much of this equipment needs to be consolidated at a core physical location that can be used for distribution within the venue and subsequently connected to the network with the other GDN venues.
Whilst the events are only using the venues for a short-term (1-3 months), the standards of the technology deployments are being undertaken with the highest professional levels in order to reduce the risks of technology component failure. Therefore the installation works are of comparable quality, fit and finish to any permanent long-term fit-out.
Interestingly, all voice and data cabling (copper and fibre-optic) at the venues is being installed new as dedicated infrastructure for the Commonwealth Games. This means that existing venue infrastructure is not to be used. Believe it or not the majority of this new M2006 infrastructure will need to be removed at the completion of the Games!
In most cases new telecommunications feeder capacity into the venues and a fair amount of new conduits and physical cable pathways within the venues are required to accommodate the cabling systems installations.
The concept of a Technology Equipment Room (TER) was developed as an important shared facility that is highly secure and environmentally robust. From mid-October to early January 2006, all venue TERs will be installed and servicing the GDN.
The commissioning of TERs involves a number of different phases including negotiation for space, building works (either to bring an existing space up to M2006 standards or, in some cases, to deploy a completely new structure), installation of cabling, power and communications, installation of equipment racks, UPS, etc. Only then can we deploy server and switch equipment and commission the wide-area networks.
Once a venue TER is ready to receive equipment, the APCS team develops a detailed minute-by-minute run schedule to co-ordinate the numerous logistical activities. This is communicated to various parties to ensure all are aware of dependencies. The pre-tested technology assets are carefully transported from the build-integrate-test lab to the venue TER and re-commissioned and tested for operation on the GDN.
Venue cabling works are an ongoing activity scheduled over many months and dependent upon site access, etc. Distribution network switches are planned to be commissioned at different times based upon the readiness of cabling and facilities furniture and extras.
To preserve the integrity of the GDN and to incorporate late service planning requests such as changes within the Venues – team relocations into different spaces, etc. all Venue distribution network switches are installed with the core when the TER is brought online within the Venue. This process makes TER deployment marginally more complex; however, it then becomes simpler to implement successful change controls and will speed up the Venue Distribution point commissioning process by relocating the network switch from the TER to its intended destination on-site.
The purpose of installing all Venue TERs early is to ensure that all environmental conditions are settled and stable at the Venue for a sufficiently long period (1-6 months) and to mitigate against the risk of site-specific (Venue access, physical constraints or other local Venue factors) or Provider delays that may have impacted against further downstream or dependent Venue deployment and operational activities.
In some cases, M2006 teams are relocating to Venues in late 2005 for early operations.
The Next Steps
Venue TER deployments are scheduled from now until early January 2006 and will consume a decent share of the attention and focus from the APCS GDN team. Whilst the process is low in complexity and essentially a mechanical exercise, it does require a fair amount of logistics planning in order to execute efficiently.
At the same time, planning for PC and printer deployments at some Venues is being progressed. From the middle of November through to mid December, early deployments for the supporting Programs such as Uniforms and Accreditation, Games Warehouse, Ceremonies teams and the Athletes Village Team will commence operations at their Venues with M2006 staff relocating from the Games Headquarters.
Test events are an opportunity to validate the integration of various provider systems and are a true benchmark of capability and operational readiness. A number of formal test events are scheduled from November 2005 through to February 2006 and despite these events having no commercial incorporation with M2006, e.g. they are standalone events, through sporting associations M2006 have secured privilege to be involved in these events at varying levels ranging from “shadowing” the event through duplicated processing to partial live scoring or results collections on M2006 systems.
In parallel to device deployments at Venues and the Test events, the GDN teams will also be validating disaster recovery strategies, preparing for the migration into support mode within the Technology Operation Centre (TOC) used to support all technology services and the recruitment of additional Venue support resources.
