Fullscreen HTML5Med Res (??MB)HTML4

By Geoff Graham The Pawsey Supercomputing Centre Just what is a supercomputer? How does it work? What do you use it for? We take you inside the Pawsey Supercomputing Centre to meet Magnus, the fastest and most powerful supercomputer in the Southern Hemisphere. F ROM THE OUTSIDE, the Pawsey Supercomputing Centre appears rather modest, just a low building set into the hillside. Located in Technology Park in the leafy suburb of Kensington in Perth, Western Australia it houses two supercomputers, a host of supporting computer systems and a huge data-storage facility. The statistics are impressive. The smaller supercomputer called Galaxy can perform at 200 teraflops – a teraflop is a million million floating point calculations per second. So it’s no slouch. However the star is Magnus, a Cray 20  Silicon Chip XC40 supercomputer capable of 1.5 petaflops. A petaflop is a thousand million million floating point calculations per second and this makes Magnus the most powerful public research computer in the southern hemisphere. There may be a more powerful computer in existence (who knows what ASIO have hidden behind their walls) but this is definitely the fastest publicly acknowledged computer. Everything here is big; the storage systems can store 70 petabytes of data with an expansion capacity to 100 petabytes. The power consump- tion is around 900MW and water is drawn from underground to keep the supercomputers cool. In the beginning The Pawsey Supercomputing Centre was named after pioneering Australian radio astronomer Dr Joe Pawsey. It started life in 2009 with an $80 million grant from the Federal Government, in part to support Australia’s push to be the southern hemisphere’s site for the Square Kilometre Array radio tele­scope (see SILICON CHIP, December 2011 & July 2012). siliconchip.com.au The Pawsey Supercomputing Centre in Technology Park, Perth, Western Australia houses the fastest supercomputer in the Southern Hemisphere. The building was designed to merge with the landscape and reflect the geosciences, a major user of the supercomputers. Photo credit: Pawsey Supercomputing Centre. The centre opened in 2013 and still processes a lot of data from the radio telescope but the rest of its capacity (about 75%) is dedicated to the five partners operating the centre (the CSIRO and four WA universities) and researchers in Australia in general. In some respects the Pawsey Centre is unique because they not only provide the computer facilities but they also train and help researchers to get the best results from the system. The centre also has a number of systems dedicated to visualising the data so that researchers can watch the result of a simulation and that makes understanding the data much more intuitive. For example, a geologist would normally take core samples in the field and then analyse these to try to map the ore deposit. By using the Pawsey supercomputers, they can go much further and calculate the distribution of the sample results through the geology of the region. In addition, by using 3D glasses along with the visualisation technology, they can stand inside the ore deposit and look around to see how it is distributed. Inside Magnus So just what is a supercomputer? siliconchip.com.au This is Magnus, the fastest computer in the Southern Hemisphere. The cabinet artwork is by Margaret Whitehurst and pays homage to the centre’s close connection to the north-west of Western Australia. It has been designed to reflect “the ground below”, in reference to geoscience, one of the areas the super­ computing centre supports most closely. Photo credit: Pawsey Supercomputing Centre. These days, it is basically just a massive set of processors that work on a problem in parallel. In the case of Magnus, that is 35,712 processing cores. The processors are standard Intel Xeon E5-2690V3 Haswell units. Each processor has 12 cores running at 2.6GHz and two processors together with some local memory make a node, which is the basic computing element. Four of these nodes are physically mounted on a blade which is a large plug-in module and is the replaceable part in the case of a component failure. Magnus consists of eight cabinets with each cabinet holding 48 blades for a total of 1488 nodes (35,712 processor cores). You might think that you could build a supercomputer like this using a lot of standard desktop motherboards but that wouldn’t work when scaled to the numbers required by Magnus for scientific workloads. Removing heat is one issue but also getting the data to each processor requires a special network. Interconnecting the nodes In Magnus, the nodes are interconnected with 4km of high-speed optical fibre and copper, making a network July 2015  21 Galaxy is the smaller of the two supercomputers. Its primary purpose is to process data coming from the radio telescope arrays (ASKAP and MWA) in the Murchison in the north of Western Australia. When they are running, these telescopes generate an amount of data equivalent to one DVD every two seconds and this data needs to be processed in real time. Photo credit: Pawsey Supercomputing Centre. keep the specialised hardware and operating software running smoothly. Interestingly, the management software for Magnus (called SLURM) runs under Linux. Linux and SLURM run on a specialised processor and are responsible for distributing work to the various compute nodes. So in practice, the researchers and technicians running the supercomputer interact with Linux – originally a hobby project by a young lad in Norway. Because a supercomputer is a scarce resource (there are not many around) getting time on it takes some effort. The researcher must make a proposal which is assessed by a committee who consider the scientific value of the application and the processing time that it would require. Once over that hurdle, the program must be prepared and queued for processing. It is rather like the old batch systems of yesterday; you submit the program and data and wait for a processing slot. However, it is worth it – in just one hour, Magnus can do more work than a conventional computer could do in two years. Using a supercomputer The layout of the Pawsey Supercomputing Centre. The large white space at the top is the supercomputer cell, below it is the I/O cell, and the lower white space is the tape cell. Each cell has different temperature and humidity requirements, differences between water cooling versus air cooling, and differences in whether mains power or uninterruptible power is used. The supercomputer cell is primarily water-cooled and on mains power. Photo credit: Pawsey Supercomputing Centre. capable of an aggregate bandwidth of over 100,000 gigabits per second. The network (called Aries) runs a special protocol designed to keep latency low. Local storage for Magnus is three petabytes with a sustained read/ write performance of 70GB/s. This is just used for temporary storage with 22  Silicon Chip the end results going to a separate 70 petabyte storage system maintained by the centre. Managing Magnus Magnus was built by Cray Inc in the USA and two Cray engineers are located permanently on site to help Because a supercomputer is a massively parallel machine it tends to work better at some jobs than others. These include simulations of physical systems, image processing and geophysical mapping. A typical application that works well is atmospheric modelling. In this, the atmosphere is divided into cubes of a few kilometres in each dimension. Each processor in the supercomputer is allocated the job of simulating the changes in one cube and while it is doing that, the other processors are working in parallel on other cubes. When one processor has finished, its will be allocated a new cube to process. Because there are many, many cubes, all the processors in the supercomputer will be busy for some time. The results of each simulation are then aggregated to gain an image of the whole system. Similar approaches are used to model ocean currents, star formation, the generation of tsunamis, investigate the electromagnetic structure of matter and more. Some more novel uses of the Pawsey supercomputers have been sequencing the genome of the cane toad and investigating the porosity of bread. This last siliconchip.com.au Helping to put you in Control Programmable Step Pulser The KTA-301 provides signal to control speed, acceleration/ decleration rate & direction to a stepper via DIP switches. 2 potentiometer input for speed & acceleration/deceleration control. 8 to 30 VDC powered, DIN rail mountable. SKU: KTA-301 Price: $89 ea + GST 2-Switch Button Control Box The Cray supercomputers use four kilometres of fibre optic cables (shown here) and copper cables to distribute data to the processing nodes. The network, called Aries, runs a special protocol designed to keep latency low. Photo credit: Pawsey Supercomputing Centre. Red, black, 2-switch push button control station contains 1 x NC contact black pushbutton ad 1 x NO contact red pushbutton. With addition of a contactorrelay the user can use this control station as a direct on line (DOL) motor starter. SKU: HEE-020 Price: $27.50 ea + GST TagTemp NFC Data Logger TagTemp NFC temperature data logger with 1 year (typical) life. LogChart-NFC an android smartphone app allows configuration and data download via the NFC link. Measure range between -40 °C to +70 °C. SKU: NOD-060 Price: $85 + GST Teensy The teensy is a breadboard free development board with a 32 bit ARM Cortex microprocessor and Arduino-like programming. It features; 64K RAM, 34 I/O pins, 12 PWM outputs, 3 UARTs, SPI/I²C/ Can Bus. 3.3 VDC powered. SKU: SFC-012 Price:$29.50 +GST Differential Pressure Transmitter This photo shows a Cray X40 supercomputer blade which is the replaceable module in case of component failure. Each blade holds four nodes and in Magnus each node consists two standard Intel Xeon E5-2690V3 Haswell processors with 12 cores running at 2.6GHz. The total number of processing cores on a blade is 96. Photo credit: Cray Inc. one might sound silly but it is quite important to Australia as the international perception of Australia’s wheat is that it is not suitable for bread making. Researchers at the CSIRO’s Food Futures National Research Flagship used X-ray micro-tomography to examine the structure of bread and the resources at the Pawsey Supercomputing Centre to visualise the structure. With the knowledge gained, it is hoped that future research will help improve Australia’s standing in this important market. Another unusual application is the Sydney-Kormoran Project which is processing images from the WWII shipwreck sites of HMAS Sydney and siliconchip.com.au HSK Kormoran. The aim is to provide a moveable 3D image of the two ships resting on the ocean floor for researchers and the public to examine. Huge amounts of data With all this processing, there is a lot of data, especially from the radio telescope arrays (ASKAP and MWA) in the Murchison in the north of Western Australia. When they are running, these telescopes generate an amount of data equivalent to one DVD every two seconds and the data needs to be processed and archived in real time. One supercomputer (called Galaxy) is dedicated to this task, with the data saved onto a sophisticated storage sys- IP54, DPS series differential pressure transmitter has a 0 to 1 mBar or 100 Pa input pressure range and loop powered 4 to 20 mA signal output. 10 to 30 VDC loop voltage. SKU: DBS-5501 Price: $199.95 ea + GST Waterproof Temp. Sensor DS18B20, digital thermometer in waterproof 6 × 30 mm probe with 15 metre cable. -55 to 125 °C range with ±0.5 °C accuracy from -10 to 85 °C. 5 VDC powered. SKU: GJS-002 Price: $19.50 ea + GST PSU With Battery Charger DIN rail, power supply with battery charger (UPS function). Provides AC fail and low battery alarms. 90 to 264 VAC input, produces 13.8 VDC <at> 4.5 A output. SKU: PSM-1171 Price: $99 ea + GST For OEM/Wholesale prices Contact Ocean Controls Ph: (03) 9782 5882 oceancontrols.com.au Prices are subjected to change without notice. July 2015  23 probably be still held on a spinning disk, so it would be returned immediately. But if they requested some very old data, the chances are that it would have been archived and a robot tape arm would then swing into action to retrieve the right tape and place it in a drive. In that case, there would be a delay of some minutes before the data is returned but other than that it would be no different from reading any other file. Tapes may be regarded as low technology and many would ask why not just use more disk drives and do without the complex tape system. The reasons are capacity, power and heat. The existing system has a capacity of up to 100 petabytes which would require an unimaginable number of disk drives and even if they were used, the power and cooling requirements would be unsustainable. Power A view inside the robotic tape library. The large black column in the centre is the robotic tape arm. This travels up and down on rails between the tape cartridges on either side, retrieving cartridges and delivering them to tape drives. The robot system is completely automated and looks like a very large disk drive to the supercomputers. Credit: Pawsey Supercomputing Centre. tem that is also used by Magnus and other systems in the centre. This storage system looks like a large single disk drive to the rest of the facility but is in fact an array of spinning disks which act as a cache to a large tape library managed by robots. The disks are high-reliability versions of the standard disk drives that we all have in our computers and on their own add up to six petabytes. The operating software distributes the data over the drives so that if one or more fail, the missing data can be reconstructed. The operating software is also responsible for automatically archiving little-used data to the tape library. This consists of robot arms which retrieve tape cartridges from storage and place them into tape drives so that data can be written and read by the main system. The software keeps track of what piece of data is written onto which tape at what position. In total, the tape system can hold up to 100 petabytes. All this is transparent to the rest of the system. A researcher could request some data that is recent and it would Gigaflops, Terabytes & More The basic units used in the supercomputing world are gigaflops, teraflops and petaflops for processing capacity and terabytes and petabytes for storage. Giga means a thousand million or 109,while Tera means a million million or 1012 and Peta is a thousand million million or 1015. A FLOP stands for floating point operations per second. Most scientific computing involves manipulating floating point numbers which is why this measure is used. Note that benchmarking computers is a tricky business and the claimed numbers can vary considerably depending on how someone runs the benchmark or calculates the result. By way of comparison, a typical dual core processor on a laptop or desktop computer would have a theoretical maximum performance of about 20 gigaflops. 24  Silicon Chip The overall supercomputing centre draws about 900MW from the Western Australian grid. You might think that a lot of this has to be backed by a UPS and diesel generators to keep the super­computers running but that is not so. Some systems are protected by a UPS but the supercomputers are not. This is because firstly they draw such a huge amount of power that a properly-sized UPS would be prohibitively expensive. The second factor is that, by its nature, a supercomputer does not need to be kept running during a power blackout. It does not store much data and any interrupted jobs can be simply restarted when the power is restored. The centre does have a system to protect the supercomputers from glitches on the power line though. This consists of a large electric motor driving a flywheel which is in turn connected to a generator. If there is a glitch in the power, the momentum of the flywheel will keep the generator running and insulate the supercomputers from any ill effects. Groundwater cooling An intriguing feature of the Pawsey Supercomputing Centre is the cooling used for the supercomputers. Magnus alone generates about 400kW of heat and a cooling system for that heat load would be expensive to provide and operate. In a world first, the CSIRO Geothersiliconchip.com.au In a world first development by the CSIRO Geothermal Project fresh water is drawn from the Mullaloo Aquifer 100 metres underground to cool the supercomputers. This water is at a constant temperature of 21.5°C and after doing its job is returned to the aquifer at about 24.5°C. Given the size and depth of the aquifer the effect on it is minimal. Credit: Pawsey Supercomputing Centre. mal Project developed a system where­ by cool water is drawn from an underground water body called the Mullaloo Aquifer, about 100m below ground. This water is at a constant temperature of 21.5°C and after being used to cool the supercomputers is returned to the aquifer at about 24.5°C. On a particularly hot day, a second return system is used but the effect on the underground water system is minimal. To further bolster the system’s green credentials, the pumps used to move the water are powered by solar panels on the roof of the centre. The power savings are significant but the most important factor in water starved Western Australia is the saving of approximately 14.5 million litres of water every year, compared to a conventional system using evaporative cooling towers. As part of the research involved in this project, it was discovered that the underground aquifer is slowly moving and in a 100 years or more it will have passed from under the supercomputer centre. By then, computers and cooling requirements will have changed so this siliconchip.com.au HMAS Sydney which was sunk with all lives in a battle with the German aux­­iliary cruiser Kormoran which was also sunk during the engagement. The Pawsey Supercomputing Centre is processing photographs of these ships lying on the seabed to create a moveable 3D image for researchers and the public to view. was not considered a concern. This system of cooling involved cutting edge research and has attracted world-wide attention. Unfortunately, funding for the group responsible was terminated with the change of government in Canberra and the expertise has dissipated. Such are the ups and downs of a publicly funded SC organisation. July 2015  25