JISC

Summary

The University of Edinburgh’s Advanced Computing Facility is cool for most of the year. The Scottish climate allows them to ‘free cool’ the water for air conditioning, drastically cutting energy use and fuel bills for their new HECToR supercomputer.

The challenge

HECToR (High End Computing Terascale Resources) is a vast computing facility housed in the University’s Advanced Computing Facility. The building was originally constructed in 1976 to house an early mainframe computer – when energy demand was half a kilowatt per square metre. The new HECToR facility, which enables researchers to carry out jobs using high powered computing, uses seven kilowatts of power per square metre. A standard computer room uses enormous amounts of energy moving air around to keep machines cool, so finding an efficient and effective way to cool the increase in energy for this large-scale facility was a challenge for Facility Manager Mike Brown and the University’s Estates team:

‘In a conventional computer room, air is cooled, passed under the floor and through vents into the room, then exhaust air is vented straight back into the room. That warmed air preheats some of the cooled air, so you get a sort of ‘short circuiting’. You can offset that by trying to concentrate the hot and cool areas of the room, but air is very dynamic and you can’t effectively partition it off. So you get inefficiencies.’

Current trends for alternating hot and cold aisles are more efficient, but if there is a mix of equipment in the room, as there is in Edinburgh, it oversupplies some machines and undersupplies others with cold air.

The innovative approach

To solve these issues, Mike’s team needed to move from having DX (direct expansion) refrigeration units in the room to a chilled water system. With the bulk of the cooling equipment in a separate room, and using chilled water instead of DX, the energy and running costs can be greatly reduced.
Water cooling is becoming popular because of its efficiency – it can extract ten times the amount of heat than air, and the volume of air needed to remove the heat is far greater that the volume of water that would be required. Using water for cooling needs a central chiller plant, with a higher capital cost and requiring a large physical space.

The plant room for HECToR is one and a half times the size of the computer room. Water, expensive electronics and high power electrical supplies are not a good mix, so a great deal of care is needed. The cooling air is drawn into each machine from under the floor, at 13˚C, and vented out the top at around 42- 45˚C, into a lowered ceiling space – so that very little hot or cold air escapes. The hot air is captured in the larger than normal (400mm deep) ceiling cavity, and cooled by air handling units that only operate in that space. These cool the air using chilled water that goes into the units at 8˚C, and returns at around 14˚C. This warmer water then goes to the plant room, where large chillers work to bring it back down to 8˚C.

The refrigerant in the chillers then needs to be compressed, to condense it back to liquid form. That is achieved using another loop of water, which is pumped through the chiller and then taken up into heat reject units on the roof of the building.

The compressors use a lot of power. The best way to reduce this is to cut the load on the chiller by ensuring the water returns to the chiller at less than the standard 14˚C. ‘14˚C is an interesting temperature,’ Mike says. ‘According to the information we got from the Met Office, roughly 70% of the time at this location, 56˚ North, the outside temperature is below 14˚C and for considerable parts of the year it’s substantially lower. So what we decided to do was, under certain conditions, instead of sending the return water direct to the chiller, we would send it up to do a little tour of the roof first.’

Results and benefits

The more the temperature in Midlothian falls, the greater the benefits seen. As soon as the thermometer hits 13˚C, a set of valves opens automatically and the water is diverted through coils in the rooftop cooling units. 25,000 litres of anti-freeze make sure it doesn’t freeze while up there.
The new water cooling system reduces the cost of power consumption and carbon emissions by around 25%. The savings achieved run into hundreds of thousands of pounds, more than enough to offset the extra cost of the free cooling units. Mike says: ‘Around Christmas last year, for nearly two weeks, the chiller didn’t run at all. It was actually quite scary, when you’re used to the sound of a plant room!’

Further Information

A longer version of this case study was originally produced by Grid Computing Now! and is available from the SusteIT project website.