Performance Evaluation of Cold Plate Liquid Cooling within Data Centers under Actual Operating Conditions

1. Introduction

During the 2020 pandemic, the surge in demand for video streaming, conferencing, online gaming, and social networks led to a global internet traffic increase of over 40%. However, this was not an isolated trend: today's internet traffic is 15 times higher than it was a decade ago [1]. In this context, data centers currently account for approximately 1% to 1.5% of global electricity consumption. Up to 40% of this energy consumption stems from cooling systems, highlighting the critical need to develop new sustainable solutions.Compared to traditional air-cooling systems, liquid cooling offers numerous advantages, such as higher heat capacity and lower energy consumption for heat transport. The effectiveness of liquid cooling technology for server cooling has been extensively validated through both immersion and indirect cold plate (also known as direct-to-chip cooling) systems [5]. Using indirect single-phase liquid cooling with cold plates can increase the coolant temperature and enable waste heat reuse for other purposes, further enhancing data center efficiency. With its superior heat transfer capability over air cooling, liquid cooling presents a promising pathway to address the thermal management challenges of data centers. Leveraging the excellent heat absorption properties of liquids, this technology promises significant reductions in energy consumption, alleviation of cooling-related bottlenecks, and paves the way for more sustainable future operations of data centers [7]. However, the adoption of liquid cooling also introduces a series of technical, economic, and logistical considerations that must be thoroughly explored to ensure successful implementation.This study describes a pilot test implemented in a live data center to evaluate a direct-to-chip liquid cooling system developed by Universal Smart Cooling SL (UniSCool) [8], [9]. The energy consumption results under specific CPU temperature limits were compared with those of the existing air-cooled system.                   

Kres(Suzhou) Power Systems Technology Co., Ltd is an emerging company focused on fluid power solutions, and serve for the markets of Energy, Construction, Agriculture, Manufacturing, Transportation, Medical, Data Centers, etc., and was founded in 2022 with leading-edge engineering capabilities and innovative products to keep up with the changing customer needs. Including OCP standard UQD, UQDB, BMQC, manifold, EPDM hose and hose assemblies, FF9 full flow dual interlock quick disconnect coupling (a better replacement for FD83 series products), API heat transfer products, Gates EPDM hose and Wika products.

2. Cold Plate Prototype for Direct-to-Chip Liquid Cooling**

The designed and manufactured cooling device will be used for direct-to-chip liquid cooling of a Dell PowerEdge R740 server. This server model contains two Intel Xeon Gold 6130 processors, each with a Thermal Design Power (TDP) of 125 W and a maximum case temperature of 87 °C. TDP represents the maximum heat a component is expected to generate under high load. The CPU Integrated Heat Spreader (IHS) measures 70x47 mm², which is the surface where the heatsink or other cooling solutions contact the CPU or GPU processor.The geometry of the manifold was designed, and the selected technology was tested based on CFD simulations for this pilot project. The geometry of the cold plate used and the design of the manifold employed in the pilot project are explained below.

Figure 1. 3D schematic of the direct-to-chip cooling system with inlet (blue arrow) and outlet (red arrow) water flow.The cooling device developed by UniSCool consists of a manifold and a copper base (Figure 1). The copper base features water flow channels for cooling the CPU. The manifold and copper base are screwed together, with an O-ring placed between the two parts to prevent leakage. The thickness of the manifold was adjusted based on the curvature of the manufactured copper part and the available space within the server.

A. Copper Cold Plate Design and Manufacturing**

The UniSCool cold plate is made of copper due to its high thermal conductivity (κCu = 385 W/m·K). The copper cold plate was manufactured using milling technology by the University of Sherbrooke. The design is based on 10 channels, each with a width of 6 mm, a height of 0.5 mm, and a length of 47 mm (Figure 2). The channel width allows for the integration of UniSCool's patented heat dissipation fins between the channels, although this aspect is beyond the scope of this study.

Figure 2. Copper cold plate designed and manufactured for the current Proof of Concept (PoC).

B. Fluid Manifold Design and Manufacturing**

The base of the manifold matches the dimensions of the heatsink used in the Dell PowerEdge R740 server to facilitate the installation of the new device (Figure 3).

The fluid manifold is designed to supply coolant to the cold plate from its top surface while maintaining a compact design. Its dimensions were adjusted to fit within the server's existing heatsink footprint and it is secured to the CPU using the same brackets. It was designed using a 3D drawing program and manufactured using a 3D resin printer (Figure 3). This 3D printing technology enables waterproof components, which is crucial for liquid cooling systems – a characteristic not achievable with traditional filament printing.

The manifold is balanced to ensure equal flow distribution across all channels. Tubes connect to the manifold via 3/8-inch fittings. Therefore, the manifold is designed with corresponding threads. The collector used in the tests described below was leak-tested by submerging it in water and introducing pressurized air above operating conditions.

Figure 3. Top view (a) and bottom view (b) images of the UniSCool designed and manufactured direct-to-chip liquid cooling system.

3. Methodology

A. Cooling Distribution Unit and Heat Rejection Unit

For this study, a hydraulic loop was designed to evaluate the performance of the developed direct-to-chip liquid cooling technology under real data center conditions (Figure 4). This loop accurately simulates the operational environment, allowing for a comprehensive assessment of cooling efficiency and the practical feasibility of implementing the technology in live data center operations.A Cooling Distribution Unit (CDU) was designed and manufactured to encapsulate the entire cooling system and contain the main components required for normal loop operation. These components include:A centrifugal pump sized to overcome the loop's pressure drop and provide sufficient flow to keep the CPU below 80°C.A liquid-to-liquid (L2L) heat exchanger responsible for cooling the cold plate water as it flows from inlet to outlet.A water collector acting as a bubble extractor.A secondary water loop circulates between the chiller and the L2L heat exchanger to ensure a constant inlet temperature of 25°C, enabling effective heat dissipation. The circuit is operated using an Arduino, which functions as a data acquisition device and regulates pump power to maintain the optimal CPU temperature.

Figure 4. Schematic diagram of the designed hydraulic system.

B. Test Procedure

This section outlines the test procedure performed on the equipment described earlier. The primary objective was to evaluate the reliability and efficiency of the UniSCool device and compare its performance with the traditional air-cooling system currently used in the data center.

First, the UniSCool direct-to-chip liquid cooling system was successfully installed to cool the CPUs of a Dell EMC PowerEdge R740 server (Figure 5). Subsequently, the entire circuit was validated in the laboratory to ensure proper functionality and to identify and correct any potential leaks. Following this, the server equipped with the UniSCool cooling system was installed in a rack within the Barcelona data center. The CDU was also installed inside the rack, and the constant temperature bath (thermostatic bath) was placed nearby. In this environment, the reliability of the UniSCool system was verified, and its performance was directly compared to the traditional air-cooling system.

Figure 5. Image of the direct-to-chip UniSCool liquid cooling system installed in a Dell PowerEdge R740 server.The tests aimed to confirm that the system met the requirements described in Table 1.Table I.List of requirements to be met by the direct-to-chip liquid cooling system.

To evaluate the defined cooling requirements, the circuit operated continuously for 4 days to assess its long-term performance and reliability. Subsequently, a series of tests were conducted to evaluate the thermal dissipation and energy consumption behavior of the CPU under various load scenarios (25%, 50%, 75%, and 100%), each applied for 10 minutes.The test procedure began with the existing air-cooling system. This phase focused on measuring CPU temperature, fan power consumption, and overall server power consumption, all recorded by Dell's iDRAC software. As the primary goal of this phase was to assess energy consumption rather than thermal performance, the heat dissipation capability of the air-cooling system was not evaluated during this test.Subsequently, the traditional air-cooling system was replaced with the UniSCool liquid cooling system. The same load scenarios were applied, and the cooling system was configured to maintain the maximum CPU junction temperature at 80°C. During these tests, the pump's power consumption was also measured to evaluate the overall efficiency of the installed liquid cooling system.

4. Results

The system operated continuously for 4 days without any issues, demonstrating its reliability and robustness. This uninterrupted operation validated the subsequent tests, indicating the system's capability to maintain consistent performance over extended periods with no reliability issues such as leaks or disconnections.During the testing phase, the UniSCool liquid cooling system achieved significant cooling energy savings, ranging from 50% to 69% depending on the applied load (Figure 6). Furthermore, when using the liquid cooling system, we observed CPU overcooling, as its maximum temperature consistently remained around 50°C. This level of cooling indicates a significant thermal performance margin during operation, suggesting that pumping power could be further reduced without compromising cooling effectiveness.Additionally, when evaluating the server's overall energy consumption, the developed UniSCool liquid cooling system achieved savings of 13% to 30% (Figure 7). This calculation is based solely on the server's power consumption and excludes the energy used by the data center's air conditioning system or the thermostatic bath required to maintain the inlet liquid temperature.

The efficiency of the UniSCool liquid cooling system could be further enhanced if combined with a free cooling system. Free cooling utilizes ambient environmental conditions to reduce the need for mechanical cooling, thereby saving even more energy on data center air conditioning. Integrating the UniSCool direct-to-chip cooling solution with such a system could create a more energy-efficient cooling solution, significantly reducing operating costs and overall energy consumption.

Figure 6. Cooling energy consumption of the test server with air cooling and direct-to-chip liquid cooling technologies, along with the energy savings achieved using UniSCool liquid cooling compared to the air-cooling solution.

Figure 7.Overall server power consumption with air cooling and direct-to-chip liquid cooling technologies, showing the energy reduction achieved due to liquid cooling compared to the air-cooling solution.

5. Conclusion

This study successfully demonstrated the installation and validation of a direct-to-chip liquid cooling system within a data center under real operating conditions. The system underwent reliability and safety testing with satisfactory results, confirming that the developed liquid cooling system could operate continuously for up to four days without issues.Furthermore, the liquid cooling system achieved significant energy savings compared to the traditional air-cooling system. Specifically, depending on the applied thermal load, this cooling system achieved energy savings of 50% to 69%. The cooling energy savings were most significant at the highest CPU loads (75% and 100%).

Additionally, it was observed that the current liquid cooling system operates with excessive cooling capacity, resulting in CPU temperatures below normal levels. This overcooling phenomenon indicates potential for further optimization of the cooling system to bring CPU temperatures closer to the manufacturer's recommended optimum (around 80°C).The impact of the liquid cooling system on the server's overall power consumption was significant. The power required for cooling fans was halved, and the liquid cooling system saved between 13% and 30% of the server's total power consumption. This reduction highlights the potential to further decrease the server's total power consumption by eliminating or downsizing the fan system.While this study quantified the energy savings achieved by the liquid cooling system, there remains potential for further improvement. Specifically, integrating the liquid cooling system into the existing cooling infrastructure could lower energy consumption. The energy required to maintain a stable 25°C water temperature is significantly lower than that required for air cooling, implying that overall energy consumption could be further reduced.

In conclusion, this proof-of-concept study confirms that direct-to-chip liquid cooling systems offer significant energy savings when implemented in data centers. The results suggest that it may only be a matter of time before liquid cooling becomes the standard technology in server environments, replacing traditional air-cooling systems.

Kres(Suzhou) Power Systems Technology Co., Ltd is an emerging company focused on fluid power solutions, and serve for the markets of Energy, Construction, Agriculture, Manufacturing, Transportation, Medical, Data Centers, etc., and was founded in 2022 with leading-edge engineering capabilities and innovative products to keep up with the changing customer needs. Including OCP standard UQD, UQDB, BMQC, manifold, EPDM hose and hose assemblies, FF9 full flow dual interlock quick disconnect coupling (a better replacement for FD83 series products), API heat transfer products, Gates EPDM hose and Wika products.

---