Updated: Mar 9
The CAGR (compound annual growth rate) of cloud based high performance computing (HPC) has been predicted to touch 25% over the next three years. It's growth has lagged commercial cloud uptake, with the need for high bandwidth interconnects being a key consideration. Amazon Web Services (AWS) responded with their Elastic Fabric Adapter (EFA), and Microsoft have had Infiniband capability on Azure for some time. One of the key benefits for computer aided engineering (CAE) is the ability to rapidly assess model setup at the start of a virtual design phase to get a handle on the trade off's between accuracy and computation time and choose a balance that best suits the project.
The FDA has for some time been running a programme to help validate and apply CAE tools in the medical device sector. One of their early case studies used a simplified blood pump model to look at the performance of community computational fluid dynamics (CFD) solution submissions and to compare to multi-lab experimental data. The data has also been used subsequently for model validation, with specific focus on turbulence models.
Turbulence modelling in CFD, which is required when the inertial forces are more prominent than the viscous forces, comes in four flavours. The first and most mathematically simple is direct numerical simulation (DNS), which doesn't require any additional modelling to account for the effects of turbulence and resolves all of the scales of the turbulence directly (hence the name). However, the time and space resolution required to do this properly is overwhelming - expert predictions suggest that a DNS simulation of a commercial aircraft at cruise will be possible in 2075 if the current trends in computational power increases continue. Due to this computational burden the Reynolds' Averaged Navier-Stokes (RANS) approach was conceived in the 1970's to account for turbulent transport in a manner equivalent to diffusion. Many RANS models have been developed and deployed since, with the main challenge being the range available to the engineer and the number of empirical constants employed in the models. Large Eddy Simulation (LES) is a formulation that resolves the large scales of the turbulence, while modelling the smaller, or "sub-grid" scales. LES is more computationally costly than RANS, but less so than DES. More recent hybrid techniques use a combination of LES and RANS - so call Detached Eddy Simulation (DES) which utilises the RANS methodology close to solid boundaries, where the LES approach requires significant spatial resolution (and hence computational effort) to be accurate, and LES away from the solid boundaries where the accuracy over RANS can be employed with a more achievable increase in computational effort.
In the context of engineering use RANS has been the workhorse of CFD. Some applications in recent years have picked up the use of DES and LES, but for the more frequent product development activities, RANS is nearly always the starting point. However, with the continued growth of compute power in terms of core count on chips and the provision of a number of virtual machines networked over a high bandwidth interconnect, the more physically representative LES approach can now be employed on fast paced product development programs. As an example of how this could be done, we used the HPC offering from Microsoft's Azure platform, along with the setup and management toolset HPCBOX by Drizti, to show both how quickly a comparison of LES and RANS approaches can be performed, and the increase in accuracy that LES can provide.
The open source CFD toolset OpenFoam was used for the study, which was installed on the HPC cluster by Drizti, which comprised of four HB60rs nodes, each with 60 AMD EPYC 7551 processor cores, 4 GB of RAM per CPU core and no hyper-threading. All models used the same mesh, generated with snappyHexMesh, with over 3.1 million cells and ten layers next to the wall boundaries (achieving a y+ of less than one in most regions). The RANS models took less than an hour to run, whereas the LES model took eighty hours, continuously over three days. One of the key areas for comparing CFD results to the multi-laboratory data generated by the FDA is in a diffuser region immediately downstream of the exit from the pump chamber, with PIV velocity data being available for the mid-plane section as illustrated below, followed by an image of the mesh resolution in that region.
The in-plane normalized velocity in this plane is shown below for the experimental data, three different types of RANS model and a "WALE" LES model which provides increased accuracy in the near wall region of the flow without requiring excessive mesh resolution in that area. It's clear that both the linear and SST RANS models have a high velocity jet extending out to the end of the diffuser which is not evident in the experimental data. The cubic RANS model improves this situation slightly, but still shows higher velocities than the experiment. The LES model shows good agreement with the experimental data, capturing the structure of the velocity field much better than the RANS models.
Access to cloud based HPC allows CAE model setup to be assessed in a timely fashion ahead of a product development cycle - the models and analysis shown here were all performed within one working week (five days). What hasn't been shown, and will be specific to each project, is the range of outputs desired from the CAE model and an assessment of the accuracy of the various model setups with regards to those key metrics. The best CAE model can then be chosen on balance of computational efficiency and accuracy across the different outputs. From a CFD perspective, the accuracy of the higher fidelity LES turbulence modelling approach is evident, and available at reasonable computational timescales.
We're intending to publish the work behind the data shown here, but if you are interested in more details, or would like to talk about how these techniques could add value to your product development program, please get in touch.