ARCCA supporting AI and Data-driven research
26 October 2020
To drive the development of services that meet the evolving and diverse needs of the Cardiff University research community, ARCCA have been working with StackHPC, a company who specialise in developing OpenStack capabilities for research computing use cases, to provide an important upgrade to the high-performance data analytics (HPDA) system, “Sparrow”.
Sparrow is an integrated OpenStack solution to store, manage and analyse complex data applicable to a range of disciplines. This upgrade deployment will deliver a new service with an emphasis on flexibility and ease of use.
A series of six workshops took place between ARCCA and StackHPC throughout July and August culminating in a detailed OpenStack reference architecture for the new Sparrow deployment. Based on previously gathered use cases for the platform, researchers representing a range of research disciplines were invited to take part in a dedicated session with StackHPC and ARCCA to discuss their requirements for the new service.
The Sparrow system will provide HPDA services supporting artificial intelligence (AI) and data-driven research opportunities, to complement the batch-queued HPC service provided by Hawk. This new architecture will deliver service enhancements to support virtualised workloads including support for automated containerised resource deployment. In addition, a series of hardware upgrades will be delivered, most notably support for GPU-accelerated workflows as well as support for handling of controlled-access data to meet enhanced security requirements.
Target platforms to be hosted by Sparrow include Apache Spark, a distributed processing system used for big data workloads, implemented using the Kubernetes orchestration system for delivering containerised applications. Storage will be provided from the Ceph flexible and scalable storage platform, as well as a variety of applications. Delivered as a workload in Kubernetes, the latter will include:
NextFlow - A reactive workflow framework that eases the writing of data-intensive computational pipelines;
Hail - A genomic data analysis application that provides powerful, easy-to-use data science tools;
Jupyter - A web-based interactive development environment that supports a wide range of workflows in data science, scientific computing, and machine learning.
With the deployment of the upgrade well underway, the launch of a pilot service is expected by the end of the year following successful completion of the associated acceptance tests. Our intention is to deliver a full production service in early 2021.