Most experimental sciences now rely on computing, and biological sciences are no exception. As datasets get bigger, so do the computing costs, making proper optimization of the codes used by scientists increasingly important. Many of the codes developed in recent years are based on the Python-based NumPy, due to its ease of use and good performance characteristics. The composable nature of NumPy, however, does not generally play well with the multi-tier nature of modern CPUs, making any non-trivial multi-step algorithm limited by the external memory access speeds, which are hundreds of times slower than the CPU’s compute capabilities. In order to fully utilize the CPU compute capabilities, one must keep the working memory footprint small enough to fit in the CPU caches, which requires splitting the problem into smaller portions and fusing together as many steps as possible. In this paper, we present changes based on these principles to two important functions in the scikit-bio library, principal coordinates analysis and the Mantel test, that resulted in over 100x speed improvement in these widely used, general-purpose tools.
High Performance Computing (HPC) is nearing the exascale era and several challenges have to be addressed in terms of application development. Future parallel programming models should not only help developers take full advantage of the underlying machine but they should also account for highly dynamic runtime conditions, including frequent hardware failures. In this paper, we analyze the porting process of a plasma confinement simulator from a traditional MPI+OpenMP approach to a parallel objects based model like Charm++. The main driver for this effort is the existence of load imbalanced input scenarios that through pure OpenMP scheduling cannot be solved. By using Charm++ adaptive runtime and integrated balancing strategies, we were able to increase total CPU usage from 45.2% to 80.2%, achieving a 1.64× acceleration, after load balancing, over the MPI+OpenMP implementation on a specific input scenario. Checkpointing was added to the simulator thanks to the pack-unpack interface implemented by Charm++, providing scientists with fault tolerance and split execution capabilities.
As many scientific computation tasks focus on solving large-scale and computationally intensive problems, a wide range of problems involving High-Throughput Computing (HTC) paradigms and data-oriented algorithms emerge. Solving these HTC problems efficiently on modern supercomputers usually requires efficient and convenient job bundling. In this research, we evaluate multiple handy tools and workflows that are used to realize efficient and convenient job bundling. We also provide some practice guidelines for users when job bundling is required.
The development of the A64FX processor by Fujitsu has allowed the re-emergence of vectorized processors and the birth of the first supercomputer to achieve top speeds in all 5 of the major HPC benchmarks. We use a variety of tools to analyze the behavior and performance of several OpenMP applications with different compilers, and how these applications scale on the A64FX processor onA64FX clusters at Stony Brook University and RIKEN in Japan.
Empirical Dynamic Modeling (EDM) is a state-of-the-art non-linear time-series analysis framework. Despite its wide applicability, EDM was not scalable to large datasets due to its expensive computational cost. To overcome this obstacle, researchers have attempted and succeeded in accelerating EDM from both algorithmic and implementational aspects. In previous work, we developed a massively parallel implementation of EDM targeting HPC systems (mpEDM). However, mpEDM maintains different backends for different architectures. This design becomes a burden in the increasingly diversifying HPC systems, when porting to new hardware. In this paper, we design and develop a performance-portable implementation of EDM based on the Kokkos performance portability framework (kEDM), which runs on both CPUs and GPUs while based on a single codebase. Furthermore, we optimize individual kernels specifically for EDM computation, and use real-world datasets to demonstrate up to 5.5x speedup compared to mpEDM in convergent cross mapping computation.
Deep learning frameworks have been widely used in image classification and segmentation tasks. In this paper we outline the methods used to adapt an image segmentation model, U-Net, to identify buildings in geospatial images. The model has been trained and tested on a set of orthophotographic and LiDAR data from the state of Indiana. Its results are compared to the results achieved by a ResNet101 and RefineNet model trained with the same data, excluding the LiDAR data. This tool has a wide range of potential uses in research involving geospatial imagery. We discuss these use cases and some of the challenges and pitfalls in tuning a model for use with geospatial data.
While the computational power of supercomputers has risen tremendously in recent years, users' increasingly intensive I/O workload can easily overwhelm file systems on the supercomputers. Generating a huge amount of data and IOPS in a brief period of time may significantly slow down file systems and in some cases may result in a crash incurring the loss of users' compute time, a great burden on administrators and user services, and poor reliability perception. Nearly a decade of close observation and study of file systems have led us to formulate new guidelines and invent several tools to alleviate the I/O issues faced in the current supercomputing environment. In this manuscript, we focus on I/O work done on the Lustre parallel file systems of Frontera and Stampede2, but also investigate other types of file systems employed on other TACC supercomputers. We also discuss common I/O issues collected from supercomputer users, including high frequency of MDS requests, overloaded OSS, large unstriped files, etc. To solve these problems, we offer important guidelines on how to choose optimal file systems for the work being run. Furthermore, we introduce novel tools and workflows, such as CDTool, Python_Cacher, OOOPS, and stripe_scratch to facilitate users' I/O work. We believe these tools will greatly benefit users who need to manage heavy I/O workloads on parallel file systems.
Asynchronous many-task runtime systems and MPI+X hybrid parallelism approaches have shown promise for helping manage the increasing complexity of nodes in current and emerging high performance computing (HPC) systems, including those for exascale. The increasing architectural diversity, however, poses challenges for large legacy runtime systems emphasizing broad support for major HPC systems. Performance portability layers (PPL) have shown promise for helping manage this diversity. This paper describes a heterogeneous MPI+PPL task scheduling approach for combining these promising solutions with additional consideration for parallel third party libraries facing similar challenges to help prepare such a runtime for the diverse heterogeneous systems accompanying exascale computing. This approach is demonstrated using a heterogeneous MPI+Kokkos task scheduler and the accompanying portable abstractions~\cite{SCI:Hol2019b} implemented in the Uintah Computational Framework, an asynchronous many-task runtime system, with additional consideration for hypre, a parallel third party library. Results are shown for two challenging problems executing workloads representative of typical Uintah applications. These results show performance improvements up to 4.4x when using this scheduler and the accompanying portable abstractions~\cite{SCI:Hol2019b} to port a previously MPI-Only problem to Kokkos::OpenMP and Kokkos::CUDA to improve multi-socket, multi-device node use. Good strong-scaling to 1,024 NVIDIA V100 GPUs and 512 IBM POWER9 processor are also shown using MPI+Kokkos::OpenMP+Kokkos::CUDA at scale.
Scientific data, its analysis, accuracy, completeness and reproducibility play a vital role in advancing science and engineering. Open Science Chain (OSC) is a cyberinfrastructure platform built using the Hyperledger Fabric (HLF) blockchain technology to address issues related to data reproducibility and accountability in scientific research. OSC preserves integrity of research datasets and enables different research groups to share datasets with the integrity information. Additionally, it enables quick verification of the exact datasets that were used for a particular published research and tracks its provenance. In this paper, we describe OSC's command line utility that will preserve the integrity of research datasets from within the researchers' environment or from remote systems such as HPC resources or campus clusters used for research. The python-based command line utility can be seamlessly integrated within research workflows and provides an easy way to preserve the integrity of research data in OSC blockchain platform.
A User Portal is being developed for NSF-funded Expanse supercomputer. The Expanse portal is based on the NSF-funded Open OnDemand HPC portal platform which has gained widespread adoption at HPC centers. The portal will provide a gateway for launching interactive applications such as MATLAB, RStudio, and an integrated web-based environment for file management and job submission. This paper discusses the early experience in deploying the portal and the customizations that were made to accommodate the requirements of the Expanse user community.
The future for science gateways warrants exploration as we consider the possibilities that extend well beyond 'science' and high-performance computing into new interfaces, applications and user communities. In this paper, we look retrospectively at the successes of representative gateways thus far. This serves to highlight existing gaps gateways need to overcome in areas such as accessibility, usability and interoperability, and in the need for broader outreach by drawing insights from technology adoption research. We explore two particularly promising opportunities for gateways - computational social sciences and virtual reality – and make the case for the gateway community to be more intentional in engaging with users to encourage adoption and implementation, especially in the area of educational usage. We conclude with a call for focused attention on legal hurdles in order to realize the full future potential of science gateways. This paper serves as a roadmap for a vision of science gateways in the next ten years.
Nexrad data along with software to convert between binary formats, perform quality control, analyze, and visualize the data are all public and open for access and download. The missing pieces are the knowledge of how to use the available components with reproducible results. A science gateway is a web-based platform to bring all the components together for a novice to learn from experts, and for expert researchers to customize the tools for science.
Understanding the capabilities of new architectures is key to informing system purchases and good long-term ROI for cluster installations. The newest AMD architecture, Milan, has become available first on Microsoft Azure and we use this opportunity to measure the performance of this 3rd generation AMD EPYC processor. In this paper single node performance is gathered for seven popular scientific applications and benchmark test-suites. Quantitative comparisons are carried out between two independent platforms, Milan and its architectural predecessor Rome, for performance evaluations. Our results have shown that the changes in the Milan architecture have improved performance and met our projections.
Fundamental progress towards reliable modern science depends on accurate anomaly detection during application execution. In this paper, we suggest a novel approach to tackle this problem by applying Convolutional Neural Network (CNN) classification methods to high-resolution visualizations that capture the end-to-end workflow execution timeline. Subtle differences in the timeline reveal information about the performance of the application and infrastructure’s components. We collect 1000 traces of a scientific workflow’s executions. We explore and evaluate the performance of CNNs trained from scratch and pre-trained on ImageNet [7]. Our initial results are promising with over 90% accuracy.
Modern smartphones contain a collection of energy-efficient sensors capable of capturing the device's movement, orientation, and location as well characteristics of its external environment (e.g. ambient temperature, sound, pressure). When paired with peripheral wearable devices like smart watches, smartphones can also facilitate the collection/aggregation of important vital signs like heart rate and oxygen saturation. Evidence suggests that signatures of health and disease, or digital biomarkers, exist within the heterogeneous, temporally-dense data gathered from smartphone sensors and wearable devices that can be leveraged for medical applications. Here we discuss our recent experiences with deploying an open-source, cloud-native framework to monitor and collect smartphone sensor data from a cohort of pregnant women over a period of one year. We highlight two open-source integrations into the pipeline we found particularly useful: 1) a dashboard--built with Grafana and backed by Graphite--to monitor and manage production server loads and data collection metrics across the study cohort and 2) a back-end storage solution with InfluxDB, a multi-tenant time series database and data exploration ecosystem, to support biomarker discovery efforts of a multidisciplinary research team.
HPC is increasingly identified as a strategic asset and enabler to accelerate the research and the business performed in all areas requiring intensive computing and large-scale Big Data analytic capabilities. The efficient exploitation of heterogeneous computing resources featuring different processor architectures and generations, coupled with the eventual presence of GPU accelerators, remains a challenge. The University of Luxembourg operates since 2007 a large academic HPC facility which remains one of the reference implementation within the country and offers a cutting-edge research infrastructure to Luxembourg public research. The HPC support team invests a significant amount of time in providing a software environment optimised for hundreds of users, but the complexity of HPC software was quickly outpacing the capabilities of classical software management tools. Since 2014, our scientific software stack is generated and deployed in an automated and consistent way through the RESIF framework, a wrapper on top of EasyBuild and Lmod meant to efficiently handle user software generation. A large code refactoring was performed in 2017 to better handle different software sets and roles across multiple clusters, all piloted through a dedicated control repository. With the advent in 2020 of a new supercomputer featuring a different CPU architecture, and to mitigate the identified limitations of the existing framework, we report in this state-of-practice article RESIF 3.0, the latest iteration of our scientific software management suit now relying on streamline EasyBuild. It permitted to reduce by around 90% the number of custom configurations previously enforced by specific Slurm and MPI settings, while sustaining optimised builds coexisting for different dimensions of CPU and GPU architectures. The workflow for contributing back to the EasyBuild community was also automated and a current work in progress aims at drastically decrease the building time of a complete software set generation. Overall, most design choices for our wrapper have been motivated by several years of experience in addressing in a flexible and convenient way the heterogeneous needs inherent to an academic environment aiming for research excellence. As the code base is available publicly, and as we wish to transparently report also the pitfalls and difficulties met, this tool may thus help other HPC centres to consolidate their own software management stack.
Research workflows will benefit from a hybrid computing environment that offers seamless integration between on-campus and off-campus cloud resources. Commercial and Federal clouds offer researchers access to novel computing platforms that may not be available on their campus, offering opportunities to improve workflows and reduce the time to research. The large number of cloud offerings, however, makes cost management, and workflow transitions to appropriate platforms challenging. Successfully mapping workflows from on-campus resources to the cloud, and leveraging the available cost structures to find economical cost models are critical steps to enabling researcher access to this vast resource. To ameliorate these concerns, here we introduce the Research Computing Bazaar (RCB) software application for resource mapping and cost estimation. RCB is a software-as-a-service platform is an elastic, scalable, and fault tolerant system. It is developed using actual data from research computing workloads, and can be easily configured to be used by users or system administrators in Slurm-based on-premise computing environments. In this pilot, we inform researchers about opportunities to leverage flexible workload orchestration in managing cloud costs on a major cloud service provider. An extension into predictive capacities with machine learning mechanisms is being developed.
A central focus of evolutionary biology is inferring the historical relationships among species and using this context to learn about how evolution has shaped diverse organisms. These historical relationships are represented by phylogenetic trees, and the methods used to infer these trees have been an active area of research for several decades. Despite this attention, phylogenetic workflows have changed little, even though extraordinary advances have occurred in the scale and pace at which genomic data have been collected in the past 20 years. Modern phylogenomic datasets have also raised fascinating new questions. Why do different parts of a genome often support different relationships among species? How are these different signals distributed across chromosomes? We developed a new computational framework, CloudForest, to tackle such questions. CloudForest is flexible, efficient, and tightly integrates a diverse set of tools. Here, we briefly describe the architecture of CloudForest, including the advantages it provides, and use it to investigate the distribution of phylogenetic signal along the entire X chromosome of 24 cat species.
As the world moves away from traditional energy sources based on fossil fuels, several alternatives have been explored. One promising clean energy source is nuclear fusion. The fusion of hydrogen atoms may provide generous consumable energy gains. However, nuclear fusion reactors are not ready to become a productive mechanism yet. To accelerate the required breakthroughs in that community, numerical simulations and scientific visualizations over high-performance computing systems are mandatory. The results from the simulations and a proper display of the data are key to design and tune up nuclear fusion reactors. We explore the scientific visualization of plasma confinement in this paper, presenting two different dimensions. First, we revisit how visualizations help scientists understand the phenomena behind the fundamental processes of nuclear fusion. Second, we explore how visualization may also work as scientific communication tools for the general public to grasp the impact of this endeavor. We introduce a computer-graphics model that uses the output of numerical simulations to create visually plausible images of plasma confinement. The model is based on a combination of computer graphics techniques implemented on a ray-tracing framework.
Chemical data are diverse and complex, are obtained from experimental and computational modeling, and may encode large degrees of freedom of movement of particles such as whole assemblies, clusters, molecules, atoms, and even nuclei and electrons. To derive knowledge from this data requires analyses using a variety of techniques including approximation, dimensionality reduction, principal component analysis, and topological analysis. In this manuscript we describe the DELTA Science Gateway that integrate several types of mathematical and topological analysis software for chemical data analysis. The focus is on energy landscape data derived from experimental and computational modeling techniques towards understanding the principals involved in structure and function of molecular moieties, particularly in delineating the mechanism of catalytic activity. The gateway design, creation and production deployment will be discussed. The DELTA gateway is hosted under the SciGaP project at Indiana University powered by Apache Airavata gateway middleware framework. The gateway provides an integrated infrastructure for simulations and analysis on XSEDE and IU HPC resources and interactive visualization through locally deployed VNC client and a JupyterHub deployed on the XSEDE Jetstream cloud using virtual clusters. The gateway provides intuitively simple user interfaces for providing simulation input data, combines available model data, and enables users to set up and execute the simulation/analyses at the HPC systems.
