Data Engineering Subteam
Who we are:
The CDS Data Engineering subteam exists to provide analysis and processing support to CDS project teams, and to develop institutional knowledge in high throughput computing.
Improve on existing high throughput computing frameworks
Develop solutions for data analysis problems in CDS projects
Provide a reservoir of reference information in data engineering
Research and publish means of improving existing DE frameworks
Spark ML Optimization: Apache Spark's machine learning modules are not as well-studied as those of other platforms. This project seeks to empirically identify optimal settings for Spark's ML modules to best utilize the platform's unique capabilities.
SkinnerDB Parallelization: This project's objective is to experiment with parallelism in Professor Trummer's recently developed database engine, SkinnerDB. The SkinnerDB uses a machine learning approach to query optimization, in contrast to the heuristic model used by most current database engines, but has not yet been expanded to allow multi-core execution.
Deterministic Query Approximation: Several recent publications have outlined methods to allow high-speed query approximation with deteministic bounds, but have not yet been applied to a wide range of queries. The objective of this project is to apply several of these techniques to the TPC-H query benchmarks to demonstrate broader applicability.
GPU Acceleration: The distributed GPU computing deals with the unique task of handling distributed deep learning tasks, which is currently well-optimized for multiple GPUs, but not necessarily across multiple machines. Our goal is to research and optimize current tools in development so that it can be adopted by CDS teams deploying large DL models.
Server monitoring: Real time visualization and monitoring of compute server resource utilization through Cockpit
Spark diagnostics: Deliberate attempts to produce errors while running Apache Spark, both locally and on our servers. Problem specifics and solutions were recorded in case similar issues develop in the future.