East China University of Science and Technology (ECUST)
Computer Science
PROJECT EXPERIENCE
OmniAdvisor - Automated HyperParameter Tuning
Orchestrated the entire design and development of OmniAdvisor, an intelligent system for end-to-end parameter tuning of Spark and Hive tasks within online systems, leveraging AI algorithms
Accomplished data preprocessing, data cleansing and other data workflows
Constructed ML pipelines, utilizing CNN, Random Forest algorithms
OmniMV - Intelligent Recommendation System for Materialized Views
Designed and implemented ETL pipelines, targeting at Spark/ Clickhouse/PostgreSQL
Generated materialized views utilizing ML algorithms (e.g. xgboost) to enhance SQL execution efficiency
Rewritten SQL physical plan to improve query execution efficiency
Data Analysis and Performance Optimization
Hardware statistics analysis, such as CPU utilization, disk I/O usage, memory usage and network bottlenecks
Performance bottleneck analysis for Big data components, such as Spark, Hive, Hadoop
Hotspot function identification and optimization
Fine-Tuning for Software, e.g. Spark, Hive, Clickhouse, Presto and etc.