Google Cloud has introduced the general availability of Lightning Engine for its Managed Service for Apache Spark. This new engine is designed to tackle performance challenges as data volumes increase, particularly in environments where autonomous agents run numerous concurrent queries.
Lightning Engine is fully compatible with existing Spark workloads, requiring no changes to current data pipelines. It can be deployed in either a serverless mode for ease of use or a managed cluster mode for more control over infrastructure.
Key performance enhancements include:
- Up to 4.9x faster performance compared to standard open-source Spark.
- Twice the price-performance of leading high-speed Spark alternatives.
How Lightning Engine Works
Lightning Engine utilizes vectorized native execution to overcome traditional Spark execution bottlenecks. By compiling Spark physical query plans into optimized C++ instructions, it significantly reduces JVM overhead.
Core features of this execution layer include:
- Vectorized sorting, which processes data in native memory to reduce CPU cycle overhead.
- Accelerated window functions that enhance calculations across row sets.
- Smart fallback that seamlessly transitions unsupported queries back to the JVM.
Optimized Data Connectors
Lightning Engine ensures efficient data handling with optimized connectors for Cloud Storage and BigQuery:
- Direct path connections that enhance scan times for complex files.
- Reduced metadata calls that minimize overhead in managing large partitioned tables.
- Native BigQuery connector that directly consumes data in Arrow format, eliminating serialization overhead.
Advanced Query Optimization
The engine features a cost-based query optimizer that introduces several custom optimization rules:
- Single HashTable caching to minimize redundant CPU cycles during joins.
- Aggregation pushdown to reduce data transfer across the network.
- Auto shuffle partitioning to dynamically adjust the number of partitions based on runtime statistics.
Getting Started with Lightning Engine
Users can enable Lightning Engine through the Google Cloud console or the gcloud CLI. To submit a serverless batch job, users should specify the premium tier in their Spark properties. For creating a managed cluster with Lightning Engine, specific commands can be executed in the terminal or through the console interface.
