Cloudflare Logpush Transformers Go Live, Enabling Pre-Export Data Manipulation
Cloudflare's new Logpush Transformers feature allows users to filter, reshape, and redact log data in real-time before it's exported, reducing costs and simplifying downstream analysis.
Cloudflare has announced the general availability of its Logpush Transformers feature, a significant enhancement to its observability suite. This new capability empowers users to manipulate log data directly within the Logpush export pipeline, enabling them to filter, reshape, and redact logs before they are sent to external systems like Security Information and Event Management (SIEM) platforms, analytics tools, or object storage.
Traditionally, organizations have had to deploy and manage separate processing services to clean and format log data after export. This often involves significant overhead in terms of deployment, scaling, monitoring, and maintenance. Furthermore, customers incur costs for exporting, ingesting, and storing log data that is ultimately discarded, such as routine health checks or unnecessary fields. Logpush Transformers addresses these challenges by integrating data transformation directly into the export workflow.
Users can define custom SQL queries to specify which log records and fields should be processed. These queries can filter out unwanted traffic, redact sensitive information like personally identifiable information (PII), and reshape log structures by adding, renaming, removing, or nesting fields. The transformed data is then exported in NDJSON format, ensuring that destinations receive data precisely as needed, thereby reducing downstream data volume and associated costs.
The feature leverages Apache DataFusion, a Rust-based query engine built on Apache Arrow, for efficient, in-process execution. This approach prioritizes reliability and high throughput by avoiding network hops that could introduce latency or failures. Apache Arrow's columnar in-memory format is optimized for analytical workloads, allowing for vectorized execution and rapid processing of high-volume log streams, which can exceed millions of records per second.
Cloudflare has made it simple for users to manage these transformations. The Cloudflare dashboard offers a "Transformer Studio" for creating and previewing queries, while an API allows for programmatic management. Transformers can be attached to compatible Logpush jobs, and versioning is supported, enabling users to track changes over time. This integration ensures that the transformation process is predictable and does not introduce additional failure points into the critical export path.
This move aligns with Cloudflare's broader commitment to making advanced observability features accessible to all users, including recent updates that made Logpush available on all self-serve plans. By embedding a powerful SQL transformation engine directly into the high-throughput Logpush export pipeline, Cloudflare aims to provide a more efficient, cost-effective, and streamlined experience for managing and analyzing log data.
The implementation of Logpush Transformers is a strategic step towards democratizing advanced data processing capabilities. It allows security and operations teams to focus more on analysis and less on the complexities of data preparation, ultimately enhancing their ability to derive insights and maintain security posture from their Cloudflare logs.