Note the job name “pubsub2splunk” matches the name used in the log filter above, which excludes logs from this specific Dataflow job. Below is an example form, populated in the Console after selecting “Cloud Pub/Sub to Splunk” template. The Pub/Sub to Splunk pipeline can be executed from the UI, gcloud, or via a REST API call ( more detail here). Note: For high availability and to scale with high-volume traffic, refer to Splunk docs on how to scale by distributing load to multiple HEC nodes fronted by an HTTP(S) load balancer.Įxecuting Pub/Sub to Splunk Dataflow template Take note of the newly created HEC token, which will be used below. If you don’t already have an Splunk HEC endpoint, refer to the Splunk docs on how to configure Splunk HEC, whether it’s on your managed Splunk Cloud service or your own Splunk Enterprise instance. In this case, it’s Pub/Sub topic my-logs. If you created the log sink using the Cloud Console, it will automatically grant the new service account permission to write to its export destinations, provided you own the destination. This is how the end-to-end logging export looks: To facilitate this setup, Google released the Pub/Sub to Splunk Dataflow template with built-in capabilities like retry with exponential backoff (for resiliency to network failures or in case Splunk is down) and batch events and/or parallelize requests (for higher throughput) as detailed below. This is where the managed Cloud Dataflow service comes into play: A Dataflow job can automatically pull logs from a Pub/Sub topic, parse and convert payloads into the Splunk HEC event format, apply an optional user-defined function (UDF) to transform or redact the logs, then finally forward to Splunk HEC. We’ve heard from you that you need a more cloud-native approach that streams logs directly to a Splunk HTTP(S) endpoint, or Splunk HEC, without the need to manage an intermediary fleet of data collectors. Requires external resource access to Google Cloud by giving permissions to aforementioned data collectors to establish subscription and pull data from Pub/Sub topic(s) Requires managing one or more data collectors (a.k.a., Splunk-heavy forwarders) with added operational complexity for high availability and scale-out with increased log volume This documented solution works well, but it does include tradeoffs that need to be taken into account: Specifically, this add-on runs a Splunk modular input that periodically pulls logs from a Pub/Sub topic that’s configured as a log sink export. Traditionally, Splunk users have the option to pull logs from Google Cloud using Splunk Add-on for Google Cloud Platform as a data collector. The step-by-step walkthrough covers the entire setup, from configuring the originating log sinks in Cloud Logging to the final Splunk destination-the Splunk HTTP Event Collector (HEC) endpoint. “Thanks to their efforts, we can leverage both Google's Pub/Sub model and Splunk's query capabilities to simplify the management of our detection and response infrastructure and process over eight million daily events.” “Google Cloud’s Pub/Sub to Splunk Dataflow template has been helpful for enabling Spotify Security to ingest highly variable log types into Splunk,” says Andy Gu, Security Engineer at Spotify. We hear that customers are using this template to meet the variety, velocity and volume of valuable data coming out of Google Cloud. That includes logs from Cloud Logging (formerly Stackdriver Logging), messages from IoT devices, or events such as security findings from Cloud Security Command Center. So, any message that can be delivered to a Pub/Sub topic can now be forwarded to Splunk. Using this Dataflow template, you can export data from Pub/Sub to Splunk. In this blog post, we’ll show you how to set up a streaming pipeline to natively push your Google Cloud data to your Splunk Cloud or Splunk Enterprise instance using the recently released Pub/Sub to Splunk Dataflow template. These use cases include IT ops, security, application development, and business analytics. Many of you use third-party monitoring solutions from Splunk to keep a tab on both on-prem and cloud environments. To learn about latest feature enhancements s ince this announcement blog below, see What’s new with Splunk Dataflow template.Īt Google Cloud, we’re focused on solving customer problems and meeting them where they are. Editor's Note: To get started, follow our tutorial on deploying production-ready log exports to Splunk using Dataflow, and take advantage of associated Terraform scripts to automate deployment.
0 Comments
Leave a Reply. |