[encoding/googlecloudlogentry] Add support for GCP Cloud DNS Logs

.chloggen/add-cgp-ldns-logs.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: breaking
+
+# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
+component:  extension/googlecloudlogentry_encoding
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note:  Parse Cloud DNS Query logs into log record attributes instead of placing it in the body as is.
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [44561]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []

extension/encoding/googlecloudlogentryencodingextension/README.md

@@ -24,6 +24,7 @@ Currently, this extension [can parse the following logs](#supported-log-types) i
   - [Regional External Application Load Balancer](https://docs.cloud.google.com/load-balancing/docs/https/https-reg-logging-monitoring)
   - [Cloud Armor logs](https://docs.cloud.google.com/armor/docs/request-logging) (embedded within load balancer logs) (extension [mapping](#cloud-armor-logs))
 - [Proxy Network Load Balancer logs](https://docs.cloud.google.com/load-balancing/docs/tcp/tcp-ssl-proxy-logging-monitoring#log-records) (extension [mapping](#proxy-network-load-balancer-logs))
+- [Cloud DNS logs](https://docs.cloud.google.com/dns/docs/monitoring#dns-log-record-format) (extension [mapping](#cloud-dns-logs))
 
 For all others logs, the payload will be placed in the log record attribute. In this case, the following configuration options are supported:
 
@@ -129,7 +130,10 @@ Examples:
 
 - Audit Logs: `encoding.format: "gcp.auditlog"`
 - VPC Flow Logs: `encoding.format: "gcp.vpcflow"`
+- Application Load Balancer Logs: `encoding.format: "gcp.load-balacer"`
 - Proxy Network Load Balancer Logs: `encoding.format: "gcp.proxy-nlb"`
+- Cloud DNS Logs: `encoding.format: "gcp.dns"`
+
 
 ### How encoding.format is determined
 
@@ -142,6 +146,10 @@ For example, `projects/my-project/logs/cloudaudit.googleapis.com%2Fsystem_event`
 2. **Map log type to format**: The extension maps specific log types to their corresponding encoding formats (`encoding.format`):
    - Audit logs (activity, data access, system event, policy): `gcp.auditlog`
    - VPC flow logs (network management-sourced and compute-sourced VPC flow logs): `gcp.vpcflow`
+   - Application Load Balancer logs (Global External and Regional External): `gcp.load-balancer`
+   - Cloud Armor logs (embedded within load balancer logs): `gcp.armorlog`
+   - Proxy Network Load Balancer logs: `gcp.proxy-nlb`
+   - Cloud DNS logs: `gcp.dns`
 
 3. **Set the attribute**: For recognized log types, the `encoding.format` attribute is set as an attribute of the `scope` field in the OTEL output log, allowing for flexible filtering and routing.
 
@@ -155,8 +163,10 @@ The following format values are supported in the `googlecloudlogentryencodingext
 |------------------|------------------|-----------------|
 | Audit Logs | `auditlog` | Google Cloud audit logs (activity, data access, system event, policy) |
 | VPC Flow Logs | `vpcflow` | Virtual Private Cloud flow log records |
+| Application Load Balancer Logs | `load-balancer` | Global and Regional External Application Load Balancer logs |
 | Armor Logs | `armorlog` | Google Cloud armor logs (security policies applied) |
 | Proxy Network Load Balancer Logs | `proxy-nlb` | Proxy Network Load Balancer connection logs |
+| Cloud DNS Logs | `dns` | Cloud DNS query and response logs |
 
 ### Cloud Audit Logs
 
@@ -432,7 +442,38 @@ Application Load Balancer logs (both [Global External](https://docs.cloud.google
 | `serverBytesReceived` | `gcp.load_balancing.proxy_nlb.server.bytes_received` |
 | `serverBytesSent` | `gcp.load_balancing.proxy_nlb.server.bytes_sent` |
 
+### Cloud DNS logs
+
+[Cloud DNS logs](https://docs.cloud.google.com/dns/docs/monitoring#dns-log-record-format) are mapped into OpenTelemetry attributes as follows:
+
+| Original field | Log record attribute |
+|---|---|
+| `queryName` | `dns.question.name` |
+| `queryType` | `dns.question.type` |
+| `responseCode` | `dns.response_code` |
+| `alias_query_response_code` | `gcp.dns.alias_query.response.code` |
+| `authAnswer` | `gcp.dns.auth_answer` |
+| `rdata` | `dns.answer.data` |
+| `destinationIP` | `server.address` |
+| `sourceNetwork` | `gcp.dns.client.vpc.name` |
+| `source_type` | `gcp.dns.client.type` |
+| `sourceIP` | `client.address` |
+| `protocol` | `network.transport` |
+| `location` | `cloud.region` |
+| `target_name` | `gcp.dns.server.name` |
+| `target_type` | `gcp.dns.server.type` |
+| `serverLatency` | `gcp.dns.server.latency` |
+| `egressError` | `gcp.dns.egress.error` |
+| `healthyIps` | `gcp.dns.healthy_ips` |
+| `unhealthyIps` | `gcp.dns.unhealthy.ips` |
+| `dns64Translated` | `gcp.dns.dns64.translated` |
+| `vmInstanceId` | `host.id` |
+| `vmInstanceName` | `host.name` |
+| `vmProjectId` | `gcp.project.id` |
+| `vmZoneName` | `cloud.availability_zone` |
+
 **Protocol translation**: The numeric protocol field from GCP is automatically translated to human-readable protocol names using the [IANA Protocol Numbers](https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml) standard. Common values include:
+
 - `6` → `tcp`
 - `17` → `udp`
 - `1` → `icmp`

extension/encoding/googlecloudlogentryencodingextension/extension_test.go

@@ -221,6 +221,16 @@ func TestPayloads(t *testing.T) {
 			logFilename:      "testdata/proxynlb/proxynlb-basic.json",
 			expectedFilename: "testdata/proxynlb/proxynlb-basic_expected.yaml",
 		},
+		{
+			name:             "dns query log - no error",
+			logFilename:      "testdata/dnslog/dns_query_no_error.json",
+			expectedFilename: "testdata/dnslog/dns_query_no_error_expected.yaml",
+		},
+		{
+			name:             "dns query log - nxdomain error",
+			logFilename:      "testdata/dnslog/dns_query_no_domain_error.json",
+			expectedFilename: "testdata/dnslog/dns_query_no_domain_error_expected.yaml",
+		},
 	}
 
 	extension := newTestExtension(t, Config{})

extension/encoding/googlecloudlogentryencodingextension/internal/constants/format.go

@@ -8,6 +8,7 @@ const (
 	FormatVPCFlowLog      = "vpcflow"
 	FormatLoadBalancerLog = "load-balancer"
 	FormatProxyNLBLog     = "proxy-nlb"
+	FormatDNSQueryLog     = "dns-query"
 
 	FormatIdentificationTag = "encoding.format"
 
@@ -17,4 +18,5 @@ const (
 	GCPFormatVPCFlowLog      = GCPFormatPrefix + FormatVPCFlowLog
 	GCPFormatLoadBalancerLog = GCPFormatPrefix + FormatLoadBalancerLog
 	GCPFormatProxyNLBLog     = GCPFormatPrefix + FormatProxyNLBLog
+	GCPFormatDNSQueryLog     = GCPFormatPrefix + FormatDNSQueryLog
 )

extension/encoding/googlecloudlogentryencodingextension/internal/dnslog/parser.go

@@ -0,0 +1,158 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+// Find more information about Cloud dns logs at:
+// https://docs.cloud.google.com/dns/docs/monitoring#dns-log-record-format
+package dnslog // import "github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/googlecloudlogentryencodingextension/internal/dnslog"
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+
+	gojson "github.com/goccy/go-json"
+	"go.opentelemetry.io/collector/pdata/pcommon"
+	semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
+
+	"github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/googlecloudlogentryencodingextension/internal/shared"
+)
+
+const (
+	CloudDNSQueryLogSuffix = "dns.googleapis.com%2Fdns_queries"
+
+	// Query related attributes. Ref: https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.2
+	// gcpDNSQueryType holds the DNS query type
+	gcpDNSQueryType = "dns.question.type" // TBD in SemConv
+
+	// Response related attributes
+	// gcpDNSResponseCode holds the DNS response code
+	gcpDNSResponseCode = "dns.response_code" // TBD in SemConv
+	// gcpDNSAliasQueryResponseCode holds the response code for alias queries
+	gcpDNSAliasQueryResponseCode = "gcp.dns.alias_query.response.code"
+	// gcpDNSAuthAnswer indicates whether the response is authoritative
+	gcpDNSAuthAnswer = "gcp.dns.auth_answer" // Ref: https://datatracker.ietf.org/doc/html/rfc1035
+	// gcpDNSAnswerData holds DNS answer in presentation format
+	gcpDNSAnswerData = "dns.answer.data" // TBD in SemConv
+
+	// Network related attributes
+	// gcpDNSClientVPCNetwork holds the client network name where the DNS query originated
+	gcpDNSClientVPCNetwork = "gcp.dns.client.vpc.name"
+	// gcpDNSClientType holds the client type of the DNS query
+	gcpDNSClientType = "gcp.dns.client.type"
+
+	// Server related attributes
+	// gcpDNSServerName holds the name name of Google Cloud instance that is responsible of resolving the query
+	gcpDNSServerName = "gcp.dns.server.name"
+	// gcpDNServerType holds the type of target resolving the DNS query
+	gcpDNServerType = "gcp.dns.server.type"
+
+	// Performance and error related attributes
+	// gcpDNSServerLatency holds the server-side latency in seconds
+	gcpDNSServerLatency = "gcp.dns.server.latency"
+	// gcpDNSEgressError holds Egress proxy error, the actual error as received from the on-premises DNS server
+	gcpDNSEgressError = "gcp.dns.egress.error"
+	// gcpDNSHealthyIPs holds addresses in the ResourceRecordSet that are known to be HEALTHY.
+	gcpDNSHealthyIPs = "gcp.dns.healthy_ips"
+	// gcpDNSUnhealthyIPs holds addresses in the ResourceRecordSet that are known to be UNHEALTHY.
+	gcpDNSUnhealthyIPs = "gcp.dns.unhealthy.ips"
+
+	// DNS feature related attributes
+	// gcpDNSDNS64Translated indicates whether DNS64 translation was applied
+	gcpDNSDNS64Translated = "gcp.dns.dns64.translated"
+
+	// VM instance related attributes
+	// gcpProjectID holds the Google Cloud project ID of the network from which the query was sent
+	gcpProjectID = "gcp.project.id"
+)
+
+type dnslog struct {
+	AliasQueryResponseCode string   `json:"alias_query_response_code"`
+	AuthAnswer             *bool    `json:"authAnswer"`
+	DestinationIP          string   `json:"destinationIP"`
+	DNS64Translated        *bool    `json:"dns64Translated"`
+	EgressError            string   `json:"egressError"`
+	HealthyIps             string   `json:"healthyIps"`
+	Location               string   `json:"location"`
+	Protocol               string   `json:"protocol"`
+	ProjectID              string   `json:"project_id"`
+	QueryName              string   `json:"queryName"`
+	QueryType              string   `json:"queryType"`
+	Rdata                  string   `json:"rdata"`
+	ResponseCode           string   `json:"responseCode"`
+	ServerLatency          *float64 `json:"serverLatency"`
+	SourceIP               string   `json:"sourceIP"`
+	SourceNetwork          string   `json:"sourceNetwork"`
+	SourceType             string   `json:"source_type"`
+	TargetName             string   `json:"target_name"`
+	TargetType             string   `json:"target_type"`
+	UnhealthyIps           string   `json:"unhealthyIps"`
+	VMInstanceID           *int64   `json:"vmInstanceId"`
+	VMInstanceName         string   `json:"vmInstanceName"`
+	VMProjectID            string   `json:"vmProjectId"`
+	VMZoneName             string   `json:"vmZoneName"`
+}
+
+func handleQueryAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutStr(string(semconv.DNSQuestionNameKey), log.QueryName, attr)
+	shared.PutStr(gcpDNSQueryType, log.QueryType, attr) // TBD in SemConv
+}
+
+func handleResponseAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutStr(gcpDNSResponseCode, log.ResponseCode, attr) // TBD in SemConv
+	shared.PutStr(gcpDNSAliasQueryResponseCode, log.AliasQueryResponseCode, attr)
+	shared.PutBool(gcpDNSAuthAnswer, log.AuthAnswer, attr)
+	shared.PutStr(gcpDNSAnswerData, log.Rdata, attr) // TBD in SemConv
+}
+
+func handleNetworkAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutStr(string(semconv.ServerAddressKey), log.DestinationIP, attr)
+	shared.PutStr(gcpDNSClientVPCNetwork, log.SourceNetwork, attr)
+	shared.PutStr(gcpDNSClientType, log.SourceType, attr)
+	shared.PutStr(string(semconv.ClientAddressKey), log.SourceIP, attr)
+	shared.PutStr(string(semconv.NetworkTransportKey), strings.ToLower(log.Protocol), attr)
+	shared.PutStr(string(semconv.CloudRegionKey), log.Location, attr)
+}
+
+func handleTargetAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutStr(gcpDNSServerName, log.TargetName, attr)
+	shared.PutStr(gcpDNServerType, log.TargetType, attr)
+}
+
+func handlePerformanceAndErrorAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutDouble(gcpDNSServerLatency, log.ServerLatency, attr)
+	shared.PutStr(gcpDNSEgressError, log.EgressError, attr)
+	shared.PutStr(gcpDNSHealthyIPs, log.HealthyIps, attr)
+	shared.PutStr(gcpDNSUnhealthyIPs, log.UnhealthyIps, attr)
+}
+
+func handleDNSFeatureAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutBool(gcpDNSDNS64Translated, log.DNS64Translated, attr)
+}
+
+func handleVMInstanceAttributes(log *dnslog, attr pcommon.Map) {
+	shared.PutInt(string(semconv.HostIDKey), log.VMInstanceID, attr)
+	shared.PutStr(string(semconv.HostNameKey), log.VMInstanceName, attr)
+	shared.PutStr(gcpProjectID, log.VMProjectID, attr)
+	shared.PutStr(string(semconv.CloudAvailabilityZoneKey), log.VMZoneName, attr)
+}
+
+func ParsePayloadIntoAttributes(payload []byte, attr pcommon.Map) error {
+	var log *dnslog
+	if err := gojson.Unmarshal(payload, &log); err != nil {
+		return fmt.Errorf("failed to unmarshal DNS log: %w", err)
+	}
+
+	if log == nil {
+		return errors.New("DNS cannot be nil after detecting payload as DNS log")
+	}
+
+	handleQueryAttributes(log, attr)
+	handleResponseAttributes(log, attr)
+	handleNetworkAttributes(log, attr)
+	handleTargetAttributes(log, attr)
+	handlePerformanceAndErrorAttributes(log, attr)
+	handleDNSFeatureAttributes(log, attr)
+	handleVMInstanceAttributes(log, attr)
+
+	return nil
+}