README
¶
otlp-wire
OTLP wire format utilities for Go. Count, shard, and route telemetry data without unmarshaling.
What It Does
- Count signals (metrics/logs/traces) without unmarshaling
- Iterate over resources with minimal allocations for parallel processing
- Extract resource metadata for routing decisions
- Access individual span fields (TraceID, SpanID, ParentSpanID) with zero allocations
Performance Characteristics
Full protobuf unmarshaling is expensive:
- Allocates thousands of Go objects
- High garbage collector pressure
- High CPU overhead
otlp-wire operates on wire format bytes:
- 35-55x faster counting than unmarshaling (zero allocations)
- 1,100-2,800x faster iteration than unmarshal+iterate (2 allocations)
- 2,800-3,700x faster splitting than unmarshal+remarshal (2 allocations)
- Minimal GC pressure (only 24 bytes per batch for error handling)
- Zero dependencies (only stdlib + protowire)
See BENCHMARKS.md for detailed comparison.
Use Cases
- Observability: Count signals for monitoring ingestion volume
- Sharding: Split batches by resource for parallel processing
- Routing: Extract resource attributes for routing decisions
- Span Processing: Extract trace/span IDs without full unmarshal
Installation
go get go.olly.garden/otlp-wire
Quick Start
import "go.olly.garden/otlp-wire"
// Count signals for observability
data := otlpwire.ExportMetricsServiceRequest(otlpBytes)
count, err := data.DataPointCount()
if err != nil {
return err
}
metrics.RecordDataPointsReceived(count)
// Iterate over resources for sharding
resources, getErr := data.ResourceMetrics()
for resource := range resources {
resourceBytes, _ := resource.Resource()
hash := fnv64a(resourceBytes)
workerID := hash % numWorkers
var buf bytes.Buffer
resource.WriteTo(&buf)
sendToWorker(workerID, buf.Bytes())
}
if err := getErr(); err != nil {
return err
}
// Access individual span fields without full unmarshal
wire := otlpwire.ExportTracesServiceRequest(otlpBytes)
rsIter, rsErr := wire.ResourceSpans()
for rs := range rsIter {
ssIter, ssErr := rs.ScopeSpans()
for ss := range ssIter {
spanIter, spanErr := ss.Spans()
for s := range spanIter {
traceID, _ := s.TraceID() // [16]byte, zero allocs
spanID, _ := s.SpanID() // [8]byte, zero allocs
parentID, _ := s.ParentSpanID() // [8]byte, zero allocs
// ... use IDs for bloom filters, trace assembly, etc.
}
if err := spanErr(); err != nil { return err }
}
if err := ssErr(); err != nil { return err }
}
if err := rsErr(); err != nil { return err }
See example_test.go for complete working examples.
API Overview
Type Hierarchy
ExportMetricsServiceRequest (OTLP message bytes)
└─ ResourceMetrics[] (one per resource)
ExportLogsServiceRequest (OTLP message bytes)
└─ ResourceLogs[] (one per resource)
ExportTracesServiceRequest (OTLP message bytes)
└─ ResourceSpans[] (one per resource)
└─ ScopeSpans[] (one per instrumentation scope)
└─ Span[] (individual spans)
├─ TraceID()
├─ SpanID()
└─ ParentSpanID()
Methods
Batch-level operations:
type ExportMetricsServiceRequest []byte
func (m ExportMetricsServiceRequest) DataPointCount() (int, error)
func (m ExportMetricsServiceRequest) ResourceMetrics() (iter.Seq[ResourceMetrics], func() error)
type ExportLogsServiceRequest []byte
func (l ExportLogsServiceRequest) LogRecordCount() (int, error)
func (l ExportLogsServiceRequest) ResourceLogs() (iter.Seq[ResourceLogs], func() error)
type ExportTracesServiceRequest []byte
func (t ExportTracesServiceRequest) SpanCount() (int, error)
func (t ExportTracesServiceRequest) ResourceSpans() (iter.Seq[ResourceSpans], func() error)
Resource-level operations:
type ResourceMetrics []byte
func (r ResourceMetrics) DataPointCount() (int, error)
func (r ResourceMetrics) Resource() ([]byte, error)
func (r ResourceMetrics) WriteTo(w io.Writer) (int64, error)
type ResourceLogs []byte
func (r ResourceLogs) LogRecordCount() (int, error)
func (r ResourceLogs) Resource() ([]byte, error)
func (r ResourceLogs) WriteTo(w io.Writer) (int64, error)
type ResourceSpans []byte
func (r ResourceSpans) SpanCount() (int, error)
func (r ResourceSpans) Resource() ([]byte, error)
func (r ResourceSpans) WriteTo(w io.Writer) (int64, error)
func (r ResourceSpans) ScopeSpans() (iter.Seq[ScopeSpans], func() error)
Scope-level operations (traces):
type ScopeSpans []byte
func (s ScopeSpans) SpanCount() (int, error)
func (s ScopeSpans) Spans() (iter.Seq[Span], func() error)
Span-level field accessors:
type Span []byte
func (s Span) TraceID() ([16]byte, error)
func (s Span) SpanID() ([8]byte, error)
func (s Span) ParentSpanID() ([8]byte, error)
Design Philosophy
This library provides:
- Raw bytes at different granularity levels
- Methods to count, iterate, and extract
- Building blocks for custom use cases
This library does not:
- Force specific hash algorithms
- Make routing decisions
- Unmarshal unless absolutely necessary
Performance
Benchmarks on Apple M4 (5 resources, 100 signals per resource):
Counting Performance
| Operation | Wire Format | Unmarshal | Speedup |
|---|---|---|---|
| DataPointCount() | 2.3 μs, 0 allocs | 81.0 μs, 5,161 allocs | 35x |
| SpanCount() | 2.1 μs, 0 allocs | 115.3 μs, 5,131 allocs | 55x |
| LogRecordCount() | 2.2 μs, 0 allocs | 108.9 μs, 6,131 allocs | 49x |
Iteration Performance
| Operation | Wire Format | Unmarshal | Speedup |
|---|---|---|---|
| ResourceMetrics() | 56 ns, 2 allocs | 158 μs, 5,161 allocs | 2,800x |
| ResourceSpans() | 61 ns, 2 allocs | 100 μs, 5,131 allocs | 1,650x |
| ResourceLogs() | 93 ns, 2 allocs | 106 μs, 6,131 allocs | 1,140x |
Split Performance (Iterate + WriteTo)
| Operation | Wire Format | Unmarshal+Remarshal | Speedup |
|---|---|---|---|
| Metrics | 50 ns, 2 allocs | 143 μs, 7,742 allocs | 2,860x |
| Traces | 51 ns, 2 allocs | 192 μs, 7,192 allocs | 3,750x |
| Logs | 51 ns, 2 allocs | 178 μs, 8,692 allocs | 3,490x |
Note: The 2 allocations (24 bytes) in iteration are from the iterator error handling pattern (closure capture mechanism).
For detailed benchmarks and methodology, see BENCHMARKS.md.
Documentation
- DESIGN.md - Architecture, design decisions, and implementation details
- BENCHMARKS.md - Performance comparison and methodology
- example_test.go - Complete working examples (observability metrics, sharding, sampling)
Requirements
- Go 1.23+ (for
iter.Seqiterator support)
License
Related Projects
- OpenTelemetry Collector - Full-featured OTLP processing
- protowire - Low-level protobuf wire format utilities
Documentation
¶
Overview ¶
Package otlpwire provides utilities for working with OTLP wire format data.
Example (ObservabilityStats) ¶
Example_observabilityStats demonstrates using Count() for observability metrics.
// Simulate receiving OTLP metrics data
metrics := createSampleMetrics(100)
marshaler := &pmetric.ProtoMarshaler{}
otlpBytes, _ := marshaler.MarshalMetrics(metrics)
// Count signals for observability
data := otlpwire.ExportMetricsServiceRequest(otlpBytes)
count, _ := data.DataPointCount()
// Emit metrics about incoming data (cardinality monitoring, billing, etc.)
fmt.Printf("Received %d data points for processing\n", count)
Output: Received 100 data points for processing
Example (ShardingByService) ¶
Example_shardingByService demonstrates splitting batches for distributed processing.
// Create metrics from multiple services
metrics := createMultiServiceMetrics()
marshaler := &pmetric.ProtoMarshaler{}
otlpBytes, _ := marshaler.MarshalMetrics(metrics)
// Split batch by resource for sharding
data := otlpwire.ExportMetricsServiceRequest(otlpBytes)
numWorkers := 3
resources, getErr := data.ResourceMetrics()
i := 0
for resource := range resources {
// Hash resource for consistent routing
resourceBytes, _ := resource.Resource()
hash := hashBytes(resourceBytes)
workerID := int(hash % uint64(numWorkers))
var buf bytes.Buffer
_, _ = resource.WriteTo(&buf)
count, _ := otlpwire.ExportMetricsServiceRequest(buf.Bytes()).DataPointCount()
fmt.Printf("Resource %d → Worker %d (%d data points)\n", i, workerID, count)
i++
}
if err := getErr(); err != nil {
fmt.Printf("Error: %v\n", err)
}
Output: Resource 0 → Worker 0 (10 data points) Resource 1 → Worker 1 (10 data points) Resource 2 → Worker 2 (10 data points)
Example (TypeComposition) ¶
Example_typeComposition demonstrates how types compose naturally.
metrics := createSampleMetrics(25)
marshaler := &pmetric.ProtoMarshaler{}
otlpBytes, _ := marshaler.MarshalMetrics(metrics)
// Count at batch level
batch := otlpwire.ExportMetricsServiceRequest(otlpBytes)
count, _ := batch.DataPointCount()
fmt.Printf("Total data points: %d\n", count)
// Iterate and count at resource level (zero allocation)
resourceCount := 0
resources, getErr := batch.ResourceMetrics()
for resource := range resources {
if resourceCount == 0 {
// Count signals in this resource (zero allocation)
dpCount, _ := resource.DataPointCount()
fmt.Printf("Resource 0 data points: %d\n", dpCount)
}
resourceCount++
}
if err := getErr(); err != nil {
fmt.Printf("Error: %v\n", err)
}
fmt.Printf("Number of resources: %d\n", resourceCount)
Output: Total data points: 25 Resource 0 data points: 25 Number of resources: 1
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ExportLogsServiceRequest ¶
type ExportLogsServiceRequest []byte
ExportLogsServiceRequest represents an OTLP ExportLogsServiceRequest message.
func (ExportLogsServiceRequest) LogRecordCount ¶
func (l ExportLogsServiceRequest) LogRecordCount() (int, error)
LogRecordCount returns the total number of log records in the batch.
func (ExportLogsServiceRequest) ResourceLogs ¶
func (l ExportLogsServiceRequest) ResourceLogs() (iter.Seq[ResourceLogs], func() error)
ResourceLogs returns an iterator over ResourceLogs in the batch. The returned function should be called after iteration to check for errors.
type ExportMetricsServiceRequest ¶
type ExportMetricsServiceRequest []byte
ExportMetricsServiceRequest represents an OTLP ExportMetricsServiceRequest message.
func (ExportMetricsServiceRequest) DataPointCount ¶
func (m ExportMetricsServiceRequest) DataPointCount() (int, error)
DataPointCount returns the total number of metric data points in the batch.
func (ExportMetricsServiceRequest) ResourceMetrics ¶
func (m ExportMetricsServiceRequest) ResourceMetrics() (iter.Seq[ResourceMetrics], func() error)
ResourceMetrics returns an iterator over ResourceMetrics in the batch. The returned function should be called after iteration to check for errors.
type ExportTracesServiceRequest ¶
type ExportTracesServiceRequest []byte
ExportTracesServiceRequest represents an OTLP ExportTracesServiceRequest message.
func (ExportTracesServiceRequest) ResourceSpans ¶
func (t ExportTracesServiceRequest) ResourceSpans() (iter.Seq[ResourceSpans], func() error)
ResourceSpans returns an iterator over ResourceSpans in the batch. The returned function should be called after iteration to check for errors.
func (ExportTracesServiceRequest) SpanCount ¶
func (t ExportTracesServiceRequest) SpanCount() (int, error)
SpanCount returns the total number of spans in the batch.
type ResourceLogs ¶
type ResourceLogs []byte
ResourceLogs represents a single ResourceLogs message.
func (ResourceLogs) LogRecordCount ¶
func (r ResourceLogs) LogRecordCount() (int, error)
LogRecordCount returns the number of log records in this resource.
func (ResourceLogs) Resource ¶
func (r ResourceLogs) Resource() ([]byte, error)
Resource returns the raw Resource message bytes.
type ResourceMetrics ¶
type ResourceMetrics []byte
ResourceMetrics represents a single ResourceMetrics message.
func (ResourceMetrics) DataPointCount ¶
func (r ResourceMetrics) DataPointCount() (int, error)
DataPointCount returns the number of metric data points in this resource.
func (ResourceMetrics) Resource ¶
func (r ResourceMetrics) Resource() ([]byte, error)
Resource returns the raw Resource message bytes.
type ResourceSpans ¶
type ResourceSpans []byte
ResourceSpans represents a single ResourceSpans message.
func (ResourceSpans) Resource ¶
func (r ResourceSpans) Resource() ([]byte, error)
Resource returns the raw Resource message bytes.
func (ResourceSpans) ScopeSpans ¶ added in v0.0.2
func (r ResourceSpans) ScopeSpans() (iter.Seq[ScopeSpans], func() error)
ScopeSpans returns an iterator over ScopeSpans in this ResourceSpans. Field 2 in the ResourceSpans protobuf message. The returned function should be called after iteration to check for errors.
func (ResourceSpans) SpanCount ¶
func (r ResourceSpans) SpanCount() (int, error)
SpanCount returns the number of spans in this resource.
type ScopeSpans ¶ added in v0.0.2
type ScopeSpans []byte
ScopeSpans represents a single ScopeSpans message (raw wire bytes).
func (ScopeSpans) SpanCount ¶ added in v0.0.2
func (s ScopeSpans) SpanCount() (int, error)
SpanCount returns the number of spans in this ScopeSpans.
type Span ¶ added in v0.0.2
type Span []byte
Span represents a single Span message (raw wire bytes).
func (Span) ParentSpanID ¶ added in v0.0.2
ParentSpanID extracts the parent span ID from the Span. Returns the raw 8 bytes from field 4. Returns zero value if the field is not present (root span).
Source Files