ParseDMARC-go
A Go implementation of the DMARC report parser, based on the original Python parsedmarc project.
📋 Conversion & Enhancements
The conversion to Go was done with Claude AI, adding significant improvements:
✅ Core enhancements:
- ClickHouse storage with pre-configured Grafana dashboard
- HTTP reporting method (RUA/RUF with https/http scheme URI)
- Prometheus daemon mode monitoring (IMAP + HTTP)
- Directory-based output mode
❌ Not converted (due to lack of testing capability):
- Elasticsearch/Opensearch/Splunk storage
- Microsoft Graph and Gmail API support
🌟 Core Features
📊 Report Parsing - Industry leading format support
- ✅ DMARC Aggregate Reports (RFC 7489)
- Draft and 1.0 standard formats
- Compressed file support (GZIP, ZIP)
- Enhanced error reporting with line numbers
- ✅ Forensic/Failure Reports (RFC 6591 ARF)
- Plain text format parsing
- 🆕 MIME multipart email parsing (LinkedIn, Domain.de, Netease)
- 🆕 Base64-encoded attachment support
- Automatic format detection and fallback
- ✅ SMTP TLS Reports (RFC 8460)
- Direct JSON format parsing
- 🆕 MIME email format parsing (Google, other providers)
- 🆕 Base64 + GZIP compressed attachment pipeline (
application/tlsrpt+gzip) - Legacy compressed file support (GZIP, ZIP)
🌐 Data Enhancement
- ✅ IP address geolocation (MaxMind database integration)
- ✅ Reverse DNS resolution with caching
- ✅ Base domain extraction and normalization
- ✅ Enhanced error diagnostics with precise line numbers
📡 Multiple Input Methods
- ✅ IMAP Email Processing - Monitor mailboxes for incoming reports
- TLS/SSL connection support
- Automatic email archiving/deletion
- Configurable check intervals
- ✅ HTTP API Server - Receive reports via HTTP POST/PUT (IETF draft)
- Rate limiting and request validation
- Multiple content-type support (
application/xml,application/json,message/rfc822) - File upload size limits and security
💾 Flexible Output & Storage
- ✅ JSON and CSV output formats with configurable fields
- ✅ Multiple output modes:
- File mode: Concatenate all reports in single file
- 🆕 Directory mode: Save each report as separate timestamped file
- Stdout: Direct console output for piping
- ✅ ClickHouse database storage with optimized schema
- ✅ Email delivery via SMTP with attachment support
- ✅ Kafka streaming for real-time processing pipelines
📈 Production Monitoring
- ✅ Built-in Prometheus metrics for observability
- ✅ Health check endpoints for load balancer integration
- ✅ Structured logging with configurable levels (JSON/console)
- ✅ Performance metrics (parsing duration, success/failure rates)
Quick Start
# Download and install
curl -L -o parsedmarc-go https://github.com/ZenProjects/parsedmarc-go/releases/latest/download/parsedmarc-go-linux-amd64
chmod +x parsedmarc-go
# Parse a report
./parsedmarc-go -input report.xml
# Run as daemon
./parsedmarc-go -daemon -config config.yaml
For detailed usage instructions, see the 📖 Documentation below.
🗄️ ClickHouse Database Schema
The program automatically creates optimized production-ready tables with proper indexing, partitioning, and performance optimizations:
📋 dmarc_aggregate_reports
Main aggregate report metadata table
- Report metadata (organization, report ID, date range)
- Policy information (DMARC alignment settings)
- Monthly partitioning by report date
- Bloom filter indexes on org_name and report_id
📊 dmarc_aggregate_records
Individual aggregate report records
- Source IP analysis (IP, country, reverse DNS)
- Authentication results (SPF, DKIM, DMARC alignment)
- Message counts and policy evaluation results
- Monthly partitioning with geolocation indexing
🔍 dmarc_forensic_reports
Forensic/failure report details
- Authentication failure analysis
- Source information and sample headers
- Parsed sample message content
- Indexed by arrival date and source IP
🔐 dmarc_smtp_tls_reports
SMTP TLS report metadata
- Organization and policy information
- Success/failure session counts
- Policy domain and type information
- Time-based partitioning for performance
⚠️ dmarc_smtp_tls_failures
Detailed SMTP TLS failure analysis
- Failure types and error codes
- MTA connection details (sending/receiving IPs)
- MX hostname and HELO information
- Normalized for efficient failure pattern analysis
🚀 Performance Features
- Time-based partitioning: Monthly partitions for optimal query performance
- Bloom filter indexes: Fast lookups on report IDs and domains
- Optimized data types: Efficient storage with proper nullable fields
- Query-optimized structure: Denormalized where appropriate for analytics
🔧 Advanced Email Format Support
parsedmarc-go features industry-leading email format compatibility, automatically handling complex report formats from major email service providers:
🎯 Forensic Reports (RUF) - Universal Compatibility
Plain Text Format
Simple feedback reports embedded directly in email body text
MIME Multipart Email Formats ⭐
Automatically parsed with full provider compatibility:
| Provider | Format | Encoding | Content-Type |
|---|---|---|---|
multipart/report |
Plain text | message/feedback-report |
|
| Domain.de | multipart/report |
Plain text | message/feedback-report; name=report |
| Netease | multipart/mixed |
Base64 | message/feedback-report; name="ATT00001" |
| Others | Auto-detected | Base64/Plain | Various MIME types |
🚀 Advanced Processing Pipeline:
- Multi-line header parsing - Handles wrapped Content-Type headers
- MIME boundary extraction - Robust parsing of complex boundaries
- Base64 decoding - Automatic detection and decoding
- Content-type detection - Intelligent format recognition
- Fallback mechanisms - Plain text parsing if MIME fails
📧 SMTP TLS Reports - Next-Generation Support
Direct JSON Format
Standard RFC 8460 JSON reports processed natively
Email-Based Reports ⭐
Advanced multi-stage processing pipeline:
| Provider | Format | Pipeline | Content-Type |
|---|---|---|---|
multipart/report |
Base64 → GZIP → JSON | application/tlsrpt+gzip |
|
| Others | Auto-detected | Base64 → Compression → JSON | application/tlsrpt+* |
🔄 Processing Pipeline:
Email Input → MIME Parse → Base64 Decode → GZIP Decompress → JSON Parse → Structured Data
📋 Supported Standards
parsedmarc-go implements the following email authentication and reporting standards with industry-leading compatibility:
- RFC 7489 - Domain-based Message Authentication, Reporting, and Conformance (DMARC)
- Aggregate reports (RUA) with enhanced parsing
- Policy configuration and validation
- 🆕 Enhanced error diagnostics with line numbers
- RFC 6591 - Authentication Failure Reporting Using the Abuse Reporting Format
- Forensic/failure reports (RUF) with MIME support
- 🆕 Advanced MIME multipart parsing
- 🆕 Base64-encoded attachment support
- RFC 8460 - SMTP TLS Reporting
- TLS connection and policy reporting with email format support
- 🆕 Email-based reports with compression support
- 🆕 Complete ClickHouse schema for analytics
📚 Documentation
📖 Table of Contents
🚀 Getting Started
- 📋 Complete Documentation - Overview and architecture
- ⚡ Installation - Installation and initial setup
- ⚙️ Configuration - Detailed configuration options
- 💡 Usage - Usage guide and examples
🗄️ Database and Visualization
- 🗃️ ClickHouse - ClickHouse configuration and optimization
- 📊 Grafana - Dashboards and visualizations
- 📈 Monitoring - Prometheus metrics and monitoring
🔌 API and Integrations
- 🌐 HTTP API - HTTP endpoints and integrations
- 📧 DMARC Mailing Lists - Mailing list configuration
📖 Technical References
- 🔒 DMARC Specification - DMARC standards details
- 🤝 Contributing - Project contribution guide
🔗 Quick Links
- Quick Setup - Get started in 5 minutes
- API Examples - Ready-to-use HTTP integrations
- Grafana Dashboards - Pre-configured visualizations
- Prometheus Metrics - Complete monitoring
💡 Tip: Start with the complete documentation for an overview, then check the installation guide to get started quickly.
License
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.
🙏 Acknowledgments
- Sean Whalen for the original Python parsedmarc project
- Claude AI for comprehensive Go conversion and advanced feature development
📞 Issues