Usage Guide

This guide covers how to use parsedmarc-go in various scenarios.

Command Line Usage

Command Line Options

Usage of parsedmarc-go:
  -config string
        Config file path (default "config.yaml")
  -daemon
        Run as daemon (enables IMAP and HTTP)
  -format string
        Output format: json, csv (default "json")
  -input string
        Input file or directory to parse
  -output string
        Output file or directory path (default: stdout)
  -version
        Show version information

Basic Report Parsing

Parse a single DMARC report file:

# Parse XML aggregate report
parsedmarc-go -input report.xml

# Parse forensic email (with MIME attachments)
parsedmarc-go -input forensic-report.eml

# Parse SMTP TLS email (with compressed attachments) 
parsedmarc-go -input smtp-tls-report.eml

Output Options

Output to JSON file

parsedmarc-go -input report.xml -output results.json -format json

Output to CSV file

parsedmarc-go -input report.xml -output results.csv -format csv

Output to directory (separate files per report)

# Create output directory
mkdir ./reports_output

# Each report will be saved as a separate file with timestamp
parsedmarc-go -input report.xml -output ./reports_output -format json
# Creates: reports_output/aggregate_20240101_120000_reportID.json

# In daemon mode, each incoming report creates a new file
parsedmarc-go -daemon -output ./reports_output -format json

Output to stdout (default)

parsedmarc-go -input report.xml -format json

Parsing Multiple Files

Parse all files in a directory:

# Concatenate all reports into a single file
parsedmarc-go -input /path/to/reports/ -output all_reports.json -format json

# Save each report as a separate file
parsedmarc-go -input /path/to/reports/ -output ./output_dir/ -format json

Daemon Mode

IMAP + HTTP Mode (Full Daemon)

Modify this section of the config.yml:

# IMAP configuration for fetching reports from email
imap:
  enabled: true                          # Enable IMAP client
  host: "imap.host.com"                  # IMAP server hostname
  port: 993                              # IMAP server port (993 for TLS, 143 for plain)
  username: "user"                       # IMAP username
  password: "pass"                       # IMAP password
  tls: true                              # Use TLS/SSL connection
  skip_verify: false                     # Skip TLS certificate verification
  mailbox: "INBOX"                       # Mailbox to monitor
  archive_mailbox: "DMARC-Archive"       # Mailbox to move processed emails
  delete_processed: false                # Delete processed emails instead of archiving
  check_interval: 300                    # Check interval in seconds (5 minutes)

# HTTP server configuration for receiving reports
http:
  enabled: true                          # Enable HTTP server
  host: "0.0.0.0"                        # Host to bind to
  port: 8080                             # Port to listen on
  tls: false                             # Enable TLS/HTTPS
  cert_file: ""                          # TLS certificate file path (required if tls: true)
  key_file: ""                           # TLS private key file path (required if tls: true)
  rate_limit: 60                         # Requests per minute per IP
  rate_burst: 10                         # Burst capacity for rate limiter
  max_upload_size: 52428800              # Max upload size in bytes (50MB)

parsedmarc-go -daemon -config config.yaml

HTTP Server Only

Modify this section of the config.yml:

# IMAP configuration for fetching reports from email
imap:
  enabled: false                          # Disable IMAP client

# HTTP server configuration for receiving reports
http:
  enabled: true                          # Enable HTTP server
  host: "0.0.0.0"                        # Host to bind to
  port: 8080                             # Port to listen on
  tls: false                             # Enable TLS/HTTPS
  cert_file: ""                          # TLS certificate file path (required if tls: true)
  key_file: ""                           # TLS private key file path (required if tls: true)
  rate_limit: 60                         # Requests per minute per IP
  rate_burst: 10                         # Burst capacity for rate limiter
  max_upload_size: 52428800              # Max upload size in bytes (50MB)

# Enable HTTP in config.yaml then:
parsedmarc-go -daemon

IMAP Client Only

Modify this section of the config.yml:

# IMAP configuration for fetching reports from email
imap:
  enabled: true                          # Enable IMAP client
  host: "imap.host.com"                  # IMAP server hostname
  port: 993                              # IMAP server port (993 for TLS, 143 for plain)
  username: "user"                       # IMAP username
  password: "pass"                       # IMAP password
  tls: true                              # Use TLS/SSL connection
  skip_verify: false                     # Skip TLS certificate verification
  mailbox: "INBOX"                       # Mailbox to monitor
  archive_mailbox: "DMARC-Archive"       # Mailbox to move processed emails
  delete_processed: false                # Delete processed emails instead of archiving
  check_interval: 300                    # Check interval in seconds (5 minutes)

# HTTP server configuration for receiving reports
http:
  enabled: false                          # Disable HTTP server

# Enable IMAP in config.yaml then:
parsedmarc-go -daemon

With Custom Configuration File

parsedmarc-go -config /path/to/config.yaml -input report.xml

Environment Variables

You can also use environment variables for configuration:

export CLICKHOUSE_HOST=clickhouse.example.com
export CLICKHOUSE_PORT=9000
export CLICKHOUSE_USERNAME=myuser
export CLICKHOUSE_PASSWORD=mypassword
export IMAP_HOST=imap.example.com
export IMAP_USERNAME=dmarc@example.com
export IMAP_PASSWORD=password
export HTTP_ENABLED=true
export HTTP_PORT=8080

parsedmarc-go -daemon

Show Version

parsedmarc-go -version

Configuration File

Create a configuration file to customize parsedmarc-go behavior:

logging:
  level: info
  format: console

parser:
  output_format: json
  strip_attachment_payloads: false
  always_use_local_nameservers: false
  nameservers:
    - "1.1.1.1"
    - "8.8.8.8"

clickhouse:
  enabled: true
  host: localhost
  port: 9000
  username: default
  password: ""
  database: dmarc
  dial_timeout: 10s
  max_open_conns: 10

imap:
  enabled: true
  host: imap.gmail.com
  port: 993
  username: your-email@example.com
  password: your-app-password
  folders:
    - INBOX
  delete_processed: false
  reports_folder: "DMARC Reports"
  check_interval: 300s
  tls: true

http:
  enabled: true
  listen: ":8080"
  max_upload_size: 10MB
  rate_limit: 100
  read_timeout: 30s
  write_timeout: 30s

IMAP Configuration

Gmail Setup

Enable 2-factor authentication
Create an app password for parsedmarc-go
Configure IMAP settings:

imap:
  enabled: true
  host: imap.gmail.com
  port: 993
  username: your-email@gmail.com
  password: your-app-password
  tls: true
  folders:
    - INBOX
  delete_processed: false
  reports_folder: "DMARC Reports"

Microsoft 365 Setup

imap:
  enabled: true
  host: outlook.office365.com
  port: 993
  username: your-email@company.com
  password: your-password
  tls: true
  folders:
    - INBOX
  delete_processed: false
  reports_folder: "DMARC"

Custom IMAP Server

imap:
  enabled: true
  host: mail.example.com
  port: 993
  username: dmarc@example.com
  password: your-password
  tls: true
  folders:
    - INBOX
    - "DMARC Reports"
  delete_processed: true
  check_interval: 600s

HTTP Endpoint Usage

Submit DMARC Reports via HTTP

The HTTP endpoint accepts DMARC reports in various formats:

XML Reports (RFC 7489)

curl -X POST http://localhost:8080/dmarc/report \
  -H "Content-Type: application/xml" \
  -d @report.xml

Compressed Reports

# Gzip compressed
curl -X POST http://localhost:8080/dmarc/report \
  -H "Content-Type: application/gzip" \
  -H "Content-Encoding: gzip" \
  --data-binary @report.xml.gz

# Zip compressed
curl -X POST http://localhost:8080/dmarc/report \
  -H "Content-Type: application/zip" \
  --data-binary @report.zip

Multipart Form Data

curl -X POST http://localhost:8080/dmarc/report \
  -F "report=@report.xml"

Health Check

Check if the service is running:

curl http://localhost:8080/health

Response:

{
  "status": "ok",
  "timestamp": "2024-12-01T10:30:45Z"
}

Metrics Endpoint

Access Prometheus metrics:

curl http://localhost:8080/metrics

Report Types

parsedmarc-go supports different types of DMARC reports:

Aggregate Reports

Standard DMARC aggregate reports (RFC 7489)
Contain statistical data about email authentication
Stored in dmarc_aggregate_reports and dmarc_aggregate_records tables

Forensic Reports

DMARC failure reports (RFC 6591)
Contain detailed information about individual failed messages
Stored in dmarc_forensic_reports table

SMTP TLS Reports

SMTP TLS reporting (RFC 8460)
Information about TLS usage in SMTP connections
Stored in smtp_tls_reports table

Output Formats

JSON Output

When parsing files directly, parsedmarc-go outputs structured JSON:

{
  "report_metadata": {
    "org_name": "google.com",
    "email": "noreply-dmarc-support@google.com",
    "extra_contact_info": "",
    "report_id": "12345678901234567890",
    "date_range": {
      "begin": "2024-11-30T00:00:00Z",
      "end": "2024-11-30T23:59:59Z"
    }
  },
  "policy_published": {
    "domain": "example.com",
    "adkim": "r",
    "aspf": "r",
    "p": "none",
    "sp": "none",
    "pct": 100
  },
  "records": [
    {
      "source": {
        "ip": "192.0.2.1",
        "country": "US",
        "reverse_dns": "mail.example.net",
        "base_domain": "example.net"
      },
      "count": 15,
      "alignment": {
        "spf": "pass",
        "dkim": "fail",
        "dmarc": "fail"
      },
      "identifiers": {
        "header_from": "example.com"
      }
    }
  ]
}

ClickHouse Storage

When ClickHouse is enabled, reports are automatically stored in normalized tables optimized for analytics and reporting.

Performance Tuning

Parser Configuration

parser:
  # Process reports concurrently
  max_workers: 4
  
  # Reduce memory usage for large attachments
  strip_attachment_payloads: true
  
  # Use local nameservers for better performance
  always_use_local_nameservers: true
  nameservers:
    - "127.0.0.1"
    - "1.1.1.1"

ClickHouse Optimization

clickhouse:
  # Increase connection pool for high throughput
  max_open_conns: 50
  max_idle_conns: 10
  
  # Batch inserts for better performance
  batch_size: 1000
  batch_timeout: 5s
  
  # Connection settings
  dial_timeout: 30s
  read_timeout: 60s
  write_timeout: 60s

HTTP Server Tuning

http:
  # Increase limits for high volume
  max_upload_size: 50MB
  rate_limit: 1000
  
  # Adjust timeouts
  read_timeout: 60s
  write_timeout: 60s
  idle_timeout: 120s
  
  # Enable compression
  compression: true

Monitoring and Logging

Log Levels

Set appropriate log levels:

logging:
  level: info  # debug, info, warn, error
  format: json # json, console
  output: stdout # stdout, stderr, or file path

Structured Logging

parsedmarc-go uses structured logging with contextual information:

Request IDs for HTTP requests
Report metadata for parsing operations
Performance metrics for database operations

Metrics

Monitor these key Prometheus metrics:

parsedmarc_reports_processed_total: Total reports processed
parsedmarc_reports_failed_total: Failed report processing
parsedmarc_http_requests_total: HTTP endpoint usage
parsedmarc_imap_messages_processed: IMAP message processing

Troubleshooting & FAQ

Common Issues & Solutions

🚨 Parsing Errors

Q: “XML syntax error at line X” - What does this mean?

# ✅ Enhanced error reporting now shows exact line numbers
Error: XML syntax error at line 15: expected attribute name in element

Solution: Check the XML file at the specified line for malformed tags, missing quotes, or invalid characters.

Q: “No feedback report found” for email files

# ❌ Old behavior: Generic error
Error: no feedback report found

# ✅ New behavior: Detailed diagnostics  
Debug: MIME parsing found 3 parts, no feedback-report content-type detected

Solution: The email may use a non-standard MIME structure. Enable debug logging to see MIME parsing details.

🔗 DNS Resolution Problems

parser:
  always_use_local_nameservers: false
  nameservers:
    - "8.8.8.8"
    - "1.1.1.1"

📧 IMAP Connection Issues

Check firewall settings for port 993/143
Verify TLS settings match server requirements
Ensure credentials are correct (use app passwords for Gmail)

🗃️ ClickHouse Connection Problems

Verify ClickHouse is running and accessible
Check database permissions
Ensure network connectivity

💾 High Memory Usage

Enable strip_attachment_payloads: true
Reduce max_workers if processing large files
Monitor with parsedmarc_memory_usage_bytes metric

🐛 Debug Mode

Enable detailed logging for troubleshooting:

# config.yaml
logging:
  level: debug    # Shows MIME parsing details
  format: console # Human-readable format

# Command line debug
./parsedmarc-go -input problem-report.eml 2>&1 | grep -E "(DEBUG|ERROR)"

🧪 Testing Your Setup

Validate Parser Functionality

# Test with sample files
./parsedmarc-go -input samples/aggregate/!example.com!1538204542!1538463818.xml
./parsedmarc-go -input samples/forensic/netease-report.eml  
./parsedmarc-go -input samples/smtp_tls/google-report.eml

Test ClickHouse Connection

# Test database connectivity
echo "SELECT version()" | clickhouse-client --host localhost --port 9000

Test HTTP API

# Test basic connectivity
curl -X GET http://localhost:8080/health

# Submit test report
curl -X POST http://localhost:8080/dmarc/report \
  -H "Content-Type: application/xml" \
  --data @test-report.xml

Testing Configuration

Validate your configuration:

parsedmarc-go -config config.yaml -input test-report.xml

Integration Examples

Docker Compose

version: '3.8'
services:
  parsedmarc-go:
    image: parsedmarc-go:latest
    ports:
      - "8080:8080"
    volumes:
      - ./config.yaml:/etc/parsedmarc-go/config.yaml
      - ./reports:/var/lib/parsedmarc-go/reports
    depends_on:
      - clickhouse
    
  clickhouse:
    image: clickhouse/clickhouse-server:latest
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - clickhouse_data:/var/lib/clickhouse
    environment:
      CLICKHOUSE_DB: dmarc

volumes:
  clickhouse_data:

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: parsedmarc-go
spec:
  replicas: 3
  selector:
    matchLabels:
      app: parsedmarc-go
  template:
    metadata:
      labels:
        app: parsedmarc-go
    spec:
      containers:
      - name: parsedmarc-go
        image: parsedmarc-go:latest
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /etc/parsedmarc-go
        env:
        - name: CLICKHOUSE_HOST
          value: "clickhouse-service"
      volumes:
      - name: config
        configMap:
          name: parsedmarc-go-config

Systemd Service

[Unit]
Description=parsedmarc-go DMARC Report Parser
After=network.target

[Service]
Type=simple
User=parsedmarc
ExecStart=/usr/local/bin/parsedmarc-go -daemon -config /etc/parsedmarc-go/config.yaml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target