Home

Laravel logs collector by golang

GitHub Repository: https://github.com/sithuaung/lara_log_collector

I built a Laravel log collector daemon in Go to solve a common problem: monitoring logs across multiple Laravel applications at scale. Instead of manually reviewing log files or waiting for users to report issues, this tool actively monitors logs and sends grouped alerts to Lark (similar to Slack) webhooks.

The Problem It Solves

In production environments with multiple Laravel applications, manually checking log files is inefficient and reactive. This tool provides proactive log monitoring by tailing Laravel log files and alerting your team about errors in real-time, making it easier to catch and respond to issues before they impact users.

Core Architecture

The log collector follows a pipeline architecture with five key components:

1. Watcher

  • Polls storage/logs/laravel-YYYY-MM-DD.log files at configurable intervals
  • Saves the last read offset per log directory to avoid re-sending entries after restarts
  • Enables safe daemon restarts without duplicate alerts

2. Parser

  • Extracts structured data from Laravel’s standard log format: [YYYY-MM-DD HH:MM:SS] env.LEVEL: message
  • Handles multiple log levels from DEBUG to EMERGENCY
  • Provides clean, structured data for downstream processing

3. Buffer

  • In-memory queue with configurable capacity
  • Handles overflow gracefully to prevent memory issues
  • Decouples log reading from log sending for better performance

4. Sender

  • Batches log entries before sending to reduce API calls
  • Implements exponential backoff retry logic for failed requests
  • Posts formatted messages to Lark webhooks

5. Suppressor

  • Filters and deduplicates unimportant errors
  • Supports both substring and regex pattern matching
  • Case-insensitive matching options to reduce noise

Key Design Patterns

Separation of Concerns

Each module has a distinct responsibility (file I/O, parsing, queuing, transmission, filtering), making the codebase maintainable and testable. You can modify the parser without touching the sender, or swap out Lark for another notification service by only changing the sender module.

Batch Processing

Instead of sending each log entry individually, the collector buffers and batches entries before sending to Lark. This dramatically reduces:

  • API calls and network overhead
  • Rate limiting issues
  • Notification fatigue (your team gets grouped alerts instead of spam)

State Persistence

The watcher saves its last read offset for each monitored directory. This means:

  • Safe restarts without re-sending old logs
  • No duplicate alerts after crashes or deployments
  • Reliable operation in production environments

Pattern Matching Flexibility

The suppressor supports multiple matching strategies:

  • Simple substring matching for common errors
  • Regex patterns for complex filtering
  • Case-insensitive options for broader matching
  • Helps reduce alert noise while keeping important errors visible

Notable Implementation Details

Polling-Based Approach: Instead of using filesystem events (inotify/fsnotify), the tool polls log files at intervals. This avoids complexity around filesystem event handling and works reliably across different operating systems.

Stack Trace Omission: Stack traces are intentionally excluded from alerts to reduce memory usage and keep notifications concise. Teams can always check the full logs on the server if needed.

Timezone-Aware Scheduling: Daily summary reports respect configured IANA timezones, ensuring reports arrive at the right time for your team regardless of server location.

Environment Variable Overrides: Critical settings like LARK_WEBHOOK_URL and MIN_LOG_LEVEL can be overridden via environment variables, making containerized deployments simple.

Systemd Integration: Includes service unit templates for running the collector as a proper Linux system service with automatic restarts and logging.

Operational Features

Multi-App Monitoring: Configure multiple Laravel applications in one collector instance. The tool automatically derives application names from directory paths.

Configurable Log Levels: Filter logs by severity (DEBUG, INFO, WARNING, ERROR, etc.) per application or globally.

Graceful Error Handling: The collector validates webhook URLs at startup and handles runtime errors without crashing, ensuring reliable operation.

Production-Ready: Designed for 24/7 operation with proper error handling, state management, and integration with system service managers.

Why Go?

Building this in Go instead of PHP or Node.js offers several advantages:

  • Low resource usage: A single binary with minimal memory footprint
  • Easy deployment: No runtime dependencies—just copy the binary
  • Reliability: Go’s goroutines handle concurrent file monitoring efficiently
  • Performance: Fast log parsing and processing even with high-volume logs

Conclusion

This Laravel log collector demonstrates several important patterns for building reliable system tools: pipeline architectures for data processing, stateful operations with persistence, and practical batch processing for external APIs. Whether you’re monitoring one Laravel app or dozens, having automated log collection with intelligent filtering makes production operations significantly smoother.