Reference

Architecture¶

LuceDev Syslog runs as a single Python process with multiple cooperating threads, backed by a tuned SQLite database. This page describes how the pieces fit together for operators who need to debug or capacity-plan.

High-level view¶

flowchart LR
    subgraph Devices["Network Devices"]
      D1[Firewalls]
      D2[Switches]
      D3[Servers]
    end

    subgraph Process["LuceDevSyslog process"]
      UDP[UDP 514
Receiver]
      TLS[TLS 6514
Receiver]
      BUF[Write Buffer
1s flushes]
      HTTP[HTTP 5000
Always on]
      HTTPS[HTTPS 5001
Optional]
      CLEAN[Cleanup
Hourly]
      LIC[License Checker
Daily]
    end

    DB[(SQLite
WAL + FTS5)]
    ARCH[(Archives
.csv.gz)]

    Devices --> UDP
    Devices --> TLS
    UDP --> BUF
    TLS --> BUF
    BUF --> DB
    HTTP --> DB
    HTTPS --> DB
    CLEAN --> DB
    CLEAN --> ARCH
    LIC -.->|cached| UDP
    LIC -.->|cached| TLS

Process model¶

The server is one OS process. Inside it, several daemon threads cooperate:

• syslog_server — runs the UDP receiver, blocking on a socket. Spawns a TLS-TCP receiver in another thread when TLS syslog is enabled.
• web_server — runs Waitress for HTTP. Spawns a second Waitress instance in another thread for HTTPS when enabled.
• db-flush — wakes every second, drains the in-memory write buffer to SQLite in a batch INSERT.
• cleanup — wakes every hour, runs archival (if enabled) then deletes expired logs in 50,000-row batches.
• license-checker — wakes every 24 hours, calls the license server if a key is configured.
• main — monitors the syslog and web threads, restarts them if they die.

This model fits PyInstaller's single-binary expectations and avoids the complexity of multi-process IPC for what is, fundamentally, an I/O-bound workload.

Storage layout¶

data/syslog.db — the live database¶

SQLite, in WAL mode, with several tuning PRAGMAs applied at connection open:

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA busy_timeout=10000;
PRAGMA cache_size=-64000;    -- 64 MB page cache
PRAGMA mmap_size=268435456;  -- 256 MB memory map
PRAGMA temp_store=MEMORY;
PRAGMA wal_autocheckpoint=1000;

Schema:

Audit events are stored in logs with host='LuceDev-Syslog' and facility='audit'.

Compound indexes¶

The logs table carries seven indexes tuned for the queries the dashboard actually runs:

• idx_logs_timestamp — descending timestamp
• idx_logs_host, idx_logs_severity — single-column filters
• idx_logs_id_desc — for "first page" queries
• idx_logs_ts_host_sev — covering compound
• idx_logs_sev_id, idx_logs_host_id, idx_logs_host_sev_id — filtered+sorted queries without a full-table sort

Hot path: ingesting a log¶

flowchart TD
    A[Packet arrives on UDP 514] --> B[Parse RFC 3164 PRI field]
    B --> C{License check
cached 60s}
    C -- not licensed --> Z[Drop silently]
    C -- licensed --> D[Append to in-memory
write buffer]
    D --> E[Severity alert?]
    E -- yes --> F[Queue background
SMTP send]
    E -- no --> G[Done]
    F --> G

    H[db-flush thread
every 1s] --> I[Drain buffer]
    I --> J[Batched INSERT into logs]
    J --> K[Sync FTS index]
    K --> L[Update stats_cache &
hourly_stats]

The in-memory buffer is the key to throughput. Without it, every syslog message would be a INSERT + COMMIT — capped at ~500 messages per second on commodity hardware. With it, the receiver hands off in microseconds and the flush thread commits up to 5,000 messages per batch.

Query hot paths¶

The dashboard tries hard not to scan logs directly:

• Unfiltered total count → reads from stats_cache
• Single-host or single-severity count → reads from stats_cache
• Chart data (logs per hour, top hosts, severity over time) → reads from hourly_stats
• Keyword search → uses the FTS5 index with prefix matching, not LIKE
• Filtered counts on multiple fields → bounded scan capped at 500K rows to avoid multi-second waits on very large databases

If you delete or archive a large chunk of logs, run Rebuild Stats Cache from Settings → Database → Maintenance to bring the cache back in sync.

Ports¶

Each TCP listener must use a unique port; the server validates this on save.

Restart behaviour¶

The "Restart Service" button on the Settings → Server tab works differently per platform:

• Linux — issues sudo systemctl restart lucedev-syslog
• Windows — creates a one-shot Windows Task Scheduler task running as SYSTEM that runs net stop + net start + cleans itself up. This approach is necessary because a child process of the service would itself be killed when the service stops.

The "Restart Service" overlay polls the dashboard URL until it comes back, then auto-reloads.

What's not in the architecture¶

• No external services required — no Redis, no Postgres, no message broker. Everything is in-process.
• No background workers — alerts go out from a fire-and-forget background thread, not a queue.
• No analytics or telemetry — the only outbound calls are the daily update check and the weekly license re-validation.