Our Secret Sauce Architecture

LogDNA uses a microservice-based architecture to split different tasks into discrete and scalable units. These microservices can be organized into two roles: log ingestion, and log retrieval.

Logs sent from your log sources to LogDNA are received by one of many ingestion endpoints. These endpoints route each log to a message queue using a proprietary data pipeline. The message queues manage the delivery of logs to a number of worker pools, which process logs for use in various LogDNA services such as parsing, indexing, live tail, graphs, and alerts. Other microservices provide features such as hosting API endpoints, providing security and authentication, and maintaining log data stores.

Ingestion Services
Ingestion services provide the backend components necessary to ingest, parse, index, and store logs. They receive logs sent by log generating components, as well as fulfill requests for log data sent from other LogDNA services.

Logs arrive into the LogDNA infrastructure through ingestion endpoints, which route logs from your applications and systems to a proprietary message brokering service called Buzzsaw. Buzzsaw is a highly optimized and highly scalable brokering service designed specifically for log data.

Buzzsaw routes logs to worker pools, which consist of microservices performing specific actions such as parsing, generating graphs, and running alerts. It also sends logs to a cluster of Elasticsearch nodes, which provide LogDNA's search and filtering functionality. Buzzsaw acts as a buffer between ingestion sources, worker pools, and Elasticsearch nodes, ensuring that performance problems in any one service doesn't impact the performance of other services.

Retrieval Services
Retrieval services provide the ability to access, view, and export log data. They act as the gateway between users and the LogDNA ingestion services and are responsible for actions such as responding to user requests, querying Elasticsearch for saved log data, and live tailing logs.

The primary services provided by retrieval services are the user interfaces, which include the LogDNA web UI and REST API. Actions performed in the LogDNA web application, command line interface (CLI), or REST API are received by these services and proxied to the relevant infrastructure services before returning the results to the user. These services also perform user authentication and enforce access controls.

Other retrieval services include features that export log data automatically, such as alerts and archives.

**

Updated about a year ago

Our Secret Sauce Architecture


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.