Auto-Parsing Logs

This will give you some insights on how LogDNA parses logs

LogDNA can intelligently detects log types, parse them to display optimally, and more importantly index them in a way that makes keyword search fast and easy to use. Custom parsing rules can be configured for any proprietary formats as well.

Supported Types

LogDNA automatically parses the following log line types:

  • Akamai
  • Ansible
  • Apache Common Log Format
  • Apache Combined Log Format
  • Aptible
  • AWS CloudWatch
  • AWS ELB
  • AWS ECS
  • AWS S3
  • Cron
  • Docker Swarm
  • Docker Cloud/Compose
  • GitHub
  • Golang
  • HAProxy
  • Heroku
  • HTTPD
  • IIS Log
  • JSON
  • Logfmt
  • LogSpout
  • Rancher
  • MongoDB
  • Nagios
  • Nginx Error Log Format
  • PostgreSQL
  • Redis
  • Ruby/Rails
  • Syslog
  • Tomcat
  • Windows Events

JSON Parsing

As long as the log message ends in a }, your JSON object will be parsed, even if the JSON object does not span the entire message. If do not want your JSON object to be parsed, you can simply append an additional character after the ending } such as . a period.

If your JSON contains a message field, that field will be used for display and search in the log viewer. We also parse out (and override any existing) log levels if you include a level field.

Reserved fields

For JSON parsed lines, LogDNA uses a number of reserved fields to keep track of specific types of data. Please note that using the following reserved fields in your root JSON object will result in an underscore (_) prepended to those fields inside the context menu (e.g. internally status is stored as _status). However, you can still search normally inside our web app without being aware of this storage behavior (e.g. you can still just search status:200 as we will automatically search both status and _status). For reference, common reserved fields can be found below:

  • _source
  • _type
  • _tag
  • auth
  • bytes
  • connect
  • method
  • namespace
  • path
  • pod
  • request
  • response
  • service
  • space
  • status
  • timestamp
  • user

If you are using custom log formats, you can create a custom parsing template to tell LogDNA's parsing engine how to parse your log lines. Refer to the documentation to learn more.

Line components

Nearly all log line strings contain the three components below.

Timestamp

The LogDNA code libraries and agent automatically provide a timestamp that is based on the time that LogDNA ingests the log. If there is also included in the log line a timestamp from when the log was written, it will get parsed and you can view it by expanding the section. The written-time timestamp is not used further by our system. LogNDA generated timestamps follow the ISO 8601 format. If you have further questions about timestamps, contact us.

Log Level

Log level typically follows timestamp and is automatically parsed. We look for common formats, such as a timestamp followed by a separator followed by the log level. Common log levels include:

  • CRITICAL
  • DEBUG
  • EMERGENCY
  • ERROR
  • FATAL
  • INFO
  • SEVERE
  • TRACE
  • WARN

Message

Message is a string that represents the core descriptive component of a log line and is usually preceded by timestamp and level. A message typically contains a mixture of static and variable substrings and allows for easy human interpretation. For example:

User [email protected] requested /API/accountdetails/

Source information

Source information metadata is also ingested alongside the log line and is displayed in the All Sources menu in the web app.

Hostname

A hostname is the name of the source of the log line, and is automatically picked up by the LogDNA agent as well as syslog-based ingestion. However, a host must be specified when submitting lines via the REST API or code libraries.

Tags

A tag can be used to group lines and more than one tag can be applied to a given line. Tags show up under the All Tags menu in the web app. Tagging is supported by both the LogDNA agent as well as custom-template supported syslog-based ingestion, such as rsyslog or syslog-ng. At the time of writing, only source tags are currently supported, but more types are planned.

Other information

Other optional source information can be specified, such as

  • IP address
  • MAC address

The above information is automatically picked up by the LogDNA agent and can be specified for the REST API. The LogDNA agent also picks up some instance metadata, such as instance type.

Application information

In addition to source information, app information is also ingested. The LogDNA agent automatically parses the app name as the filename (e.g. error.log) while syslog-based ingestion uses the syslog-generated APP-NAME tag. For the REST API and code libraries, the app name must be specified.

Metadata

Metadata is a field reserved for custom information associated with a log line. Sending metadata is currently supported by the REST API, as well as our Node.JS, and Python code libraries.

Updated 2 months ago


Auto-Parsing Logs


This will give you some insights on how LogDNA parses logs

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.