The LogDNA Amazon S3 integration relies on AWS Lambda to route your logs from S3 to LogDNA.
Configure the LogDNA AWS Lambda function
- Create a new lambda function and select
Author from scratch
- For the basic information:
- Function Name:
logdna-s3
(you can choose what to name it) - Runtime:
Node.js.10.x
- Click on the Lambda function to edit the details:
- Code entry type:
Upload a .ZIP file
- Upload our LogDNA Lambda function .ZIP File
- Handler:
index.handler
- Runtime:
Node.js.10.x
- Environment variables:
LOGDNA_KEY
:YOUR_INGESTION_KEY_HERE
(Required)LOGDNA_HOSTNAME
: Alternative Host Name (Optional)LOGDNA_TAGS
: Comma-separated Tags (Optional)LOGDNA_URL
: Custom Ingestion URL (Optional)
- For Execution role, assign a role that has the following policies:
- It is recommended to set the
Timeout
to 30 seconds if the function is going to be used to stream the logs fromgzipped
files. You may changeMemory (MB)
limit as well depending on how heavy those files are going to be
Configure your AWS S3 Bucket
You have the option of connecting your Amazon S3 Bucket within the S3 Lambda function console or in your S3 console.
In the S3 Lambda Function
- Add S3 as a Trigger and click it to Configure:
- Select the
Bucket
andEvent type
to stream the logs from to LogDNA
a. Optional You can also specifyPrefix
andSuffix
for the files to capture - Make sure you check
Enable trigger
box:
- Repeat steps 1-3 to add multiple buckets
Optional Environment Variables
The following variables can be used to tune this S3 Lambda function for specific use cases.
- LOGDNA_BATCH_INTERVAL: How frequently (in
milliseconds
) to flush the batch of logs, Optional
* Default: 50 - LOGDNA_BATCH_LIMIT: The maximum number of logs in one batch, Optional
* Default: 25 - LOGDNA_FREE_SOCKET_TIMEOUT: How long (in
milliseconds
) to wait for inactivity before timing out on the free socket, Optional- Default: 300000
- Source: agentkeepalive#agentoptions
- LOGDNA_MAX_LINE_LENGTH: The maximum character length for each line, Optional
- Default: 32000
- LOGDNA_MAX_REQUEST_RETRIES: The maximum number of retries for sending a line when there are network failures, Optional
- Default: 5
- LOGDNA_MAX_REQUEST_TIMEOUT: Time limit (in
seconds
) for requests made by this HTTP Client, Optional- Default: 300
- LOGDNA_REQUEST_RETRY_INTERVAL: How frequently (in
milliseconds
) to retry for sending a line when there are network failures, Optional- Default: 100
Notes
- This S3 Lambda function and S3 Bucket should be in the same region
- You can specify a bucket only in one trigger and/or S3 Lambda function since a bucket accepts only one subscription
Node.js.10.x
is the minimum runtime requirement for successfully running this S3 Lambda function
Updated 5 months ago