AWS recently introduced logging from the pods directly to CloudWatch; see user guide. The FluentBit setup could achieve that, but it’s a bit tricky. For those who would expect a side container in the pod the way it’s described at “How to capture application logs when using Amazon EKS on AWS Fargate”, you might be surprised.
The user guide helped a lot, and after a setup of the namespace, I was almost done. The first thing which seemed to be a bit odd was the difference between the user guide and the blog post. Both described the output configuration of FluentBit differently. The guide has the following snippet:
[OUTPUT] Name cloudwatch_logs Match * region <us-east-1> log_group_name fluent-bit-cloudwatch log_stream_prefix from-fluent-bit- auto_create_group true
The problem is that the configuration didn’t work until I changed the region to `us-east-1` instead of `<us-east-1>`. I am not sure if that was the root cause, but it worked for me. I ended up with:
[OUTPUT] Name cloudwatch_logs Match * region us-east-1 log_group_name my-beloved-project log_stream_prefix app- log_key log auto_create_group On
If you were able to put the original configuration into work, please, let me know in the comments. I wasn’t that lucky. The other thing I had to add was the parser configuration:
[PARSER] Name docker Format json Time_Key time
Without the parser, I was not able to make JSON logs work. It was always displayed in the CloudWatch as `stdout`. At the end, I ended up with this config map:
apiVersion: v1 data: output.conf: | [OUTPUT] Name cloudwatch_logs Match * region us-east-1 log_group_name my-beloved-project log_stream_prefix app- log_key log auto_create_group On parsers.conf: |- [PARSER] Name docker Format json Time_Key time kind: ConfigMap metadata: name: aws-logging namespace: aws-observability
If you wouldn’t like to do the setup on your own, use the published terraform module at https://registry.terraform.io/modules/AckeeCZ/fargate-logs/aws/latest
