-
Notifications
You must be signed in to change notification settings - Fork 6
Introduce Logging Stack: Add fluentd, add loki #1058
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
mrnicegyu11
wants to merge
33
commits into
ITISFoundation:main
Choose a base branch
from
mrnicegyu11:2025/add/fluentd
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
f0d8cf0
wip
mrnicegyu11 e906b41
Merge remote-tracking branch 'upstream/main' into main
mrnicegyu11 14c751d
Merge remote-tracking branch 'upstream/main' into main
mrnicegyu11 293f63c
Add csi-s3 and have portainer use it
mrnicegyu11 f7f72ec
Change request @hrytsuk 1GB max portainer volume size
mrnicegyu11 94cfb76
t push
mrnicegyu11 509c717
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 1a65ecf
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 77ee45e
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 c9c70d6
Arch Linux Certificates Customization
mrnicegyu11 7b8be53
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 bcd61cd
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 58e1030
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 ed8d479
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 dda6e01
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 f6f4f36
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 5dca5c3
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 4a653ef
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 3a21f0f
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 48fbbca
Fix pgsql exporter failure
mrnicegyu11 08c57db
Merge remote-tracking branch 'upstream/main'
mrnicegyu11 5ecbfec
[Kubernetes] Introduce on-prem persistent Storage (Longhorn) :tada: …
YuryHrytsuk 3ea41b5
Experimental: Try to add tracing to simcore-traefik on master
mrnicegyu11 1cf605d
Merge remote-tracking branch 'upstream/main' into 2025/add/fluentd
mrnicegyu11 bcc67d4
wip
mrnicegyu11 57947e3
Merge remote-tracking branch 'upstream/main' into 2025/add/fluentd
mrnicegyu11 88e4ed5
Merge remote-tracking branch 'upstream/main' into 2025/add/fluentd
mrnicegyu11 cae2c20
Merge remote-tracking branch 'upstream/main' into 2025/add/fluentd
mrnicegyu11 a7cb5ae
t push
mrnicegyu11 5e1766a
Fix accidental commit
mrnicegyu11 a91bcea
Fluentd fixes
mrnicegyu11 c39cbb1
Add placement constraints
mrnicegyu11 92070e2
revert arch linux changes
mrnicegyu11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
services: | ||
mongodb: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
elasticsearch: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
graylog: | ||
dns: # Add this always for AWS, otherwise we get "No such image: " for docker services | ||
8.8.8.8 | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
fluentd: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
loki: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
services: | ||
mongodb: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
elasticsearch: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
graylog: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
fluentd: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
loki: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true |
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,3 +13,8 @@ services: | |
deploy: | ||
placement: | ||
constraints: [] | ||
|
||
fluentd: | ||
deploy: | ||
placement: | ||
constraints: [] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
services: | ||
mongodb: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
elasticsearch: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
graylog: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
fluentd: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true | ||
|
||
loki: | ||
deploy: | ||
placement: | ||
constraints: | ||
- node.labels.logging==true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,197 @@ | ||
services: | ||
# MongoDB: https://hub.docker.com/_/mongo/ | ||
mongodb: | ||
image: mongo:6.0.6 | ||
init: true | ||
volumes: | ||
# data persistency | ||
- mongo_data:/data/db | ||
deploy: | ||
replicas: 1 | ||
restart_policy: | ||
condition: on-failure | ||
resources: | ||
limits: | ||
memory: 1.2G | ||
cpus: "1" | ||
reservations: | ||
memory: 300M | ||
cpus: "0.1" | ||
networks: | ||
graylog: | ||
aliases: | ||
- mongo # needed because of graylog configuration | ||
|
||
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html | ||
elasticsearch: | ||
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 | ||
init: true | ||
volumes: | ||
# data persistency | ||
- elasticsearch_data:/usr/share/elasticsearch/data | ||
environment: | ||
- http.host=0.0.0.0 | ||
- transport.host=localhost | ||
- network.host=0.0.0.0 | ||
- "ES_JAVA_OPTS=-Xms512m -Xmx512m" | ||
deploy: | ||
replicas: 1 | ||
restart_policy: | ||
condition: on-failure | ||
resources: | ||
limits: | ||
memory: 2G | ||
cpus: "2" | ||
reservations: | ||
memory: 1G | ||
cpus: "0.1" | ||
networks: | ||
graylog: | ||
# Graylog: https://hub.docker.com/r/graylog/graylog/ | ||
graylog: | ||
image: graylog/graylog:6.0.5 | ||
init: true | ||
# user: "1000:1001" | ||
configs: | ||
- source: graylog_config | ||
target: /files/osparc-custom-content-pack-v2.json | ||
volumes: | ||
# Mount local configuration directory into Docker container | ||
# - graylog_config:/usr/share/graylog/data/config | ||
# data persistency | ||
- graylog_journal:/usr/share/graylog/data/journal | ||
env_file: | ||
- .env | ||
environment: | ||
# CHANGE ME (must be at least 16 characters)! | ||
- GRAYLOG_PASSWORD_SECRET=${GRAYLOG_PASSWORD_SECRET} | ||
# Username: admin | ||
- GRAYLOG_ROOT_PASSWORD_SHA2=${GRAYLOG_ROOT_PASSWORD_SHA2} | ||
- GRAYLOG_HTTP_EXTERNAL_URI=${GRAYLOG_HTTP_EXTERNAL_URI} | ||
- GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200, | ||
networks: | ||
public: | ||
monitoring: | ||
graylog: | ||
aliases: | ||
- graylog | ||
ports: | ||
- 12201:12201/udp | ||
- 12202:12202/udp | ||
deploy: | ||
replicas: 1 | ||
restart_policy: | ||
condition: on-failure | ||
resources: | ||
limits: | ||
cpus: "2.00" | ||
memory: 5G | ||
reservations: | ||
cpus: "0.1" | ||
memory: 1G | ||
labels: | ||
- traefik.enable=true | ||
- traefik.docker.network=${PUBLIC_NETWORK} | ||
# direct access through port | ||
- traefik.http.services.graylog.loadbalancer.server.port=9000 | ||
- traefik.http.routers.graylog.rule=Host(`${MONITORING_DOMAIN}`) && PathPrefix(`/graylog`) | ||
- traefik.http.routers.graylog.entrypoints=https | ||
- traefik.http.routers.graylog.tls=true | ||
- traefik.http.middlewares.graylog_replace_regex.replacepathregex.regex=^/graylog/?(.*)$$ | ||
- traefik.http.middlewares.graylog_replace_regex.replacepathregex.replacement=/$${1} | ||
- traefik.http.routers.graylog.middlewares=ops_whitelist_ips@swarm, ops_gzip@swarm, graylog_replace_regex | ||
fluentd: | ||
image: itisfoundation/fluentd:v1.16.9-1.0 | ||
configs: | ||
- source: fluentd_config | ||
target: /fluentd/etc/fluent.conf | ||
environment: | ||
- GRAYLOG_HOST=graylog | ||
- GRAYLOG_PORT=12201 | ||
- LOKI_URL=http://loki:3100 | ||
- FLUENTD_HOSTNAME={% raw %}{{.Node.Hostname}}{% endraw %} | ||
ports: | ||
- "24224:24224/tcp" | ||
deploy: | ||
#mode: global # Run on all nodes | ||
restart_policy: | ||
condition: on-failure | ||
resources: | ||
limits: | ||
cpus: '1.0' | ||
memory: 1G | ||
reservations: | ||
cpus: '0.5' | ||
memory: 512M | ||
update_config: | ||
parallelism: 1 | ||
delay: 10s | ||
order: start-first | ||
networks: | ||
- monitoring | ||
- graylog | ||
healthcheck: | ||
test: ["CMD", "curl", "-f", "http://0.0.0.0:24220/api/plugins"] | ||
interval: 30s | ||
timeout: 10s | ||
retries: 3 | ||
start_period: 40s | ||
|
||
loki: | ||
image: grafana/loki:3.5.0 | ||
configs: | ||
- source: loki_config | ||
target: /etc/loki/loki.yaml | ||
command: -config.file=/etc/loki/loki.yaml | ||
deploy: | ||
placement: | ||
constraints: [] | ||
replicas: 1 | ||
restart_policy: | ||
condition: any | ||
delay: 5s | ||
resources: | ||
limits: | ||
cpus: '1.0' | ||
memory: 2G | ||
reservations: | ||
cpus: '0.5' | ||
memory: 1G | ||
update_config: | ||
parallelism: 1 | ||
delay: 10s | ||
order: start-first | ||
networks: | ||
- monitoring | ||
healthcheck: | ||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://0.0.0.0:3100/ready"] | ||
interval: 30s | ||
timeout: 10s | ||
retries: 3 | ||
start_period: 40s | ||
|
||
|
||
volumes: | ||
loki-data: | ||
mongo_data: | ||
elasticsearch_data: | ||
graylog_journal: | ||
|
||
networks: | ||
graylog: | ||
public: | ||
external: true | ||
name: ${PUBLIC_NETWORK} | ||
monitoring: | ||
external: true | ||
name: ${MONITORED_NETWORK} | ||
configs: | ||
graylog_config: | ||
name: ${STACK_NAME}_graylog_config_{{ "./data/contentpacks/osparc-custom-content-pack-v2.json" | sha256file | substring(0,10) }} | ||
file: ./data/contentpacks/osparc-custom-content-pack-v2.json | ||
fluentd_config: | ||
name: ${STACK_NAME}_fluentd_config_{{ "./fluentd/fluent.conf" | sha256file | substring(0,10) }} | ||
file: ./fluentd/fluent.conf | ||
loki_config: | ||
name: ${STACK_NAME}_loki_config_{{ "./loki.yaml" | sha256file | substring(0,10) }} | ||
file: ./loki.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
FROM fluent/fluentd:v1.16.9-1.0 | ||
|
||
USER root | ||
|
||
# Install dependencies and plugins | ||
RUN apk add --no-cache --update --virtual .build-deps \ | ||
sudo build-base ruby-dev curl \ | ||
&& sudo gem install fluent-plugin-grafana-loki \ | ||
&& sudo gem install fluent-plugin-gelf-best \ | ||
&& sudo gem install fluent-plugin-prometheus \ | ||
&& apk del .build-deps \ | ||
&& apk add --no-cache curl jq \ | ||
&& rm -rf /var/cache/apk/* \ | ||
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem | ||
|
||
# Create directories with appropriate permissions | ||
RUN mkdir -p /fluentd/buffer /fluentd/log \ | ||
&& chown -R fluent:fluent /fluentd/buffer /fluentd/log | ||
|
||
# Health check | ||
HEALTHCHECK --interval=30s --timeout=30s --retries=3 \ | ||
CMD curl -s http://localhost:24220/api/plugins | jq -e '.plugins | length > 0' || exit 1 | ||
|
||
USER fluent | ||
|
||
ENTRYPOINT ["fluentd", "-c", "/fluentd/etc/fluent.conf"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not necessary when the docker swarm base network is setup as I believe we do now.