Using scrutiny with Docker Swarm

[DomiStyle]

Any idea if it's possible to use scrutiny with Docker Swarm?

Unfortunately in Docker Swarm mode devices does not work and device_cgroup_rules does not exist anymore in the docker-compose format v3.9.

I tried passing through the entirety of /dev as volume, which almost works since smartctl can recognize the drives but it obviously fails to access them due to permissions:

time="2022-11-21T10:22:55Z" level=info msg="Executing command: smartctl --scan --json" type=metrics
time="2022-11-21T10:22:55Z" level=info msg="Executing command: smartctl --info --json /dev/sds" type=metrics
time="2022-11-21T10:22:55Z" level=error msg="Could not retrieve device information for sds: exit status 2" type=metrics

When running the command manually:

     "messages": [
      {
        "string": "Smartctl open device: /dev/sds failed: Operation not permitted",
        "severity": "error"
      }
    ],

I ran the smartctl command on the host just to make sure and no issues there.

[DomiStyle]

Got it working by deploying the collector manually outside of the container. Not ideal but seems to work for now.

[AnalogJ]

Hey @DomiStyle

I don't use Swarm personally, so this may be difficult to debug.
However, have you tried mounting /dev and passing in the additional capabilities that smartctl requires? (cap-add in docker cli)

[DomiStyle]

However, have you tried mounting /dev and passing in the additional capabilities that smartctl requires? (cap-add in docker cli)

Yes, I used the following compose to try it:

services:
  scrutiny:
    image: ghcr.io/analogj/scrutiny:master-omnibus
    cap_add: ['SYS_RAWIO']
    volumes:
      - config:/opt/scrutiny/config
      - influxdb:/opt/scrutiny/influxdb
      - /hostudev:/run/udev:ro
      - /dev:/dev
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/api/health"]
      interval: 5s
      timeout: 10s
      retries: 20
      start_period: 10s

smartctl can see the drives like this but can't access them. I assume the devices key manages permissions to access the disk as well and volumes does not.

I also tried making the container privileged but same result.

[AnalogJ]

Scrutiny should definitely be able to run with privileged, that's basically as-if the process was running on the host.
You mentioned that calling smartctl on the host worked successfully. Have you tried calling smartctl within the container as well (rather than have Scrutiny do it)?

[DomiStyle]

Have you tried calling smartctl within the container as well (rather than have Scrutiny do it)?

Yep, I believe privileged containers are also not supported in Docker Swarm right now. It comes up with an empty /dev instead of having all the drives there.

[AnalogJ]

Ah, that would definitely cause an issue.
Seems like you've tried --device, -v /dev and --privileged, but maybe try --privileged with -v /dev, just to force docker swarm to mount the /dev folder?

IIRC -v mounts still work with privileged.

[DomiStyle]

but maybe try --privileged with -v /dev, just to force docker swarm to mount the /dev folder?

Yes, that works but causes the same error message as in the initial issue.

I assume privileged does nothing in Docker Swarm mode and thus the container has no permissions to access the block devices.

Since cgroups also can't be set in Docker Swarm mode I don't think there's a way around this issue until either --device or --privileged is implemented.

[AnalogJ]

😞 yeah that's what it sounds like unfortunately. Any chance there's an open docker-swarm issue that I can follow about this lack of support for device and privileged?

[DomiStyle]

There's an issue that's open since 2016 for privileged support here: moby/moby#24862
Also a recent PR for it here: moby/swarmkit#3072

There's also this issue about device support from 2018 here: moby/swarmkit#2682

So it might be a while.

[Enissay]

@DomiStyle there's a workaround to launch a privileged docker from within another one (a wrapper)... It works finally after many many tests...

For some reason I did not manage to make it work with the multi-line version for a better readability (commented part: not idea why, so I gave up)...

version: '3.8'
services:
  collector_wrapper:
    image: mavenugo/swarm-exec:17.03.0-ce
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/etc/localtime:/etc/localtime:ro"
      - "/etc/timezone:/etc/timezone:ro"
      - "/run/udev:/run/udev:ro"
      - "/dev/disk:/dev/disk"
    networks:
      - common-net
    deploy: 
      mode: replicated
    ## default run daily at midnight
    command: /bin/sh -c "docker rm -f collector_privileged &> /dev/null && docker run --restart always --privileged=true -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro -v /run/udev:/run/udev:ro -v /dev/disk:/dev/disk --cap-add SYS_RAWIO -e COLLECTOR_API_ENDPOINT=http://web:8080 -e COLLECTOR_CRON_SCHEDULE=\"0 0 * * *\" --name collector_privileged ghcr.io/analogj/scrutiny:master-collector"
#      - /bin/sh
#      - -c
#      - 
#      - > 
#         docker rm -f collector_privileged &> /dev/null
#         && docker run 
#            --restart always
#            --privileged=true
#            -v /etc/localtime:/etc/localtime:ro 
#            -v /etc/timezone:/etc/timezone:ro
#            -v /run/udev:/run/udev:ro 
#            -v /dev/disk:/dev/disk
#            --cap-add SYS_RAWIO 
#            -e COLLECTOR_API_ENDPOINT=http://web:8080
#            -e COLLECTOR_CRON_SCHEDULE=\"0 0 * * *\"
#            --name collector_privileged
#            ghcr.io/analogj/scrutiny:master-collector

BTW this is for the Hub deployment (not bundled omnibus). You can see the complete sample compose file here: https://github.com/AnalogJ/scrutiny/blob/master/docker/example.hubspoke.docker-compose.yml. Loose the depends_on as it is not needed (not applied for swarm).

It is sad that the collector does not make a first run at startup then waits for the next schedule... Maybe something to add @AnalogJ ?

[AnalogJ]

@Enissay running the collector on first startup for the collector tagged image is a great idea. Can you open a new issue for that?