ejs files not recompiled on change

[rowild]

I have this setup:

module: {
    rules: [{
        test: /\.ejs$/,
        exclude: /(node_modules)/,
        use: [{
            loader: 'compile-ejs-loader',
            options: {...}
          }
        }]
      },
    },
    plugins: [
      new HtmlWebpackPlugin({
        template: '!!compile-ejs-loader!./src/index.ejs',
        filename: 'index.html',
        title: 'something',
        minify: {... },
      }),
   ]
}

Whenever I edit something in a ejs file, webpack 4 is not re-compiling the changes.

Why is this happening? Anything that I am doing wrong here?
Advice much appreciated!