diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index f86566a94..0b785c359 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -15,6 +15,9 @@ matrix_prometheus_node_exporter_enabled: true # You can remove this, if unnecessary. matrix_prometheus_postgres_exporter_enabled: true +# You can remove this, if unnecessary. +matrix_prometheus_nginxlog_exporter_enabled: true + matrix_grafana_enabled: true matrix_grafana_anonymous_access: false @@ -39,6 +42,7 @@ Name | Description `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. `matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures `matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. +`matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in `matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option. `matrix_grafana_default_admin_user`
`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here @@ -73,6 +77,7 @@ Name | Description `matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) `matrix_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) +`matrix_prometheus_nginxlog_exporter_enabled`|Set this to `true` to enable the [NGINX Log exporter](configuring-playbook-prometheus-nginxlog.md) (locally, on the container network) `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_bridge_hookshot_metrics_enabled`|Set this to `true` to make [Hookshot](configuring-playbook-bridge-hookshot.md) expose metrics (locally, on the container network) `matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) diff --git a/docs/configuring-playbook-prometheus-nginxlog.md b/docs/configuring-playbook-prometheus-nginxlog.md new file mode 100644 index 000000000..832efbd8d --- /dev/null +++ b/docs/configuring-playbook-prometheus-nginxlog.md @@ -0,0 +1,59 @@ +# Enabling metrics and graphs for NginX logs (optional) + +It can be useful to have some (visual) insight into NignX logs. + +This adds [prometheus-nginxlog-exporter](https://github.com/martin-helmich/prometheus-nginxlog-exporter/) to your matrix deployment. +It will provide a prometheus 'metrics' endpoint exposing data from both the `matrix-nginx-proxy` and `matrix-synapse-reverse-proxy-companion` logs and automatically aggregates the data with prometheus. +Optionally it visualizes the data, if [`matrix-grafana`](configuring-playbook-prometheus-grafana.md) is enabled, by means of a dedicated Grafana dashboard named `NGINX PROXY` + +You can enable this role by adding the following settings in your configuration file (`inventory/host_vars/matrix./vars.yml`): + +```yaml +matrix_prometheus_nginxlog_exporter_enabled: true + +# required depency +matrix_prometheus_enabled: true + +# optional for visualization +matrix_grafana_enabled: true +``` + +x | Prerequisites | Variable | Description +|:--:|:--:|:--:|:--| +**REQUIRED** | `matrix-prometheus`| `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. +_Optional_ | [`matrix-grafana`](configuring-playbook-prometheus-grafana.md) | [`matrix_grafana_enabled`](configuring-playbook-prometheus-grafana.md)|[Grafana](https://grafana.com) is the visual component. It shows (on the `stats.` subdomain) graphs that we're interested in. When enabled the `NGINX PROXY` dashboard is automatically added. + +## Docker Image Compatibility + +At the moment of writing only images for `amd64` and `arm64` architectures are available + +The playbook currently does not support building an image. +You can however use a custom-build image by setting +```yaml +matrix_prometheus_nginxlog_exporter_docker_image_arch_check_enabled: false +matrix_prometheus_nginxlog_exporter_docker_image: path/to/docker/image:tag +``` + +## Security and privacy + +Metrics and resulting graphs can contain a lot of information. NginX logs contain information like IP address, URLs, UserAgents and more. This information can reveal usage patterns and could be considered Personally Identifiable Information (PII). Think about this before enabling (anonymous) access. +Please make sure you change the default Grafana password. + +## Save metrics on an external Prometheus server + +The playbook will automatically integrate the metrics into the Prometheus server provided with this playbook. You can choose to save data on an external Prometheus instance. + +The metrics of this role will be exposed on `https://matrix.DOMAIN/metrics/nginxlog` when setting +```yaml +matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled: true + +# required dependency +matrix_nginx_proxy_proxy_matrix_metrics_enabled: true +``` +The playbook can provide a single endpoint (`https://matrix.DOMAIN/metrics/*`), under which various services may expose their metrics (e.g. `/metrics/node-exporter`, `/metrics/postgres-exporter`, `/metrics/nginxlog`, etc). To enable this `/metrics/*` feature, use `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. To protect access using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication), see `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`. + +The following variables may be of interest: + +Name | Description +-----|---------- +`matrix_nginx_proxy_proxy_matrix_metrics_enabled`|Set this to `true` to enable metrics exposure for various services on `https://matrix.DOMAIN/metrics/*`. Refer to the individual `matrix_SERVICE_metrics_proxying_enabled` variables below for exposing metrics for each individual service. \ No newline at end of file diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 71bd18812..e3f63bb33 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -297,6 +297,8 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus-postgres-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_postgres_exporter_enabled else []) + + ([{'name': (matrix_prometheus_nginxlog_exporter_container_hostname + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_postgres_exporter_enabled else []) + + ([{'name': 'matrix-redis', 'priority': 750, 'groups': ['matrix', 'redis']}] if matrix_redis_enabled else []) + ([{'name': 'matrix-registration.service', 'priority': 4000, 'groups': ['matrix', 'registration']}] if matrix_registration_enabled else []) @@ -2235,6 +2237,9 @@ matrix_ssl_architecture: "{{ matrix_ssl_pre_obtaining_required_service_name: "{{ 'matrix-dynamic-dns' if matrix_dynamic_dns_enabled else '' }}" +matrix_nginx_proxy_access_log_syslog_integration_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}" +matrix_nginx_proxy_access_log_syslog_integration_server_port: "{{ (matrix_prometheus_nginxlog_exporter_container_hostname | string +':'+ matrix_prometheus_nginxlog_exporter_container_syslog_port | string) | default('') }}" + ###################################################################### # # /matrix-nginx-proxy @@ -2820,6 +2825,9 @@ matrix_synapse_reverse_proxy_companion_synapse_stream_writer_presence_stream_wor matrix_synapse_reverse_proxy_companion_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}" matrix_synapse_reverse_proxy_companion_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_worker_client_server_endpoints|default([]) }}" +matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}" +matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_server_port: "{{ (matrix_prometheus_nginxlog_exporter_container_hostname | string +':'+ matrix_prometheus_nginxlog_exporter_container_syslog_port | string) | default('') }}" + ###################################################################### # # /matrix-synapse-reverse-proxy-companion @@ -2861,6 +2869,20 @@ matrix_prometheus_node_exporter_enabled: false # ###################################################################### +###################################################################### +# +# matrix-prometheus-nginxlog-exporter +# +###################################################################### + +matrix_prometheus_nginxlog_exporter_enabled: false + +###################################################################### +# +# /matrix-prometheus-nginxlog-exporter +# +###################################################################### + ###################################################################### # # matrix-prometheus @@ -2888,6 +2910,10 @@ matrix_prometheus_scraper_postgres_targets: "{{ ['matrix-prometheus-postgres-exp matrix_prometheus_scraper_hookshot_enabled: "{{ matrix_hookshot_metrics_enabled|default(false) }}" matrix_prometheus_scraper_hookshot_targets: "{{ [matrix_hookshot_container_url | string +':'+ matrix_hookshot_metrics_port | string] if matrix_hookshot_metrics_enabled else [] }}" +matrix_prometheus_scraper_nginxlog_enabled: "{{ matrix_prometheus_nginxlog_exporter_enabled }}" +matrix_prometheus_scraper_nginxlog_server_port: "{{ (matrix_prometheus_nginxlog_exporter_container_hostname | string +':'+ matrix_prometheus_nginxlog_exporter_container_syslog_port | string) +| default('') }}" + ###################################################################### # # /matrix-prometheus @@ -2937,6 +2963,8 @@ matrix_grafana_dashboard_download_urls: | (matrix_prometheus_node_exporter_dashboard_urls if matrix_prometheus_node_exporter_enabled else []) + (matrix_prometheus_postgres_exporter_dashboard_urls if matrix_prometheus_postgres_exporter_enabled else []) + + + (matrix_prometheus_nginxlog_exporter_dashboard_urls if matrix_prometheus_nginxlog_exporter_enabled else []) }} matrix_grafana_default_home_dashboard_path: |- diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 5b6f1ccea..92e027fb5 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -80,6 +80,7 @@ - custom/matrix-synapse-admin - custom/matrix-prometheus-node-exporter - custom/matrix-prometheus-postgres-exporter + - custom/matrix-prometheus-nginxlog-exporter - custom/matrix-prometheus - custom/matrix-grafana - custom/matrix-registration diff --git a/roles/custom/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 b/roles/custom/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 index 5aeeb6f40..6b56878a1 100644 --- a/roles/custom/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 +++ b/roles/custom/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 @@ -39,7 +39,18 @@ http { {% if matrix_nginx_proxy_access_log_enabled %} access_log /var/log/nginx/access.log main; - {% else %} + {% endif %} + + {% if matrix_nginx_proxy_access_log_syslog_integration_enabled %} + log_format prometheus_fmt 'matrix-nginx-proxy $server_name - $upstream_addr - $remote_addr - $remote_user [$time_local] ' + '$host "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log syslog:server={{ matrix_nginx_proxy_access_log_syslog_integration_server_port }},tag=matrix_nginx_proxy prometheus_fmt; + {% endif %} + + {% if not matrix_nginx_proxy_access_log_enabled and not matrix_nginx_proxy_access_log_syslog_integration_enabled %} access_log off; {% endif %} diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml new file mode 100644 index 000000000..2e85d6298 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/defaults/main.yml @@ -0,0 +1,49 @@ +--- +# prometheus-nginxlog-exporter exports nginx logs in a prometheus usable format on a `/metrics/ endpoint +# See: https://github.com/martin-helmich/prometheus-nginxlog-exporter/ + +matrix_prometheus_nginxlog_exporter_enabled: true +matrix_prometheus_nginxlog_exporter_version: v1.10.0 + +matrix_prometheus_nginxlog_exporter_container_hostname: 'matrix-prometheus-nginxlog-exporter' +matrix_prometheus_nginxlog_exporter_container_metrics_port: '4040' +matrix_prometheus_nginxlog_exporter_container_syslog_port: '6514' + +matrix_prometheus_nginxlog_exporter_docker_image_arch: ['amd64', 'arm64'] +matrix_prometheus_nginxlog_exporter_docker_image_name_prefix: "{{ 'ghcr.io/martin-helmich/' }}" +matrix_prometheus_nginxlog_exporter_docker_image_tag: "{{ 'latest' if matrix_prometheus_nginxlog_exporter_version == 'master' else matrix_prometheus_nginxlog_exporter_version }}-{{ matrix_architecture }}" +matrix_prometheus_nginxlog_exporter_docker_image: "{{ matrix_prometheus_nginxlog_exporter_docker_image_name_prefix }}prometheus-nginxlog-exporter/exporter:{{ matrix_prometheus_nginxlog_exporter_docker_image_tag }}" +matrix_prometheus_nginxlog_exporter_docker_image_force_pull: "{{ matrix_prometheus_nginxlog_exporter_docker_image.endswith(':latest') }}" +matrix_prometheus_nginxlog_exporter_docker_image_arch_check_enabled: true + +# Controls whether prometheus-nginxlog-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/nginxlog`. +# This will only take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled: false + +# matrix_prometheus_nginxlog_exporter_dashboard_urls contains a list of URLs with Grafana dashboard definitions. +# If the Grafana role is enabled, these dashboards will be downloaded. +matrix_prometheus_nginxlog_exporter_dashboard_urls: [] + +matrix_prometheus_nginxlog_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-nginxlog-exporter" +matrix_prometheus_nginxlog_exporter_config_path: "{{ matrix_prometheus_nginxlog_exporter_base_path }}/config" + +# A list of extra docker arguments to pass to the container +matrix_prometheus_nginxlog_exporter_container_extra_arguments: [] + +# List of systemd services that matrix-prometheus_nginxlog_exporter.service depends on +matrix_prometheus_nginxlog_exporter_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-prometheus_nginxlog_exporter.service wants +matrix_prometheus_nginxlog_exporter_systemd_wanted_services_list: [] + +# Controls whether the matrix-prometheus_nginxlog_exporter container exposes its HTTP port (tcp/4040 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:4040"), or empty string to not expose. +matrix_prometheus_nginxlog_exporter_container_http_host_bind_port: '' + +# Controls whether the matrix-prometheus_nginxlog_exporter container exposes its SYSLOG port (udp/6514 in the container). +# +# Takes an ":" or "" value (e.g. "127.0.0.1:6514"), or empty string to not expose. +# NOTE: is an UDP connection! +matrix_prometheus_nginxlog_exporter_container_syslog_host_bind_port: '' diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/examples/grafana.png b/roles/custom/matrix-prometheus-nginxlog-exporter/examples/grafana.png new file mode 100644 index 000000000..49fa02113 Binary files /dev/null and b/roles/custom/matrix-prometheus-nginxlog-exporter/examples/grafana.png differ diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/examples/metrics.png b/roles/custom/matrix-prometheus-nginxlog-exporter/examples/metrics.png new file mode 100644 index 000000000..b81265268 Binary files /dev/null and b/roles/custom/matrix-prometheus-nginxlog-exporter/examples/metrics.png differ diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/inject_into_nginx_proxy.yml new file mode 100644 index 000000000..15317de09 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/inject_into_nginx_proxy.yml @@ -0,0 +1,31 @@ +--- + +- name: Fail if matrix-nginx-proxy role already executed + ansible.builtin.fail: + msg: >- + Trying to append prometheus-nginxlog-exporter's configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role is already executed. + To fix this, please change the order of roles in your playbook to make sure that + the matrix-nginx-proxy role is run after the prometheus-nginxlog-exporter role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- name: Generate prometheus-nginxlog-exporter's metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/nginxlog) + ansible.builtin.set_fact: + matrix_prometheus_nginxlog_exporter_matrix_nginx_proxy_metrics_configuration_matrix_domain: | + location /metrics/nginxlog { + {# Use the embedded DNS resolver in Docker containers to discover the service #} + resolver 127.0.0.11 valid=5s; + set $backend "{{ matrix_prometheus_nginxlog_exporter_container_hostname }}:{{ matrix_prometheus_nginxlog_exporter_container_metrics_port }}"; + proxy_pass http://$backend/metrics; + } + when: matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled | bool and matrix_nginx_proxy_enabled | bool + +- name: Register prometheus-nginxlog-exporter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/nginxlog) + ansible.builtin.set_fact: + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: | + {{ + matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + + + [matrix_prometheus_nginxlog_exporter_matrix_nginx_proxy_metrics_configuration_matrix_domain] + }} + when: matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled | bool diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml new file mode 100644 index 000000000..b205dec36 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/main.yml @@ -0,0 +1,29 @@ +--- + +- block: + - when: matrix_prometheus_nginxlog_exporter_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-prometheus-nginxlog-exporter + - install-all + - install-prometheus-nginxlog-exporter + +- block: + - when: matrix_prometheus_nginxlog_exporter_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/validate_config.yml" + + - when: matrix_prometheus_nginxlog_exporter_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" + tags: + - setup-all + - setup-prometheus-nginxlog-exporter + - install-all + - install-prometheus-nginxlog-exporter + +- block: + - when: not matrix_prometheus_nginxlog_exporter_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + tags: + - setup-all + - setup-prometheus-nginxlog-exporter diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml new file mode 100644 index 000000000..9b2ac3e60 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_install.yml @@ -0,0 +1,47 @@ +--- + +- name: Ensure matrix-prometheus-nginxlog-exporter image is pulled + community.docker.docker_image: + name: "{{ matrix_prometheus_nginxlog_exporter_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_prometheus_nginxlog_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_nginxlog_exporter_docker_image_force_pull }}" + register: result + retries: "{{ devture_playbook_help_container_retries_count }}" + delay: "{{ devture_playbook_help_container_retries_delay }}" + until: result is not failed + +- name: Ensure prometheus-nginxlog-exporter paths exists + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - "{{ matrix_prometheus_nginxlog_exporter_base_path }}" + - "{{ matrix_prometheus_nginxlog_exporter_config_path }}" + +- name: Ensure prometheus-nginxlog-exporter.yaml present + ansible.builtin.template: + src: "{{ role_path }}/templates/prometheus-nginxlog-exporter.yaml.j2" + dest: "{{ matrix_prometheus_nginxlog_exporter_config_path }}/prometheus-nginxlog-exporter.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure grafana dashboard is installed + ansible.builtin.template: + src: "{{ role_path }}/templates/nginx-proxy.json" + dest: "{{ matrix_grafana_config_path }}/dashboards/" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + when: matrix_grafana_enabled + +- name: Ensure matrix-prometheus-nginxlog-exporter.service installed + ansible.builtin.template: + src: "{{ role_path }}/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2" + dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-nginxlog-exporter.service" + mode: 0644 + register: matrix_prometheus_nginxlog_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml new file mode 100644 index 000000000..c0d0a8338 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/setup_uninstall.yml @@ -0,0 +1,39 @@ +--- + +- name: Check existence of matrix-prometheus-nginxlog-exporter service + ansible.builtin.stat: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-nginxlog-exporter.service" + register: matrix_prometheus_nginxlog_exporter_service_stat + +- when: matrix_prometheus_nginxlog_exporter_service_stat.stat.exists | bool + block: + - name: Fail when not cleaning up nginx and prometheus configs + ansible.builtin.fail: + msg: > + This role has added to configs in 'matrix-nginx-proxy', 'matrix-synapse-reverse-proxy-companion' and 'matrix-prometheus'. + Running 'setup-synapse-reverse-proxy-companion' WILL NOT remove those settings from those roles. + Run the playbook again with the `setup-all` tag or all three 'setup-nginx-proxy,setup-synapse-reverse-proxy-companion,setup-prometheus' tags while + 'matrix_prometheus_enabled: false' to rebuild their configs. + when: not ('setup-all' in ansible_run_tags or ('setup-nginx-proxy' in ansible_run_tags and 'setup-synapse-reverse-proxy-companion' in ansible_run_tags and 'setup-prometheus' in ansible_run_tags)) + + - name: Ensure matrix-prometheus-nginxlog-exporter is stopped + ansible.builtin.service: + name: matrix-prometheus-nginxlog-exporter + state: stopped + enabled: false + daemon_reload: true + + - name: Ensure matrix-prometheus-nginxlog-exporter.service doesn't exist + ansible.builtin.file: + path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-nginxlog-exporter.service" + state: absent + + - name: Ensure matrix-prometheus-nginxlog-exporter paths don't exist + ansible.builtin.file: + path: "{{ matrix_prometheus_nginxlog_exporter_base_path }}" + state: absent + + - name: Ensure matrix-prometheus-nginxlog-exporter grafana dashboard doesn't exist + ansible.builtin.file: + path: "{{ matrix_grafana_config_path }}/dashboards/nginx-proxy.json" + state: absent diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml new file mode 100644 index 000000000..a8eccaeed --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/tasks/validate_config.yml @@ -0,0 +1,32 @@ +--- + +- name: Fail if Prometheus not enabled + ansible.builtin.fail: + msg: > + You need to set `matrix_prometheus_enabled: true` or configure an external Prometheus database + as this roles requires Prometheus for data storage. + when: not matrix_prometheus_enabled and not matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled + +- name: Fail if docker image not availble for arch + ansible.builtin.fail: + msg: > + 'prometheus-nginxlog-exporter' docker image is not available for your arch '{{ matrix_architecture }}'. + + We currently do not support building an image using this playbook. + + You can use a custom-build image by setting + 'matrix_prometheus_nginxlog_exporter_docker_image_arch_check_enabled: false' + 'matrix_prometheus_nginxlog_exporter_docker_image: path/to/docker/image:tag' + + or disable 'prometheus-nginxlog-exporter' by setting + 'matrix_prometheus_nginxlog_exporter: false' + + in vars.yml + when: matrix_prometheus_nginxlog_exporter_docker_image_arch_check_enabled and matrix_architecture not in matrix_prometheus_nginxlog_exporter_docker_image_arch + + +- name: Fail if nginx-proxy is not set to proxy metrics while prometheus-nginxlog-exporter is + ansible.builtin.fail: + msg: > + 'matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled' is set but 'matrix_nginx_proxy_proxy_matrix_metrics_enabled' is not + when: matrix_prometheus_nginxlog_exporter_metrics_proxying_enabled | bool and not matrix_nginx_proxy_proxy_matrix_metrics_enabled | bool diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/nginx-proxy.json b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/nginx-proxy.json new file mode 100644 index 000000000..415334885 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/nginx-proxy.json @@ -0,0 +1,1705 @@ +{ + "__inputs": [ + { + "name": "DS_PROMETHEUS", + "label": "Prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "9.3.1" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "timeseries", + "name": "Time series", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "target": { + "limit": 100, + "matchAny": false, + "tags": [], + "type": "dashboard" + }, + "type": "dashboard" + } + ] + }, + "description": "", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "liveNow": false, + "panels": [ + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 23, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 17 + }, + "id": 20, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"1.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "1xx by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 17 + }, + "id": 21, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"1.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "1xx by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 25 + }, + "id": 18, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"2.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "2xx by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 25 + }, + "id": 19, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"2.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "2xx by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 33 + }, + "id": 16, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"3.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "3xx by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 33 + }, + "id": 17, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"3.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "3xx by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "1xx - 3xx - 2xx Status Code .....................................................", + "type": "row" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 1 + }, + "id": 2, + "panels": [], + "title": "4xx Status Code ......................................................................", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 2 + }, + "id": 9, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"404\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "404 by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 2 + }, + "id": 6, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"404\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "404 by uri [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 10 + }, + "id": 14, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"4.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "4xx by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 10 + }, + "id": 15, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"4.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "4xx by uri [$request_host]", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 18 + }, + "id": 8, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 11 + }, + "id": 10, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr, status) (rate(http_response_count_total{status=~\"504|502\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "502-504 by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 11 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_host, status) (rate(http_response_count_total{status=~\"504|502\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "502-504 by host [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 19 + }, + "id": 12, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"5.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "5xx by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 19 + }, + "id": 13, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_host) (rate(http_response_count_total{status=~\"5.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "5xx by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "5xx Status Code ......................................................................", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 19 + }, + "id": 27, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 4 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "expr": "sum by(remote_addr) (rate(http_response_count_total{status=~\"[6-9]{1}.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "hide": false, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "6xx > by remote addr [$request_host]", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 4 + }, + "id": 25, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "builder", + "exemplar": false, + "expr": "sum by(request_uri) (rate(http_response_count_total{status=~\"[6-9]{1}.*\", request_host=~\"$request_host\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": true, + "refId": "B" + } + ], + "title": "6xx > by uri [$request_host]", + "type": "timeseries" + } + ], + "title": "6xx > Status Code ...................................................................", + "type": "row" + } + ], + "refresh": "30s", + "schemaVersion": 37, + "style": "dark", + "tags": [], + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": "Prometheus", + "value": "Prometheus" + }, + "hide": 0, + "includeAll": false, + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "queryValue": "", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(http_response_count_total, request_host)", + "hide": 0, + "includeAll": true, + "multi": true, + "name": "request_host", + "options": [], + "query": { + "query": "label_values(http_response_count_total, request_host)", + "refId": "StandardVariableQuery" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "type": "query" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "NGINX PROXY", + "uid": "x2_jWNF4k", + "version": 11, + "weekStart": "" +} \ No newline at end of file diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/prometheus-nginxlog-exporter.yaml.j2 b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/prometheus-nginxlog-exporter.yaml.j2 new file mode 100644 index 000000000..1e2492cde --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/prometheus-nginxlog-exporter.yaml.j2 @@ -0,0 +1,37 @@ +listen: + port: {{ matrix_prometheus_nginxlog_exporter_container_metrics_port }} + address: "0.0.0.0" + metrics_endpoint: "/metrics" + +namespaces: + - name: matrix + metrics_override: + preffix: "myprefix" + namespace_label: "namespace" + format: "$log_source $server_name - $upstream_addr - $remote_addr - $remote_user [$time_local] $host \"$request\" $status \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\"" + # enable to print to console + # print_log: true + source: + syslog: + listen_address: "udp://0.0.0.0:{{ matrix_prometheus_nginxlog_exporter_container_syslog_port }}" + # format options: "rfc3164" ,"rfc5424", "rfc6587", "auto" + format: auto + tags: ["matrix_nginx_proxy", "matrix_nginx_proxy_companion"] + relabel_configs: + - target_label: request_uri + from: request + split: 2 + separator: ' ' + matches: + - regexp: "^([^\\?]+)(.*)" + replacement: "$1" + - target_label: remote_addr + from: remote_addr + - target_label: server_name + from: server_name + - target_label: http_x_forwarded_for + from: http_x_forwarded_for + - target_label: request_host + from: host + - target_label: log_source + from: log_source \ No newline at end of file diff --git a/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 new file mode 100644 index 000000000..e799380e3 --- /dev/null +++ b/roles/custom/matrix-prometheus-nginxlog-exporter/templates/systemd/matrix-prometheus-nginxlog-exporter.service.j2 @@ -0,0 +1,46 @@ +#jinja2: lstrip_blocks: "True" +[Unit] +Description={{ matrix_prometheus_nginxlog_exporter_container_hostname }} +{% for service in matrix_prometheus_nginxlog_exporter_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_prometheus_nginxlog_exporter_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no + +[Service] +Type=simple +Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill {{ matrix_prometheus_nginxlog_exporter_container_hostname }} 2>/dev/null || true' +ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_prometheus_nginxlog_exporter_container_hostname }} 2>/dev/null || true' + + +ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name {{ matrix_prometheus_nginxlog_exporter_container_hostname }} \ + --log-driver=none \ + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ + --cap-drop=ALL \ + --read-only \ + --network={{ matrix_docker_network }} \ + {% if matrix_prometheus_nginxlog_exporter_container_http_host_bind_port %} + -p {{ matrix_prometheus_nginxlog_exporter_container_http_host_bind_port }}:{{ matrix_prometheus_nginxlog_exporter_container_metrics_port }} \ + {% endif %} + {% if matrix_prometheus_nginxlog_exporter_container_syslog_host_bind_port %} + -p {{ matrix_prometheus_nginxlog_exporter_container_syslog_host_bind_port }}:{{ matrix_prometheus_nginxlog_exporter_container_syslog_port }}/udp \ + {% endif %} + -v {{ matrix_prometheus_nginxlog_exporter_config_path }}:/etc/prometheus-nginxlog-exporter:z \ + {% for arg in matrix_prometheus_nginxlog_exporter_container_extra_arguments %} + {{ arg }} \ + {% endfor %} + {{ matrix_prometheus_nginxlog_exporter_docker_image }} \ + -config-file /etc/prometheus-nginxlog-exporter/prometheus-nginxlog-exporter.yaml + +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill {{ matrix_prometheus_nginxlog_exporter_container_hostname }} 2>/dev/null || true' +ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm {{ matrix_prometheus_nginxlog_exporter_container_hostname }} 2>/dev/null || true' +Restart=always +RestartSec=30 +SyslogIdentifier={{ matrix_prometheus_nginxlog_exporter_container_hostname }} + +[Install] +WantedBy=multi-user.target diff --git a/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 b/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 index 83ae8a9a1..6ed7cd922 100644 --- a/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 +++ b/roles/custom/matrix-prometheus/templates/prometheus.yml.j2 @@ -64,3 +64,10 @@ scrape_configs: static_configs: - targets: {{ matrix_prometheus_scraper_hookshot_targets|to_json }} {% endif %} + + {% if matrix_prometheus_scraper_nginxlog_enabled %} + - job_name: nginxlog + static_configs: + - targets: + - {{ matrix_prometheus_scraper_nginxlog_server_port}} + {% endif %} diff --git a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/nginx/nginx.conf.j2 b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/nginx/nginx.conf.j2 index a54701b82..309bb5301 100644 --- a/roles/custom/matrix-synapse-reverse-proxy-companion/templates/nginx/nginx.conf.j2 +++ b/roles/custom/matrix-synapse-reverse-proxy-companion/templates/nginx/nginx.conf.j2 @@ -39,7 +39,18 @@ http { {% if matrix_synapse_reverse_proxy_companion_access_log_enabled %} access_log /var/log/nginx/access.log main; - {% else %} + {% endif %} + + {% if matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_enabled %} + log_format prometheus_fmt 'matrix-synapse-reverse-proxy-companion $server_name - $upstream_addr - $remote_addr - $remote_user [$time_local] ' + '$host "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log syslog:server={{ matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_server_port }},tag=matrix_nginx_proxy_companion prometheus_fmt; + {% endif %} + + {% if not matrix_synapse_reverse_proxy_companion_access_log_enabled and not matrix_synapse_reverse_proxy_companion_access_log_syslog_integration_enabled %} access_log off; {% endif %}