コンテンツにスキップ

Sensu

{{toc}}

Go言語で書き直されたSensu。

パッケージは以下の3つ

  • sensu-go-agent
  • sensu-go-backend
  • sensu-go-cli

インストール

https://docs.sensu.io/sensu-go/5.5/installation/install-sensu/

Backend

curl -s https://packagecloud.io/install/repositories/sensu/stable/script.deb.sh | sudo bash
sudo apt update
sudo apt-get install sensu-go-backend
sudo cp /usr/share/doc/sensu-go-backend-5.5.1/backend.yml.example /etc/sensu/backend.yml
sudo systemctl start sensu-backend
sudo systemctl enable sensu-backend
sudo systemctl status sensu-backend

CLI

sudo apt-get install sensu-go-cli
sensuctl configure
? Sensu Backend URL: http://127.0.0.1:8080
? Username: admin
? Password: *********
? Namespace: default
? Preferred output format: tabular
export TOKEN=$(cat ~/.config/sensu/sensuctl/cluster | jq -r .access_token)
curl -s http://127.0.0.1:8080/api/core/v2/namespaces/default/events -H "Authorization: Bearer ${TOKEN}" | jq

Non-interactive: https://docs.sensu.io/sensu-go/5.0/sensuctl/reference/#non-interactive

sensuctl configure -n --url http://127.0.0.1:8080 --username admin --password P@ssw0rd! --format tabular

with namespace

sensuctl configure -n --url http://127.0.0.1:8080 --username admin --password P@ssw0rd! --format tabular --namespace user01

Agent

sudo apt-get install sensu-go-agent
sudo systemctl start sensu-agent
sudo systemctl enable sensu-agent
sudo systemctl status sensu-agent

リモートホストの場合はテンプレートを持ってきて、127.0.0.1をbackendに向ける。

sudo cp /usr/share/doc/sensu-go-agent-5.5.1/agent.yml.example /etc/sensu/agent.yml

Plugin installer

https://docs.sensu.io/sensu-go/5.5/installation/plugins/

curl -s https://packagecloud.io/install/repositories/sensu/community/script.deb.sh | sudo bash
sudo apt install -y sensu-plugins-ruby

Plugins

https://github.com/sensu-plugins からダウンロード対象を選ぶ。

例えば sensu-plugins-http であれば

sudo sensu-install -p http

プラグインによっては拡張のビルドに失敗したりする。httpの場合は以下も必要かもしれない。

sudo apt install -y build-essential

主要なプラグイン

sudo sensu-install -p http
sudo sensu-install -p sensu-plugins-network-checks
sudo sensu-install -p dns

パスの追加

※これじゃダメそう

$ cat /etc/profile.d/05-sensu-plugins.sh 
export PATH=$PATH:/opt/sensu-plugins-ruby/embedded/bin

Usage

HTTP監視の追加

プラグイン挙動の確認

$ /opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u https://www.ainoniwa.net
example.curl_timings.time_total 0.013437 1556126157
example.curl_timings.time_namelookup 0.004224 1556126157
example.curl_timings.time_connect 0.004775 1556126157
example.curl_timings.time_pretransfer 0.012694 1556126157
example.curl_timings.time_redirect 0.000000 1556126157
example.curl_timings.time_starttransfer 0.013400 1556126157
example.curl_timings.http_code 200 1556126157

ちなみにこれは curl コマンドの結果を整形したに過ぎない

sh -c LC_NUMERIC=C curl --silent --output /dev/null  -w "%{time_total},%{time_namelookup},%{time_connect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{http_code}" http://localhost

追加

sensuctl create --file curl_timings-check.json

curl_timings-check.json

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "curl_timings",
    "namespace": "default"
  },
  "spec": {
    "check_hooks": null,
    "command": "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://www.ainoniwa.net",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "output_metric_format": "graphite_plaintext",
    "output_metric_handlers": [],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [],
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "entity:example"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

subscriptions の部分で実行先のentityを決めているらしい。
https://docs.sensu.io/sensu-go/5.5/reference/checks/#subscriptions

後から追加する場合は以下のように書ける。

sensuctl check set-subscriptions curl_timings '["entity:example", "entity:example-dev2"]'

Slack通知の追加

$ cat sensu-slack-handler.json
{
  "type": "Handler",
  "spec": {
    "metadata": {
      "name": "slack",
      "namespace": "default",
      "labels": null,
      "annotations": null
    },
    "type": "pipe",
    "command": "sensu-slack-handler -w ${SLACK_WEBHOOK_URL} -c ${SLACK_CHANNEL}",
    "timeout": 0,
    "handlers": [],
    "filters": [""],
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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",
      "SLACK_CHANNEL=@yamabukir"
    ],
    "runtime_assets": ["sensu-slack-handler"]
  }
}
$ sensuctl create --file sensu-slack-handler.json
$ sensuctl entity list
    ID     Class    OS     Subscriptions             Last Seen            
 ──────── ─────── ─────── ─────────────── ─────────────────────────────── 
  ubuntu   agent   linux   entity:ubuntu   2019-04-24 18:35:16 +0900 JST  
$ sensuctl event list
  Entity     Check                                  Output                                Status   Silenced             Timestamp            
 ──────── ─────────── ────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── 
  ubuntu   keepalive   Keepalive last sent from ubuntu at 2019-04-24 09:35:36 +0000 UTC        0   false      2019-04-24 18:35:36 +0900 JST  
$ sensuctl handler list
  Name    Type   Timeout   Filters   Mutator                                  Execute                                                                                   Environment Variables                                                           Assets         
 ─────── ────── ───────── ───────── ───────── ─────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ───────────────────── 
  slack   pipe         0                       RUN:  sensu-slack-handler -w ${SLACK_WEBHOOK_URL} -c ${SLACK_CHANNEL}   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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,SLACK_CHANNEL=@yamabukir   sensu-slack-handler

Memo

どうやらbackendとserverの間で設定するhandlerは keepalive という予約語になるらしい。
https://github.com/sensu/sensu-go/issues/1960

Built-in filters に障害が発生した場合を指すFilter is_incident があるらしい。大体はこれをセットしておけばいい。(逆にFilterをセットしないと、ポーリング間隔毎にhandlerが動いてしまうので、slack通知が無限に飛んできたりする。
https://docs.sensu.io/sensu-go/5.5/reference/filters/#how-do-sensu-filters-work

sensuctl handler create keepalive \
--type pipe \
--env-vars "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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" \
--command "sensu-slack-handler --channel '@yamabukir'" \
--runtime-assets sensu-slack-handler

1つのAgentは1つのEntityで表現されるので、ユーザー毎に異なる名前空間に

for num in `seq -w 1 1000`; do
    sensuctl namespace create uid000${num};
    sensuctl check create curl_local \
    --command "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://192.168.122.201" \
    --interval 30 \
    --subscriptions "location:jp_tokyo" \
    --namespace "uid000${num}" \
    --output-metric-format "graphite_plaintext" \
    --output-metric-handlers "output_metric_handlers"
done

REST API Calling

$ curl -s -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization:Basic YWRtaW46UEBzc3cwcmQh" http://127.0.0.1:8080/auth | jq
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTcyMDM1MDIsImp0aSI6IjBkYWNhMDIzMjEwNTM2NjgxMmVmMTBlNjUwNzExYjczIiwic3ViIjoiYWRtaW4iLCJncm91cHMiOlsiY2x1c3Rlci1hZG1pbnMiLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInVzZXJfaWQiOiJhZG1pbiJ9fQ.6rynHV0DSHqUw4jU8DbzcEMeqtY8roIG-fr_WvE3Mms",
  "expires_at": 1557203502,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJkN2E5ZDZiNTM3MTZiYTQ1ZjA4NzJmOTNkYTBmZGFmMCIsInN1YiI6ImFkbWluIiwiZ3JvdXBzIjpudWxsLCJwcm92aWRlciI6eyJwcm92aWRlcl9pZCI6IiIsInVzZXJfaWQiOiIifX0.PukUNX1cL3Z9l_7iw6auaMzOUM3PuBI3RPfhTZLALFA"
}
$ curl http://127.0.0.1:8080/api/core/v2/entities -H "Authorization: Bearer eyJhbGciOi..."

コピペ用

TOKEN=`curl -s -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization:Basic YWRtaW46UEBzc3cwcmQh" http://127.0.0.1:8080/auth | jq -r .access_token`
curl http://127.0.0.1:8080/api/core/v2/checks -s -H "Authorization: Bearer ${TOKEN}"

Filtering (Need to enterprise license)

https://docs.sensu.io/sensu-go/5.6/api/overview/#filtering

$ curl http://127.0.0.1:8080/api/core/v2/entities -H "Authorization: Bearer ${TOKEN}" -G --data-urlencode 'labelSelector=entity_class == "proxy"'
{"message":"missing or invalid license","code":6}

Agent

基本は Nagios compatible に沿って作ればいい

InfluxDB連携

InfludDB setup

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release 
sudo sh -c "echo \"deb https://repos.influxdata.com/${ID} ${UBUNTU_CODENAME} stable\" > /etc/apt/sources.list.d/influxdb.list"
sudo apt update
sudo apt install -y influxdb
sudo systemctl start influxdb
sudo systemctl enable influxdb
sudo systemctl status influxdb
influx
> CREATE DATABASE sensu
> CREATE USER sensu WITH PASSWORD 'password' WITH ALL PRIVILEGES

Sensu backend setup

sensuctl asset create sensu-influxdb-handler --url "https://github.com/sensu/sensu-influxdb-handler/releases/download/3.1.2/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz" --sha512 "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"
sensuctl handler create influxdb --command "sensu-influxdb-handler -d sensu" -i 10 --env-vars 'INFLUXDB_ADDR=http://127.0.0.1:8086, INFLUXDB_USER=sensu, INFLUXDB_PASS=password' --filters "has_metrics" --runtime-assets sensu-influxdb-handler
sensuctl check create curl_to_influxdb \
--command "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://127.0.0.1" \
--interval 30 \
--subscriptions "location:jp_tokyo" \
--output-metric-format "graphite_plaintext" \
--output-metric-handlers "influxdb"

Show datastore

CLI

$ influx
Connected to http://localhost:8086 version 1.7.6
InfluxDB shell version: 1.7.6
Enter an InfluxQL query
> use sensu
Using database sensu
> SELECT * FROM example
name: example
time                curl_timings.http_code curl_timings.time_connect curl_timings.time_namelookup curl_timings.time_pretransfer curl_timings.time_redirect curl_timings.time_starttransfer curl_timings.time_total sensu_entity_name
----                ---------------------- ------------------------- ---------------------------- ----------------------------- -------------------------- ------------------------------- ----------------------- -----------------
1557208260000000000 200                    0.000171                  0.000019                     0.000258                      0                          0.000262                        0.000281                example
1557208290000000000 200                    0.000104                  0.000019                     0.000196                      0                          0.0002                          0.00022                 example
1557208320000000000 200                    0.000133                  0.00002                      0.000236                      0                          0.00024                         0.000258                example
1557208350000000000 200                    0.000099                  0.00002                      0.000188                      0                          0.000192                        0.000209                example
1557208380000000000 200                    0.000102                  0.000019                     0.000216                      0                          0.00022                         0.000251                example
1557208410000000000 200                    0.000103                  0.00002                      0.000192                      0                          0.000195                        0.000212                example
  • sensu_entity_nameを見ると分かるが、Agent名が入ってしまうため、HTTP/HTTPSで2つの監視設定を入れるとデータベースが混ざってしまうため、check単位で異なるproxy_entity_nameを設定する必要がある

API

curl 'http://127.0.0.1:8086/query?db=sensu' --data-urlencode 'q=SELECT * FROM "example"' | python3 -m json.tool

InfluxDBのデータをChronografで確認する場合

curl -L -O https://dl.influxdata.com/chronograf/releases/chronograf_1.7.11_amd64.deb
sudo dpkg -i chronograf_1.7.11_amd64.deb

Access: http://localhost:8888/

Telegraf Database Name/Dashboard/Kapacitorの設定はとりあえずSKIPしてもOK。

マルチエージェント用のsystemd

/etc/systemd/system/sensu-agent-namespace@.service

[Unit]
Description=The Sensu Agent process for each namespaces.
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=sensu
Group=sensu
EnvironmentFile=-/etc/default/sensu-agent
EnvironmentFile=-/etc/sysconfig/sensu-agent
LimitNOFILE=65535
ExecStart=/usr/sbin/sensu-agent start -c /etc/sensu/namespaces/%i.yml
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

使い方

$ sudo systemctl daemon-reload
$ sudo cat /etc/sensu/namespaces/user0001.yml
---
name: "jp_tokyo"
namespace: "user0001"
backend-url:
  - "ws://192.168.122.201:8081"
cache-dir: "/var/cache/sensu/sensu-agent/namespaces/user0001"
api-port: 10001
socket-port: 20001
statsd-metrics-port: 30001
$ sudo systemctl start sensu-agent-namespace@user0001.service

最終更新日: 2021-06-06 09:29:57