Sensu¶
{{toc}}
Go言語で書き直されたSensu。
パッケージは以下の3つ
- sensu-go-agent
- sensu-go-backend
- sensu-go-cli
インストール¶
https://docs.sensu.io/sensu-go/5.5/installation/install-sensu/
Backend¶
curl -s https://packagecloud.io/install/repositories/sensu/stable/script.deb.sh | sudo bash
sudo apt update
sudo apt-get install sensu-go-backend
sudo cp /usr/share/doc/sensu-go-backend-5.5.1/backend.yml.example /etc/sensu/backend.yml
sudo systemctl start sensu-backend
sudo systemctl enable sensu-backend
sudo systemctl status sensu-backend
CLI¶
sudo apt-get install sensu-go-cli
sensuctl configure
? Sensu Backend URL: http://127.0.0.1:8080
? Username: admin
? Password: *********
? Namespace: default
? Preferred output format: tabular
export TOKEN=$(cat ~/.config/sensu/sensuctl/cluster | jq -r .access_token)
curl -s http://127.0.0.1:8080/api/core/v2/namespaces/default/events -H "Authorization: Bearer ${TOKEN}" | jq
Non-interactive: https://docs.sensu.io/sensu-go/5.0/sensuctl/reference/#non-interactive
sensuctl configure -n --url http://127.0.0.1:8080 --username admin --password P@ssw0rd! --format tabular
with namespace
sensuctl configure -n --url http://127.0.0.1:8080 --username admin --password P@ssw0rd! --format tabular --namespace user01
Agent¶
sudo apt-get install sensu-go-agent
sudo systemctl start sensu-agent
sudo systemctl enable sensu-agent
sudo systemctl status sensu-agent
リモートホストの場合はテンプレートを持ってきて、127.0.0.1をbackendに向ける。
sudo cp /usr/share/doc/sensu-go-agent-5.5.1/agent.yml.example /etc/sensu/agent.yml
Plugin installer¶
https://docs.sensu.io/sensu-go/5.5/installation/plugins/
curl -s https://packagecloud.io/install/repositories/sensu/community/script.deb.sh | sudo bash
sudo apt install -y sensu-plugins-ruby
Plugins¶
https://github.com/sensu-plugins からダウンロード対象を選ぶ。
例えば sensu-plugins-http
であれば
sudo sensu-install -p http
プラグインによっては拡張のビルドに失敗したりする。httpの場合は以下も必要かもしれない。
sudo apt install -y build-essential
主要なプラグイン¶
sudo sensu-install -p http
sudo sensu-install -p sensu-plugins-network-checks
sudo sensu-install -p dns
- https://github.com/sensu-plugins/sensu-plugins-http
- https://github.com/sensu-plugins/sensu-plugins-network-checks
パスの追加¶
※これじゃダメそう
$ cat /etc/profile.d/05-sensu-plugins.sh
export PATH=$PATH:/opt/sensu-plugins-ruby/embedded/bin
Usage¶
HTTP監視の追加¶
プラグイン挙動の確認
$ /opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u https://www.ainoniwa.net
example.curl_timings.time_total 0.013437 1556126157
example.curl_timings.time_namelookup 0.004224 1556126157
example.curl_timings.time_connect 0.004775 1556126157
example.curl_timings.time_pretransfer 0.012694 1556126157
example.curl_timings.time_redirect 0.000000 1556126157
example.curl_timings.time_starttransfer 0.013400 1556126157
example.curl_timings.http_code 200 1556126157
ちなみにこれは curl コマンドの結果を整形したに過ぎない
sh -c LC_NUMERIC=C curl --silent --output /dev/null -w "%{time_total},%{time_namelookup},%{time_connect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{http_code}" http://localhost
追加
sensuctl create --file curl_timings-check.json
curl_timings-check.json
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "curl_timings",
"namespace": "default"
},
"spec": {
"check_hooks": null,
"command": "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://www.ainoniwa.net",
"env_vars": null,
"handlers": [],
"high_flap_threshold": 0,
"interval": 10,
"low_flap_threshold": 0,
"output_metric_format": "graphite_plaintext",
"output_metric_handlers": [],
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [],
"stdin": false,
"subdue": null,
"subscriptions": [
"entity:example"
],
"timeout": 0,
"ttl": 0
}
}
subscriptions
の部分で実行先のentityを決めているらしい。
https://docs.sensu.io/sensu-go/5.5/reference/checks/#subscriptions
後から追加する場合は以下のように書ける。
sensuctl check set-subscriptions curl_timings '["entity:example", "entity:example-dev2"]'
Slack通知の追加¶
$ cat sensu-slack-handler.json
{
"type": "Handler",
"spec": {
"metadata": {
"name": "slack",
"namespace": "default",
"labels": null,
"annotations": null
},
"type": "pipe",
"command": "sensu-slack-handler -w ${SLACK_WEBHOOK_URL} -c ${SLACK_CHANNEL}",
"timeout": 0,
"handlers": [],
"filters": [""],
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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",
"SLACK_CHANNEL=@yamabukir"
],
"runtime_assets": ["sensu-slack-handler"]
}
}
$ sensuctl create --file sensu-slack-handler.json
$ sensuctl entity list
ID Class OS Subscriptions Last Seen
──────── ─────── ─────── ─────────────── ───────────────────────────────
ubuntu agent linux entity:ubuntu 2019-04-24 18:35:16 +0900 JST
$ sensuctl event list
Entity Check Output Status Silenced Timestamp
──────── ─────────── ────────────────────────────────────────────────────────────────── ──────── ────────── ───────────────────────────────
ubuntu keepalive Keepalive last sent from ubuntu at 2019-04-24 09:35:36 +0000 UTC 0 false 2019-04-24 18:35:36 +0900 JST
$ sensuctl handler list
Name Type Timeout Filters Mutator Execute Environment Variables Assets
─────── ────── ───────── ───────── ───────── ─────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ─────────────────────
slack pipe 0 RUN: sensu-slack-handler -w ${SLACK_WEBHOOK_URL} -c ${SLACK_CHANNEL} SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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,SLACK_CHANNEL=@yamabukir sensu-slack-handler
Memo¶
どうやらbackendとserverの間で設定するhandlerは keepalive
という予約語になるらしい。
https://github.com/sensu/sensu-go/issues/1960
Built-in filters に障害が発生した場合を指すFilter is_incident
があるらしい。大体はこれをセットしておけばいい。(逆にFilterをセットしないと、ポーリング間隔毎にhandlerが動いてしまうので、slack通知が無限に飛んできたりする。
https://docs.sensu.io/sensu-go/5.5/reference/filters/#how-do-sensu-filters-work
sensuctl handler create keepalive \
--type pipe \
--env-vars "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/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" \
--command "sensu-slack-handler --channel '@yamabukir'" \
--runtime-assets sensu-slack-handler
1つのAgentは1つのEntityで表現されるので、ユーザー毎に異なる名前空間に
for num in `seq -w 1 1000`; do
sensuctl namespace create uid000${num};
sensuctl check create curl_local \
--command "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://192.168.122.201" \
--interval 30 \
--subscriptions "location:jp_tokyo" \
--namespace "uid000${num}" \
--output-metric-format "graphite_plaintext" \
--output-metric-handlers "output_metric_handlers"
done
REST API Calling¶
$ curl -s -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization:Basic YWRtaW46UEBzc3cwcmQh" http://127.0.0.1:8080/auth | jq
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTcyMDM1MDIsImp0aSI6IjBkYWNhMDIzMjEwNTM2NjgxMmVmMTBlNjUwNzExYjczIiwic3ViIjoiYWRtaW4iLCJncm91cHMiOlsiY2x1c3Rlci1hZG1pbnMiLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInVzZXJfaWQiOiJhZG1pbiJ9fQ.6rynHV0DSHqUw4jU8DbzcEMeqtY8roIG-fr_WvE3Mms",
"expires_at": 1557203502,
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJkN2E5ZDZiNTM3MTZiYTQ1ZjA4NzJmOTNkYTBmZGFmMCIsInN1YiI6ImFkbWluIiwiZ3JvdXBzIjpudWxsLCJwcm92aWRlciI6eyJwcm92aWRlcl9pZCI6IiIsInVzZXJfaWQiOiIifX0.PukUNX1cL3Z9l_7iw6auaMzOUM3PuBI3RPfhTZLALFA"
}
$ curl http://127.0.0.1:8080/api/core/v2/entities -H "Authorization: Bearer eyJhbGciOi..."
コピペ用
TOKEN=`curl -s -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization:Basic YWRtaW46UEBzc3cwcmQh" http://127.0.0.1:8080/auth | jq -r .access_token`
curl http://127.0.0.1:8080/api/core/v2/checks -s -H "Authorization: Bearer ${TOKEN}"
Filtering (Need to enterprise license)¶
https://docs.sensu.io/sensu-go/5.6/api/overview/#filtering
$ curl http://127.0.0.1:8080/api/core/v2/entities -H "Authorization: Bearer ${TOKEN}" -G --data-urlencode 'labelSelector=entity_class == "proxy"'
{"message":"missing or invalid license","code":6}
Agent¶
基本は Nagios compatible に沿って作ればいい
- python用テンプレート https://github.com/sensu-plugins/sensu-plugin-python
InfluxDB連携¶
InfludDB setup¶
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release
sudo sh -c "echo \"deb https://repos.influxdata.com/${ID} ${UBUNTU_CODENAME} stable\" > /etc/apt/sources.list.d/influxdb.list"
sudo apt update
sudo apt install -y influxdb
sudo systemctl start influxdb
sudo systemctl enable influxdb
sudo systemctl status influxdb
influx
> CREATE DATABASE sensu
> CREATE USER sensu WITH PASSWORD 'password' WITH ALL PRIVILEGES
Sensu backend setup¶
sensuctl asset create sensu-influxdb-handler --url "https://github.com/sensu/sensu-influxdb-handler/releases/download/3.1.2/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz" --sha512 "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"
sensuctl handler create influxdb --command "sensu-influxdb-handler -d sensu" -i 10 --env-vars 'INFLUXDB_ADDR=http://127.0.0.1:8086, INFLUXDB_USER=sensu, INFLUXDB_PASS=password' --filters "has_metrics" --runtime-assets sensu-influxdb-handler
sensuctl check create curl_to_influxdb \
--command "/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u http://127.0.0.1" \
--interval 30 \
--subscriptions "location:jp_tokyo" \
--output-metric-format "graphite_plaintext" \
--output-metric-handlers "influxdb"
Show datastore¶
CLI¶
$ influx
Connected to http://localhost:8086 version 1.7.6
InfluxDB shell version: 1.7.6
Enter an InfluxQL query
> use sensu
Using database sensu
> SELECT * FROM example
name: example
time curl_timings.http_code curl_timings.time_connect curl_timings.time_namelookup curl_timings.time_pretransfer curl_timings.time_redirect curl_timings.time_starttransfer curl_timings.time_total sensu_entity_name
---- ---------------------- ------------------------- ---------------------------- ----------------------------- -------------------------- ------------------------------- ----------------------- -----------------
1557208260000000000 200 0.000171 0.000019 0.000258 0 0.000262 0.000281 example
1557208290000000000 200 0.000104 0.000019 0.000196 0 0.0002 0.00022 example
1557208320000000000 200 0.000133 0.00002 0.000236 0 0.00024 0.000258 example
1557208350000000000 200 0.000099 0.00002 0.000188 0 0.000192 0.000209 example
1557208380000000000 200 0.000102 0.000019 0.000216 0 0.00022 0.000251 example
1557208410000000000 200 0.000103 0.00002 0.000192 0 0.000195 0.000212 example
- sensu_entity_nameを見ると分かるが、Agent名が入ってしまうため、HTTP/HTTPSで2つの監視設定を入れるとデータベースが混ざってしまうため、check単位で異なるproxy_entity_nameを設定する必要がある
API¶
curl 'http://127.0.0.1:8086/query?db=sensu' --data-urlencode 'q=SELECT * FROM "example"' | python3 -m json.tool
InfluxDBのデータをChronografで確認する場合¶
curl -L -O https://dl.influxdata.com/chronograf/releases/chronograf_1.7.11_amd64.deb
sudo dpkg -i chronograf_1.7.11_amd64.deb
Access: http://localhost:8888/
Telegraf Database Name/Dashboard/Kapacitorの設定はとりあえずSKIPしてもOK。
マルチエージェント用のsystemd¶
/etc/systemd/system/sensu-agent-namespace@.service
[Unit]
Description=The Sensu Agent process for each namespaces.
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=sensu
Group=sensu
EnvironmentFile=-/etc/default/sensu-agent
EnvironmentFile=-/etc/sysconfig/sensu-agent
LimitNOFILE=65535
ExecStart=/usr/sbin/sensu-agent start -c /etc/sensu/namespaces/%i.yml
Restart=always
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
使い方¶
$ sudo systemctl daemon-reload
$ sudo cat /etc/sensu/namespaces/user0001.yml
---
name: "jp_tokyo"
namespace: "user0001"
backend-url:
- "ws://192.168.122.201:8081"
cache-dir: "/var/cache/sensu/sensu-agent/namespaces/user0001"
api-port: 10001
socket-port: 20001
statsd-metrics-port: 30001
$ sudo systemctl start sensu-agent-namespace@user0001.service