articles

Cyber threat insights with Crowdsec, VictoriaMetrics and Grafana

• Publish Date
• Weakness in Crowdsec OpenMetrics statistics
• Different solution
• References

Brewing a low-coding open source cyber threats analysis solution based on Crowdsec ¹, VictoriaMetrics ², and Grafana ³.

Publish Date

• 2024-08-10

Weakness in Crowdsec OpenMetrics statistics

Out of the box, each Crowdsec instance can provide the statistics in the Prometheus ⁴ or OpenMetrics ⁵ format about its engine status, such as:

• decisions
• alerts
• etc.

All possible data slices are explained in the Crowdec metrics documentation ⁶. There are official Grafana dashboards for Crowdsec ⁷, but these metrics do not contain the information that most operators are interested in, such as cyber threat statistics. For example, active decisions and alerts contain only the scenario metrics:

~> curl -s localhost:6060/metrics | egrep 'cs_alerts{|cs_active_decisions{'
cs_active_decisions{action="ban",origin="crowdsec",reason="crowdsecurity/ssh-bf"} 1
cs_alerts{reason="crowdsecurity/ssh-bf"} 24
cs_alerts{reason="crowdsecurity/ssh-bf_user-enum"} 5
cs_alerts{reason="crowdsecurity/ssh-slow-bf"} 11

To obtain cyber threat statistics, Crowdsec requires the use of Console ⁸ as the central management system:

Also, the Metabase ⁹ can provide this statistics with cscli dashboard ¹⁰:

The 1^st option requires binding to the centralized Crowdsec ecosystem and feeding it with enterprise infrastructure data, the 2^nd requires to start Metabase on every existing Crowdsec node and looks mostly unmanageable.

Different solution

The other self-hosted low-code ¹¹ solution with the almost same results can be reached by integration of Crowdsec with VictoriaMetrics. Later the stored data easily can be represented in Grafana:

You can download this dashboard at Grafana dashboards.

There are also available all powerful Grafana controls for the analysis:

• Data filtering
• Data links
• Value Mapping
• etc.

For example, dynamic filtering the entire dashboard by a single country simply clicking on chart:

Additional data links are helping with analysis of a particular ASN record with information from RIRs ¹²:

The subnet information could be investigated by both information from RIR and online security databases such as Crowdsec CTI ¹³, Shodan ¹⁴, Censys ¹⁵, Criminal IP ¹⁶, and similar:

The IP address can only be analyzed with online security databases:

Also there is a geomap with coordinates found by the built-in Crowdsec GeoIP module ¹⁷ based on Maxmind Lite ¹⁸ databases.

Integration steps

Overall integration and call-flow landscape can be depicted as follows:

As shown above, the /api/v1/import ¹⁹ endpoint on VictoriaMetrics instance is utilized to collect the data from Crowdsec nodes, it accepts JSON Lines ²⁰ text format.

The coding skills are not required to make all this work, only the template configuration for Crowdsec HTTP notification plugin ²¹ is needed within the /path/to/notifications/http.yaml file:

type: http
name: http_default
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"<METRIC_NAME>","instance":"<INSTANCE_NAME>","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: <VICTORIAMETRICS_INSTANCE_PROTO>://<VICTORIAMETRICS_INSTANCE>/api/v1/import
method: POST
headers:
  Content-Type: application/json
  Authorization: "<AUTH_STRING>"

Where <VAR_NAME> are placeholders required to be filled with real data:

• <METRIC_NAME> - the name of the metric being produced, the current implementation contains cs_lapi_decision.
• <INSTANCE_NAME> - the name of the instance that sends this data, e.g. host00.domain.tld.
• <VICTORIAMETRICS_INSTANCE_PROTO> - the VictoriaMetrics instance’s API protocol, http or https.
• <VICTORIAMETRICS_INSTANCE> - domain name or IP of the VictoriaMetrics instance.
• <AUTH_STRING> - the contents of the Authorization header ²², for example, could be Basic Authorization ²³ or OAuth2 Bearer ²⁴. If authorization does not require, the entire Authorization header needs to be removed from template.

Each time the template produces a value of 1, which means one record. This is the trick of converting Log records to Metric records. For more information on observability signals please read Observability Whitepaper ²⁵.

And since the value is 1 each time, it’s easy to count, summarize, and perform other simple math operations for these records.

Finally, the following example JSON payload will be rendered by this template as a single line, since it strictly needs to be compatible with JSON Lines, here it is formatted for clarity:

{
    "metric": {
        "__name__": "cs_lapi_decision",
        "instance": "host00.domain.tld",
        "country": "SG",
        "asname": "Tencent Building, Kejizhongyi Avenue",
        "asnumber": "132203",
        "latitude": "1.2868",
        "longitude": "103.8503",
        "iprange": "129.226.0.0/16",
        "scenario": "crowdsecurity/ssh-slow-bf",
        "type": "ban",
        "duration": "4h",
        "scope": "Ip",
        "ip": "129.226.84.230"
    },
    "values": [1],
    "timestamps": [1723013301000]
}

In VictoriaMertics, this will be shown as a sample of cs_lapi_decision time series:

Please do not forget to enable HTTP notifications plugin in the main Crowdsec configuration config.yaml:

...
config_paths:
  ...
  notification_dir: /path/to/notifications/
  ...
api:
  ...
  server:
    ...
    profiles_path: /path/to/profiles.yaml
...

and profiles.yaml:

...
notifications:
  ...
  - http_default
  ...
...

References

1. Crowdsec - Real-time & crowdsourced protection against aggressive IPs ↩
2. VictoriaMetrics - High-performance, open source time series database & monitoring solution ↩
3. Grafana - Multi-platform open source analytics and interactive visualization web application ↩
4. Prometheus - Open source systems monitoring and alerting toolkit ↩
5. OpenMetrics - Standard for transmitting cloud-native metrics at scale ↩
6. Crowdec metrics documentation ↩
7. Grafana dashboards for Crowdsec ↩
8. CrowdSec Console - Central management web interface ↩
9. Metabase - Open source business intelligence platform ↩
10. cscli dashboard ↩
11. low-code - Mostly visual approach to software development ↩
12. RIR - Regional Internet registry ↩
13. Crowdsec CTI - OSINT search engine specialized in attack surface assessment and threat hunting ↩
14. Shodan - OSINT search engine specialized in attack surface assessment and threat hunting ↩
15. Censys - OSINT search engine specialized in attack surface assessment and threat hunting ↩
16. Criminal IP - OSINT search engine specialized in attack surface assessment and threat hunting ↩
17. Crowdsec GeoIP module - GeoIP module relies on geolite database to provide enrichment on source ip ↩
18. Maxmind Lite - Less accurate free IP geolocation databases ↩
19. Import data to VictoriaMetrics in JSON line format ↩
20. JSON Lines - Convenient format for storing structured data that may be processed one record at a time ↩
21. Crowdsec HTTP notification plugin ↩
22. HTTP Authorization header ↩
23. HTTP Basic Authorization ↩
24. HTTP OAuth2 Authentication ↩
25. Observability Whitepaper ↩