✨ add alone mode

README.md

@@ -16,7 +16,7 @@ The crowdsec utility will provide the community blocklist which contains highly
 
 When used with crowdsec it will leverage the local API which will analyze traefik logs and take decisions on the requests made by users/bots. Malicious actors will be banned based on patterns against your website.
 
-There are 3 operating modes (CrowdsecMode) for this plugin:
+There are 4 operating modes (CrowdsecMode) for this plugin:
 - none -> If the client IP is on ban list, it will get a http code 403 response.
     Otherwise, request will continue as usual. All request call the Crowdsec LAPI
 
@@ -30,11 +30,47 @@ There are 3 operating modes (CrowdsecMode) for this plugin:
     every requests that does not hit the cache is authorized.
     Every minute, the cache is updated with news from the Crowdsec LAPI.
 
+- alone -> Streaming mode but the blacklisted IPs are fetched on the CAPI.
+    Every 2 hours, the cache is updated with news from the Crowdsec CAPI.
+
 The recommanded mode for performance is the streaming mode, decisions are updated every 60 sec by default and that's the only communication between traefik and crowdsec. Every requests that happens hits the cache for quick decisions.
 
 ## Usage
 
-
+### Variables
+- Enabled
+  - bool
+  - enable the plugin
+- CrowdsecMode
+  - string
+  - default: `stream`, expected value are: `none`, `live`, `stream`, `alone`
+- CrowdsecLapiScheme
+  - string
+  - default: `http`, expected value are: `http`, `https`
+- CrowdsecLapiHost
+  - string
+  - default: "crowdsec:8080"
+  - Crowdsec LAPI available on which host.
+- CrowdsecLapiKey
+  - string
+  - Crowdsec LAPI generated key for the bouncer. 
+- CrowdsecCapiLogin
+  - string
+  - Used only in `alone` mode, login for Crowdsec CAPI
+- CrowdsecCapiPwd
+  - string
+  - Used only in `alone` mode, password for Crowdsec CAPI
+- CrowdsecCapiScenarios
+  - []string
+  - Used only in `alone` mode, login for Crowdsec CAPI
+- UpdateIntervalSeconds
+  - int64
+  - default: 60
+  - User only in `stream` mode, interval of time between fetching blacklisted IPs from LAPI
+- DefaultDecisionSeconds
+  - int64
+  - default: 60
+  - User only in `live` mode, decision duration of accepted IPs
 
 ### Configuration
 
@@ -75,14 +111,86 @@ http:
       plugin:
         bouncer:
           enabled: false
+          updateIntervalSeconds: 60
+          defaultDecisionSeconds: 60
+          crowdsecMode: stream
           crowdsecLapiKey: privateKey
           crowdsecLapiHost: crowdsec:8080
           crowdsecLapiScheme: http
-          crowdsecMode: stream
+          crowdsecCapiLogin: login
-          updateIntervalSeconds: 60
+          crowdsecCapiPwd: password
-          defaultDecisionSeconds: 60
+          crowdsecCapiScenarios:
+            - scenario1
+            - scenario2
+            ...
+
+```
+Except for the crowdsecLapiKey, crowdsecCapiLogin, crowdsecCapiPwd, crowdsecCapiScenarios, these are the default value of the plugin.
+
+#### Generate LAPI KEY (exept for `alone` mode)
+You need to generate a crowdsec API key for the LAPI.
+You can follow the documentation here: https://docs.crowdsec.net/docs/user_guides/lapi_mgmt/
+
+```bash
+docker-compose -f docker-compose-local.yml up -d crowdsec
+docker exec crowdsec cscli bouncers add crowdsecBouncer
+```
+
+This LAPI key must be set where is noted FIXME-LAPI-KEY in the docker-compose-test.yml
+```yaml
+...
+whoami:
+  labels:
+    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=FIXME-LAPI-KEY"
+...
+crowdsec:
+  environment:
+    BOUNCER_KEY_TRAEFIK: FIXME-LAPI-KEY
+...
+```
+
+You can then run all the containers:
+```bash
+docker-compose up -d
+```
+
+#### Generate CAPI credentials (only for `alone` mode)
+You need to create a crowdsec API credentials for the CAPI.
+You can follow the documentation here: https://docs.crowdsec.net/docs/central_api/intro
+
+```bash
+curl -X POST "https://api.crowdsec.net/v2/watchers" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{ \"password\": \"PASSWORD\",  \"machine_id\": \"LOGIN\"}"
+```
+
+These CAPI credentials must be set in your docker-compose.yml or in your config files
+```yaml
+...
+traefik:
+  command:
+    ...
+    - "--experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
+    - "--experimental.plugins.bouncer.version=v1.0.0"
+    ...
+whoami:
+  labels:
+    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapilogin=LOGIN"
+    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapipwd=PASSWORD"
+    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapiscenarios=scenario1, scenario2, ..."
+    - "traefik.http.middlewares.crowdsec.plugin.bouncer.enabled=true"
+```
+
+You can then run all the containers:
+```bash
+docker-compose up -d
+```
+
+#### Add manually an IP to the blocklist (testing purpose)
+
+```bash
+docker-compose up -d crowdsec
+docker exec crowdsec cscli decisions add --ip 10.0.0.10 # this will be effective 4h
+docker exec crowdsec cscli decisions remove --ip 10.0.0.10
 ```
-Except for the crowdsecLapiKey, these are the default value of the plugin.
 
 ### Local Mode
 
@@ -114,71 +222,9 @@ For local developpement a docker-compose.local.yml is provided and reproduce the
 docker-compose -f docker-compose.local.yml up -d
 ```
 
-#### Generate LAPI-KEY
-You need to generate a crowdsec API key for the LAPI.
-You can follow the documentation here: https://docs.crowdsec.net/docs/user_guides/lapi_mgmt/
-
-```bash
-docker-compose -f docker-compose.local.yml up -d crowdsec
-docker exec crowdsec cscli bouncers add TRAEFIK
-```
-
-This LApi key must be set where is noted FIXME-LAPI-KEY in the docker-compose-test.yml
-```yaml
-...
-whoami:
-  labels:
-    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=FIXME-LAPI-KEY"
-...
-crowdsec:
-  environment:
-    BOUNCER_KEY_TRAEFIK: FIXME-LAPI-KEY
-...
-```
-
-You can then run all the containers:
-```bash
-docker-compose -f docker-compose-local.yml up -d
-```
-
-#### Generate LAPI-KEY
-You need to generate a crowdsec API key for the LAPI.
-You can follow the documentation here: https://docs.crowdsec.net/docs/user_guides/lapi_mgmt/
-
-```bash
-docker-compose -f docker-compose-local.yml up -d crowdsec
-docker exec crowdsec cscli bouncers add crowdsecBouncer
-```
-
-This LApi key must be set where is noted FIXME-LAPI-KEY in the docker-compose-test.yml
-```yaml
-...
-whoami:
-  labels:
-    - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=FIXME-LAPI-KEY"
-...
-crowdsec:
-  environment:
-    BOUNCER_KEY_TRAEFIK: FIXME-LAPI-KEY
-...
-```
-
-You can then run all the containers:
-```bash
-docker-compose -f docker-compose-local.yml up -d
-```
-
-#### Add manually an IP to the blocklist
-
-```bash
-docker-compose -f docker-compose-local.yml up -d crowdsec
-docker exec crowdsec cscli decisions add --ip 10.0.0.10 # this will be effective 4h
-docker exec crowdsec cscli decisions remove --ip 10.0.0.10
-```
-
 ### About
 
-[maxlerebourg](https://github.com/maxlerebourg) and [I](https://github.com/mhanotaux) have been using traefik since 2020.
+Me and [mhanotaux](https://github.com/mhanotaux) have been using traefik since 2020 at [Primadviz](https://primadviz.com).
 We come from web developper and security engineer background and wanted to add the power of a very promesing technology (Crowdsec) into the edge router we love.
 
 We initially run into this project: https://github.com/fbonalair/traefik-crowdsec-bouncer

bouncer.go

@@ -11,6 +11,7 @@ import (
 	"net"
 	"net/http"
 	"net/url"
+	"strings"
 	"text/template"
 	"time"
 
@@ -18,9 +19,12 @@ import (
 )
 
 const (
-	crowdsecAuthHeader  = "X-Api-Key"
+	crowdsecLapiHeader    = "X-Api-Key"
-	crowdsecRoute       = "v1/decisions"
+	crowdsecCapiHeader    = "Authorization"
-	crowdsecStreamRoute = "v1/decisions/stream"
+	crowdsecLapiRoute         = "v1/decisions"
+	crowdsecLapiStreamRoute   = "v1/decisions/stream"
+	crowdsecCapiLogin     = "v2/watchers/login"
+	crowdsecCapiDecisions = "v2/decisions/stream"
 	cacheBannedValue      = "t"
 	cacheNoBannedValue    = "f"
 )
@@ -32,6 +36,9 @@ type Config struct {
 	CrowdsecLapiScheme     string   `json:"crowdsecLapiScheme,omitempty"`
 	CrowdsecLapiHost       string   `json:"crowdsecLapiHost,omitempty"`
 	CrowdsecLapiKey        string   `json:"crowdsecLapiKey,omitempty"`
+	CrowdsecCapiLogin      string   `json:"crowdsecCapiLogin,omitempty"`
+	CrowdsecCapiPwd        string   `json:"crowdsecCapiPwd,omitempty"`
+	CrowdsecCapiScenarios  []string `json:"crowdsecCapiScenarios,omitempty"`
 	UpdateIntervalSeconds  int64    `json:"updateIntervalSeconds,omitempty"`
 	DefaultDecisionSeconds int64    `json:"defaultDecisionSeconds,omitempty"`
 }
@@ -40,10 +47,13 @@ type Config struct {
 func CreateConfig() *Config {
 	return &Config{
 		Enabled:                false,
-		CrowdsecMode:           "stream",
+		CrowdsecMode:           "alone",
 		CrowdsecLapiScheme:     "http",
 		CrowdsecLapiHost:       "crowdsec:8080",
 		CrowdsecLapiKey:        "",
+		CrowdsecCapiLogin:      "",
+		CrowdsecCapiPwd:        "",
+		CrowdsecCapiScenarios:  []string{},
 		UpdateIntervalSeconds:  60,
 		DefaultDecisionSeconds: 60,
 	}
@@ -63,52 +73,61 @@ type Bouncer struct {
 	crowdsecMode           string
 	updateInterval         int64
 	defaultDecisionTimeout int64
+	crowdsecLogin          string
+	crowdsecPwd            string
+	crowdsecScenarios      []string
 	client                 *http.Client
 	cache                  *ttl_map.Heap
 }
 
 // New creates the crowdsec bouncer plugin.
 func New(ctx context.Context, next http.Handler, config *Config, name string) (http.Handler, error) {
-	requiredStrings := map[string]string{
+	var requiredStrings map[string]string
+	if config.CrowdsecMode == "alone" {
+		requiredStrings = map[string]string{
+			"CrowdsecCapiLogin": config.CrowdsecLapiScheme,
+			"CrowdsecCapiPwd":   config.CrowdsecLapiHost,
+		}
+		for _, val := range config.CrowdsecCapiScenarios {
+			if len(val) == 0 {
+				return nil, fmt.Errorf("CrowdsecCapiScenarios: one or more scenario are empty")
+			}
+		}
+		config.UpdateIntervalSeconds = 7200
+		config.CrowdsecLapiKey = ""
+		config.CrowdsecLapiScheme = "https"
+		config.CrowdsecLapiHost = "api.crowdsec.net"
+	} else {
+		requiredStrings = map[string]string{
 			"CrowdsecLapiScheme": config.CrowdsecLapiScheme,
 			"CrowdsecLapiHost":   config.CrowdsecLapiHost,
 			"CrowdsecLapiKey":    config.CrowdsecLapiKey,
 			"CrowdsecMode":       config.CrowdsecMode,
 		}
-	for key, val := range requiredStrings {
-		if len(val) == 0 {
-			return nil, fmt.Errorf("%v cannot be empty", key)
-		}
-	}
 		requiredInt := map[string]int64{
 			"UpdateIntervalSeconds":  config.UpdateIntervalSeconds,
 			"DefaultDecisionSeconds": config.DefaultDecisionSeconds,
 		}
 		for key, val := range requiredInt {
 			if val < 1 {
-			return nil, fmt.Errorf("%v cannot be less than 1", key)
+				return nil, fmt.Errorf("%v: cannot be less than 1", key)
 			}
 		}
-	// none -> If the client IP is on ban list, it will get a http code 403 response.
+	}
-	//         Otherwise, request will continue as usual. All request call the Crowdsec LAPI
+	for key, val := range requiredStrings {
-	// live ->  If the client IP is on ban list, it will get a http code 403 response.
+		if len(val) == 0 {
-	//          Otherwise, request will continue as usual.
+			return nil, fmt.Errorf("%v: cannot be empty", key)
-	//          The bouncer can leverage use of a local cache in order to reduce the number
+		}
-	//          of requests made to the Crowdsec LAPI. It will keep in cache the status for
+	}
-	//          each IP that makes queries.
+	if !contains([]string{"none", "live", "stream", "alone"}, config.CrowdsecMode) {
-	// stream -> Stream Streaming mode allows you to keep in the local cache only the Banned IPs,
+		return nil, fmt.Errorf("CrowdsecMode: must be one of 'none', 'live' or 'stream'")
-	// 			every requests that does not hit the cache is authorized.
-	// 			Every minute, the cache is updated with news from the Crowdsec LAPI.
-	if !contains([]string{"none", "live", "stream"}, config.CrowdsecMode) {
-		return nil, fmt.Errorf("CrowdsecMode must be one of: none, live or stream")
 	}
 	if !contains([]string{"http", "https"}, config.CrowdsecLapiScheme) {
-		return nil, fmt.Errorf("CrowdsecLapiScheme must be one of: http, https")
+		return nil, fmt.Errorf("CrowdsecLapiScheme: must be one of 'http' or 'https'")
 	}
 	testURL := url.URL{
 		Scheme: config.CrowdsecLapiScheme,
 		Host:   config.CrowdsecLapiHost,
-		Path:   crowdsecRoute,
 	}
 	_, err := http.NewRequest(http.MethodGet, testURL.String(), nil)
 	if err != nil {
@@ -126,6 +145,9 @@ func New(ctx context.Context, next http.Handler, config *Config, name string) (h
 		crowdsecScheme:         config.CrowdsecLapiScheme,
 		crowdsecHost:           config.CrowdsecLapiHost,
 		crowdsecKey:            config.CrowdsecLapiKey,
+		crowdsecLogin:          config.CrowdsecCapiLogin,
+		crowdsecPwd:            config.CrowdsecCapiPwd,
+		crowdsecScenarios:      config.CrowdsecCapiScenarios,
 		updateInterval:         config.UpdateIntervalSeconds,
 		defaultDecisionTimeout: config.DefaultDecisionSeconds,
 		client: &http.Client{
@@ -140,6 +162,11 @@ func New(ctx context.Context, next http.Handler, config *Config, name string) (h
 	// if we are on a stream mode, we fetch in a go routine every minute the new decisions.
 	if config.CrowdsecMode == "stream" {
 		go handleStreamCache(bouncer, true)
+	} else if config.CrowdsecMode == "alone" {
+		getToken(bouncer)
+		time.AfterFunc(10*time.Second, func() {
+			handleStreamCache(bouncer, false)
+		})
 	}
 	return bouncer, nil
 }
@@ -204,6 +231,13 @@ type Stream struct {
 	New     []Decision `json:"new"`
 }
 
+// Login Body returned from Crowdsec Login CAPI.
+type Login struct {
+	Code   int    `json:"code"`
+	Token  string `json:"token"`
+	Expire string `json:"expire"`
+}
+
 func contains(source []string, target string) bool {
 	for _, a := range source {
 		if a == target {
@@ -226,6 +260,7 @@ func getDecision(cache *ttl_map.Heap, clientIP string) (bool, error) {
 
 func setDecision(cache *ttl_map.Heap, clientIP string, isBanned bool, duration int64) {
 	if isBanned {
+		log.Printf("%v banned", clientIP)
 		cache.Set(clientIP, cacheBannedValue, duration)
 	} else {
 		cache.Set(clientIP, cacheNoBannedValue, duration)
@@ -237,10 +272,10 @@ func handleNoStreamCache(a *Bouncer, rw http.ResponseWriter, req *http.Request,
 	routeURL := url.URL{
 		Scheme:   a.crowdsecScheme,
 		Host:     a.crowdsecHost,
-		Path:     crowdsecRoute,
+		Path:     crowdsecLapiRoute,
 		RawQuery: fmt.Sprintf("ip=%v&banned=true", remoteHost),
 	}
-	body := crowdsecQuery(a, routeURL.String())
+	body := crowdsecQuery(a, routeURL.String(), false)
 
 	if bytes.Equal(body, []byte("null")) {
 		if a.crowdsecMode == "live" {
@@ -278,13 +313,22 @@ func handleStreamCache(a *Bouncer, initialized bool) {
 	time.AfterFunc(time.Duration(a.updateInterval)*time.Second, func() {
 		handleStreamCache(a, false)
 	})
+	var rawQuery string
+	var path string
+	if a.crowdsecMode == "alone" {
+		rawQuery = ""
+		path = crowdsecCapiDecisions
+	} else {
+		rawQuery = fmt.Sprintf("startup=%t", initialized)
+		path = crowdsecLapiStreamRoute
+	}
 	streamRouteURL := url.URL{
 		Scheme:   a.crowdsecScheme,
 		Host:     a.crowdsecHost,
-		Path:     crowdsecStreamRoute,
+		Path:     path,
-		RawQuery: fmt.Sprintf("startup=%t", initialized),
+		RawQuery: rawQuery,
 	}
-	body := crowdsecQuery(a, streamRouteURL.String())
+	body := crowdsecQuery(a, streamRouteURL.String(), false)
 	var stream Stream
 	err := json.Unmarshal(body, &stream)
 	if err != nil {
@@ -304,16 +348,59 @@ func handleStreamCache(a *Bouncer, initialized bool) {
 	a.crowdsecStreamHealthy = true
 }
 
-func crowdsecQuery(a *Bouncer, stringURL string) []byte {
+func getToken(a *Bouncer) {
-	req, _ := http.NewRequest(http.MethodGet, stringURL, nil)
+	loginURL := url.URL{
-	req.Header.Add(crowdsecAuthHeader, a.crowdsecKey)
+		Scheme: a.crowdsecScheme,
+		Host:   a.crowdsecHost,
+		Path:   crowdsecCapiLogin,
+	}
+	body := crowdsecQuery(a, loginURL.String(), true)
+	var login Login
+	err := json.Unmarshal(body, &login)
+	if err != nil {
+		log.Printf("error while parsing body: %s", err)
+		a.crowdsecStreamHealthy = false
+		return
+	}
+	if login.Code == 200 && len(login.Token) > 0 {
+		a.crowdsecKey = login.Token
+	}
+}
+
+func crowdsecQuery(a *Bouncer, stringURL string, isPost bool) []byte {
+	var req *http.Request
+	if isPost {
+		data := []byte(fmt.Sprintf(
+			`{"machine_id": "%v","password": "%v","scenarios": ["%v"]}`,
+			a.crowdsecLogin,
+			a.crowdsecPwd,
+			strings.Join(a.crowdsecScenarios, `","`),
+		))
+		req, _ = http.NewRequest(http.MethodPost, stringURL, bytes.NewBuffer(data))
+	} else {
+		req, _ = http.NewRequest(http.MethodGet, stringURL, nil)
+	}
+	if a.crowdsecMode == "alone" {
+		req.Header.Add(crowdsecCapiHeader, a.crowdsecKey)
+	} else {
+		req.Header.Add(crowdsecLapiHeader, a.crowdsecKey)
+	}
 	res, err := a.client.Do(req)
 	if err != nil {
 		log.Printf("error while fetching %v: %s", stringURL, err)
 		a.crowdsecStreamHealthy = false
 		return nil
 	}
-	if res.StatusCode == http.StatusForbidden {
+	if res.StatusCode == http.StatusUnauthorized && a.crowdsecMode == "alone" {
+		oldToken := a.crowdsecKey
+		getToken(a)
+		if oldToken == a.crowdsecKey {
+			a.crowdsecStreamHealthy = false
+			return nil
+		}
+		return crowdsecQuery(a, stringURL, false)
+	}
+	if res.StatusCode != http.StatusOK {
 		log.Printf("error while fetching %v, status code: %d", stringURL, res.StatusCode)
 		a.crowdsecStreamHealthy = false
 		return nil

docker-compose.local.yml

@@ -34,6 +34,9 @@ services:
       - "traefik.http.routers.whoami.middlewares=crowdsec@docker"
       - "traefik.http.middlewares.crowdsec.plugin.bouncer.enabled=true"
       - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey=40796d93c2958f9e58345514e67740e5"
+      - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapilogin=3829a6c9870e4726a377d8951ebb64a1m8psGvMaq1ykJ3zX"
+      - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapipwd=Q5pgN8bRNInHGdx6QCksdPOJVLeLQ7ipJntSeuP3r8088zXzRVs4G8liXAKfI1k6"
+      - "traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseccapiscenarios=crowdsecurity/http-backdoors-attempts,baudneo/zoneminder-bf"
 
   crowdsec:
     image: crowdsecurity/crowdsec:v1.4.1