📝 Add documentation exemple 4 for vm binary usage (#45)

* 📝 Add documentation exemple 4 for vm binary usage

* :hammer:Update binary version

* 🔨 update

* 🔨 update

* 🚧 Working Crowdsec with tls auth

* 🐛 Add changes to download the plugin from the service

* 🔧 Add config middle for https

* 🔧 Update config for the exemple

* 🔧 Add conf for certs, working example

* 📝 Add doc for binary vm Readme

* 📝 update documentation for exemple and make

* 🚨 Fix lint

* 🚨 Fix Lint End of File
This commit is contained in:
mathieuHa
2022-12-04 15:36:15 +01:00
committed by GitHub
parent f69faaf66c
commit c29d8a20d3
24 changed files with 679 additions and 0 deletions

1
.gitignore vendored
View File

@@ -1,5 +1,6 @@
.idea/
.DS_Store
./config/
conf
db
logs

View File

@@ -34,6 +34,9 @@ run_cacheredis:
run_trustedips:
docker-compose -f exemples/trusted-ips/docker-compose.trusted.yml up -d --remove-orphans
run_binaryvm:
cd exemples/binary-vm/ && sudo vagrant up
run_tlsauth:
docker-compose -f exemples/tls-auth/docker-compose.tls-auth.yml down && docker-compose -f exemples/tls-auth/docker-compose.tls-auth.yml up -d && docker-compose -f exemples/tls-auth/docker-compose.tls-auth.yml restart && docker-compose -f exemples/tls-auth/docker-compose.tls-auth.yml logs -f
@@ -78,6 +81,10 @@ clean_all_docker:
docker-compose -f docker-compose.local.yml down --remove-orphans
docker-compose -f docker-compose.yml down --remove-orphans
clean_vagrant:
cd exemples/binary-vm/ && sudo vagrant destroy -f
show_metrics:
docker exec crowdsec cscli metrics

View File

@@ -387,6 +387,15 @@ To play the demo environment run:
make run_trustedips
```
4. Using Crowdsec and Traefik installed as binary in a single VM
Please see details in `exemples/binary-vm/README.md`
To play the demo environment run:
```bash
make run_binaryvm
```
5. Using https communication and tls authentication with Crowdsec
##### Summary

1
exemples/binary-vm/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
.vagrant/

View File

@@ -0,0 +1,60 @@
### Install vagrant
##### On linux
```bash
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.asc
echo "deb [ signed-by=/usr/share/keyrings/hashicorp-archive-keyring.asc ] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install vagrant
```
### Install libvirt
```bash
sudo apt install -y qemu-kvm virt-manager libvirt-daemon-system virtinst libvirt-clients bridge-utils
sudo systemctl enable --now libvirtd
sudo systemctl start libvirtd
sudo usermod -aG kvm $USER
sudo usermod -aG libvirt $USER
```
### Install the plugin vagrant-libvirt
```bash
vagrant plugin install vagrant-libvirt
```
#### Start the VM
```bash
sudo vagrant up --provider=libvirt
```
#### Destroy the VM
```bash
sudo vagrant destroy -f
```
#### SSH in the VM
```bash
sudo vagrant ssh
```
### Context
Traefik is installed as a systemd service.
It is configured with the dashboard activated and listening on port 8081 and port 80 for the web
Crowdsec is started and listening on port 8080.
Certificates are generated on the provision step of vagrant.
Whoami is installed as a systemd service.
It is configured to listen on port 9000.
Whoami is accessible from traefik on port 80 at any domain and path
For example: curl http://localhost:80/test
The Plugin / Bouncer use certificates to validate the server certificates and authenticates with the Crowdsec local api.

81
exemples/binary-vm/Vagrantfile vendored Normal file
View File

@@ -0,0 +1,81 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Every Vagrant development environment requires a box. You can search for
# boxes at https://vagrantcloud.com/search.
config.vm.box = "generic/debian11"
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# NOTE: This will enable public access to the opened port
# config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine and only allow access
# via 127.0.0.1 to disable public access
# config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"
config.vm.network "forwarded_port", guest: 8081, host: 8081
config.vm.network "forwarded_port", guest: 80, host: 80
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
config.vm.provider "libvirt" do |lv|
graphics_type = "none"
lv.cpus = 1
# lv.vm.network :private_network, :ip => "10.20.30.40"
# Customize the amount of memory on the VM:
lv.memory = "2048"
end
# config.vm.provider "virtualbox" do |vb|
# vb.gui = false
# vb.cpus = 1
# # vb.vm.network :private_network, :ip => "10.20.30.40"
# # Customize the amount of memory on the VM:
# vb.memory = "2048"
# end
config.vm.provision "file", source: "./files", destination: "/home/vagrant/vagrant_data"
config.vm.provision "shell", path: "scripts/install_traefik.sh"
config.vm.provision "shell", path: "scripts/configure_traefik.sh"
config.vm.provision "shell", path: "scripts/install_whoami.sh"
config.vm.provision "shell", path: "scripts/install_crowdsec.sh"
config.vm.provision "shell", path: "scripts/configure_crowdsec_certs.sh"
end

View File

@@ -0,0 +1,4 @@
filenames:
- /var/log/traefik/access.log
labels:
type: traefik

View File

@@ -0,0 +1,16 @@
{
"CN": "myagent",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "agent-ou",
"ST": "France"
}
]
}

View File

@@ -0,0 +1,17 @@
{
"CN": "whoami",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "bouncer-ou",
"ST": "France"
}
]
}

View File

@@ -0,0 +1,16 @@
{
"CN": "CrowdSec Test CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec",
"ST": "France"
}
]
}

View File

@@ -0,0 +1,19 @@
{
"CN": "CrowdSec Test CA Intermediate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec Intermediate",
"ST": "France"
}
],
"ca": {
"expiry": "42720h"
}
}

View File

@@ -0,0 +1,44 @@
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"intermediate_ca": {
"usages": [
"signing",
"digital signature",
"key encipherment",
"cert sign",
"crl sign",
"server auth",
"client auth"
],
"expiry": "8760h",
"ca_constraint": {
"is_ca": true,
"max_path_len": 0,
"max_path_len_zero": true
}
},
"server": {
"usages": [
"signing",
"digital signing",
"key encipherment",
"server auth"
],
"expiry": "8760h"
},
"client": {
"usages": [
"signing",
"digital signature",
"key encipherment",
"client auth"
],
"expiry": "8760h"
}
}
}
}

View File

@@ -0,0 +1,20 @@
{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec Server",
"ST": "France"
}
],
"hosts": [
"127.0.0.1",
"localhost"
]
}

View File

@@ -0,0 +1,62 @@
common:
daemonize: true
pid_dir: /var/run/
log_media: file
log_level: debug
log_dir: /var/log/
log_max_size: 20
compress_logs: true
log_max_files: 10
working_dir: .
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data/
simulation_path: /etc/crowdsec/simulation.yaml
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /usr/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
acquisition_dir: /etc/crowdsec/acquis.d
parser_routines: 1
cscli:
output: human
color: auto
db_config:
log_level: info
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
#max_open_conns: 100
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
plugin_config:
user: nobody # plugin process would be ran on behalf of this user
group: nogroup # plugin process would be ran on behalf of this group
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
log_level: debug
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
console_path: /etc/crowdsec/console.yaml
online_client: # Central API credentials (to push signals and receive bad IPs)
credentials_path: /etc/crowdsec/online_api_credentials.yaml
trusted_ips: # IP ranges, or IPs which can have admin API access
- 127.0.0.1
tls:
cert_file: /etc/crowdsec/certs/server.pem #Server side cert
key_file: /etc/crowdsec/certs/server-key.pem #Server side key
ca_cert_path: /etc/crowdsec/certs/inter.pem #CA used to verify the client certs
bouncers_allowed_ou: #OU allowed for bouncers
- bouncer-ou
agents_allowed_ou: #OU allowed for agents
- agent-ou

View File

@@ -0,0 +1,4 @@
url: https://localhost:8080
ca_cert_path: /etc/crowdsec/certs/inter.pem #CA to trust the server certificate
key_path: /etc/crowdsec/certs/agent-key.pem #Client key
cert_path: /etc/crowdsec/certs/agent.pem #Client cert

View File

@@ -0,0 +1,31 @@
http:
routers:
to-whoami-http-service:
rule: "PathPrefix(`/`)"
service: whoami-service
middlewares:
- "crowdsec-whoami"
entryPoints:
- web
services:
whoami-service:
loadBalancer:
servers:
- url: "http://localhost:9000/"
middlewares:
crowdsec-whoami:
plugin:
bouncer:
enabled: true
crowdseclapikey: whoami-demo
updateintervalseconds: 60
crowdsecmode: live
loglevel: "DEBUG"
crowdsecLapiScheme: https
crowdsecLapiHost: localhost:8080
crowdsecLapiTLSInsecureVerify: false
crowdsecLapiTLSCertificateAuthorityFile: /etc/traefik/crowdsec-certs/inter.pem
crowdsecLapiTLSCertificateBouncerFile: /etc/traefik/crowdsec-certs/bouncer.pem
crowdsecLapiTLSCertificateBouncerKeyFile: /etc/traefik/crowdsec-certs/bouncer-key.pem

View File

@@ -0,0 +1,40 @@
[Unit]
Description=traefik proxy
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
[Service]
Restart=on-abnormal
; User and group the process will run as.
User=traefik
Group=traefik
; Always set "-root" to something safe in case it gets forgotten in the traefikfile.
ExecStart=/usr/local/bin/traefik --configfile=/etc/traefik/traefik.yml
WorkingDirectory=/etc/traefik
; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Use private /tmp and /var/tmp, which are discarded after traefik stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=false
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
;ProtectSystem=full
; … except /etc/ssl/traefik, because we want Letsencrypt-certificates there.
; This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
;ReadWriteDirectories=/etc/traefik/acme
;ReadWriteDirectories=/etc/traefik/plugins-storage
; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by traefik. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,100 @@
################################################################
# Global configuration
################################################################
global:
checkNewVersion: false
sendAnonymousUsage: false
################################################################
# EntryPoints configuration
################################################################
entryPoints:
web:
address: :80
traefik:
address: :8081
################################################################
# Provider file configuration
################################################################
providers:
file:
directory: "/etc/traefik/conf"
################################################################
# Plugin configuration
################################################################
experimental:
plugins:
bouncer:
moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
version: v1.1.5
################################################################
# Certificate Resolver
################################################################
serversTransport:
insecureSkipVerify: false
################################################################
# Traefik logs configuration
################################################################
# Traefik logs
# Enabled by default and log to stdout
log:
filePath: /var/log/traefik/traefik.log
level: DEBUG
# format: json
################################################################
# Access logs configuration
################################################################
# Enable access logs
# By default it will write to stdout and produce logs in the textual
# Common Log Format (CLF), extended with additional fields.
accessLog:
# Sets the file path for the access log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
filePath: /var/log/traefik/access.log
fields:
defaultMode: keep
names:
ClientUsername: keep
headers:
defaultMode: keep
# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
# format: json
################################################################
# API and dashboard configuration
################################################################
# Enable API and dashboard
#
# Optional
#
api:
# Enable the API in insecure mode
#
# Optional
# Default: false
#
insecure: true
# Enabled Dashboard
#
# Optional
# Default: true
#
dashboard: true

View File

@@ -0,0 +1,31 @@
[Unit]
Description=whoami web server
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
[Service]
Restart=on-abnormal
; User and group the process will run as.
User=whoami
Group=whoami
; Always set "-root" to something safe in case it gets forgotten in the traefikfile.
ExecStart=/usr/local/bin/whoami --port=9000
; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Use private /tmp and /var/tmp, which are discarded after traefik stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=false
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
NoNewPrivileges=true
[Install]
WantedBy=multi-user.

View File

@@ -0,0 +1,46 @@
#!/bin/bash
basepath="/home/vagrant/vagrant_data/crowdsec/certs"
VERSION=$(curl --silent "https://api.github.com/repos/cloudflare/cfssl/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
VNUMBER=${VERSION#"v"}
wget https://github.com/cloudflare/cfssl/releases/download/${VERSION}/cfssl_${VNUMBER}_linux_amd64 -O cfssl
chmod +x cfssl
sudo mv cfssl /usr/local/bin
VERSION=$(curl --silent "https://api.github.com/repos/cloudflare/cfssl/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
VNUMBER=${VERSION#"v"}
wget https://github.com/cloudflare/cfssl/releases/download/${VERSION}/cfssljson_${VNUMBER}_linux_amd64 -O cfssljson
chmod +x cfssljson
sudo mv cfssljson /usr/local/bin
cfssljson -version
mkdir -p /etc/crowdsec/certs
# Generate the CA
cfssl gencert --initca ${basepath}/ca.json 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/ca"
# Generate an intermediate certificate that will be used to sign the client certificates
cfssl gencert --initca ${basepath}/intermediate.json 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/inter"
cfssl sign -ca "/etc/crowdsec/certs/ca.pem" -ca-key "/etc/crowdsec/certs/ca-key.pem" -config ${basepath}/profiles.json -profile intermediate_ca "/etc/crowdsec/certs/inter.csr" 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/inter"
# Generate a server side certificate
cfssl gencert -ca "/etc/crowdsec/certs/inter.pem" -ca-key "/etc/crowdsec/certs/inter-key.pem" -config ${basepath}/profiles.json -profile=server ${basepath}/server.json 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/server"
# Generate a client certificate for the bouncer whoami
cfssl gencert -ca "/etc/crowdsec/certs/inter.pem" -ca-key "/etc/crowdsec/certs/inter-key.pem" -config ${basepath}/profiles.json -profile=client ${basepath}/bouncer.json 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/bouncer"
# Generate a client certificate for the agent
cfssl gencert -ca "/etc/crowdsec/certs/inter.pem" -ca-key "/etc/crowdsec/certs/inter-key.pem" -config ${basepath}/profiles.json -profile=client ${basepath}/agent.json 2>/dev/null | cfssljson --bare "/etc/crowdsec/certs/agent"
cp /home/vagrant/vagrant_data/crowdsec/config/config.yaml /etc/crowdsec/config.yaml.local
cp /home/vagrant/vagrant_data/crowdsec/config/local_api_credentials.yaml /etc/crowdsec/
chmod +r /etc/crowdsec/config.yaml
chmod +r /etc/crowdsec/local_api_credentials.yaml
systemctl restart crowdsec
mkdir /etc/traefik/crowdsec-certs
cp /etc/crowdsec/certs/inter.pem /etc/traefik/crowdsec-certs/inter.pem
cp /etc/crowdsec/certs/bouncer.pem /etc/traefik/crowdsec-certs/bouncer.pem
cp /etc/crowdsec/certs/bouncer-key.pem /etc/traefik/crowdsec-certs/bouncer-key.pem
chown -R traefik:traefik /etc/traefik/crowdsec-certs/
systemctl restart traefik

View File

@@ -0,0 +1,11 @@
#!/bin/bash
sudo cp /home/vagrant/vagrant_data/traefik/traefik.yml /etc/traefik/traefik.yml
sudo cp -a /home/vagrant/vagrant_data/traefik/conf /etc/traefik/
sudo chown -R traefik:traefik /etc/traefik
sudo mkdir /var/log/traefik
sudo chown -R traefik:traefik /var/log/traefik
sudo systemctl restart traefik.service
sudo systemctl status traefik.service

View File

@@ -0,0 +1,5 @@
#!/bin/bash
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt install crowdsec -y
sudo cp /home/vagrant/vagrant_data/crowdsec/acquis.yaml /etc/crowdsec/acquis.yaml

View File

@@ -0,0 +1,31 @@
#!/bin/bash
sudo apt-get update && apt-get install wget -y && apt-get upgrade -y
wget -O traefik.tar.gz "https://github.com/traefik/traefik/releases/download/v2.9.5/traefik_v2.9.5_linux_amd64.tar.gz"
tar -zxvf traefik.tar.gz
# inspired from https://gist.github.com/ubergesundheit/7c9d875befc2d7bfd0bf43d8b3862d85
sudo mv ./traefik /usr/local/bin/
sudo chown root:root /usr/local/bin/traefik
sudo chmod 755 /usr/local/bin/traefik
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/traefik
sudo groupadd -g 321 traefik
sudo useradd \
-g traefik --no-user-group \
--home-dir /var/www --no-create-home \
--shell /usr/sbin/nologin \
--system --uid 321 traefik
sudo mkdir /etc/traefik
sudo mkdir /etc/traefik/acme
sudo mkdir /etc/traefik/plugins-storage
sudo chown -R root:root /etc/traefik
sudo chown -R traefik:traefik /etc/traefik/acme
sudo chown -R traefik:traefik /etc/traefik/plugins-storage
sudo cp /home/vagrant/vagrant_data/traefik/traefik.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/traefik.service
sudo chmod 644 /etc/systemd/system/traefik.service
sudo systemctl daemon-reload
sudo systemctl start traefik.service
sudo systemctl enable traefik.service

View File

@@ -0,0 +1,23 @@
#!/bin/bash
sudo apt-get update && apt-get install wget -y
wget -O whoami.tar.gz "https://github.com/traefik/whoami/releases/download/v1.8.7/whoami_v1.8.7_linux_amd64.tar.gz"
tar -zxvf whoami.tar.gz
# inspired from https://gist.github.com/ubergesundheit/7c9d875befc2d7bfd0bf43d8b3862d85
sudo mv ./whoami /usr/local/bin/
sudo chown root:root /usr/local/bin/whoami
sudo chmod 755 /usr/local/bin/whoami
sudo groupadd -g 322 whoami
sudo useradd \
-g whoami --no-user-group \
--home-dir /var/www --no-create-home \
--shell /usr/sbin/nologin \
--system --uid 322 whoami
sudo cp /home/vagrant/vagrant_data/whoami.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/whoami.service
sudo chmod 644 /etc/systemd/system/whoami.service
sudo systemctl daemon-reload
sudo systemctl start whoami.service
sudo systemctl enable whoami.service