articles
Build your own VPN node with Traefik v2, MTProto Proxy, WireGuard and BIRD 2.0 / Part 2
Today plenty of people already know what VPN 1 is and why they need to use it. Many ISPs have the same knowledge and need to block VPNs in accordance with country laws. In this article we build uncommon installation for you own private VPN node.
Publish Date
- 2021-04-25
Server Side Configurations
Please see Part 1 2 for overall solution overview.
Note: We will not attempt to build SD-WAN 3 through this series of articles, nor will we implement any software controller for management plane. These articles describe topology contained only one VPN host communicating with two Linux-based CPEs 4. But this doesn’t mean it cannot be scaled horizontally.
The Docker Compose 5 file docker-compose.yaml. Please see VPN box configurations in repository 6 for more information:
version: "2.4"
networks:
1frontend:
name: 1frontend
driver: bridge
ipam:
config:
- subnet: 192.168.100.0/24
gateway: 192.168.100.1
services:
traefik:
image: traefik:latest
container_name: traefik
command:
- --entrypoints.http.address=:80/tcp
- --entrypoints.https.address=:443/tcp
- --entrypoints.wireguard.address=:443/udp
- --providers.docker
- --api
- --log=true
- --log.level=${TRAEFIK_LOG_LEVEL}
- --certificatesresolvers.leresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
- --certificatesresolvers.leresolver.acme.email=${TRAEFIK_ACME_MAIL}
- --certificatesresolvers.leresolver.acme.storage=/acme.json
- --certificatesresolvers.leresolver.acme.tlschallenge=true
ports:
- 80:80/tcp
- 443:443/tcp
- 443:443/udp
networks:
- 1frontend
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /docker/traefik/acme.json:/acme.json
- /docker/traefik/htpasswd:/.htpasswd
labels:
# Dashboard
- traefik.http.routers.traefik.rule=Host(`${TRAEFIK_DASHBOARD_HOST}`)
- traefik.http.routers.traefik.service=api@internal
- traefik.http.routers.traefik.tls.certresolver=leresolver
- traefik.http.routers.traefik.entrypoints=https
- traefik.http.routers.traefik.middlewares=authtraefik
- traefik.http.middlewares.authtraefik.basicauth.usersfile=/.htpasswd
# global redirect to https
- traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)
- traefik.http.routers.http-catchall.entrypoints=http
- traefik.http.routers.http-catchall.middlewares=redirect-to-https
# middleware redirect
- traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
- traefik.docker.network=1frontend
wireguard:
container_name: wireguard
build:
context: wireguard
dockerfile: /docker/wireguard/build/Dockerfile
cap_add:
- NET_ADMIN
- SYS_MODULE
environment:
- PUID=${PUID}
- PGID=${PGID}
- TZ=${TIMEZONE}
networks:
- 1frontend
volumes:
- /docker/wireguard/periodic-config:/etc/periodic
- /docker/wireguard/wireguard-config:/etc/wireguard
- /docker/wireguard/bird-config:/etc/bird
sysctls:
- net.ipv4.conf.all.src_valid_mark=1
- net.ipv4.ip_forward=1
- net.ipv4.tcp_congestion_control=bbr
labels:
- traefik.enable=true
- traefik.udp.routers.wireguard.entrypoints=wireguard
- traefik.udp.services.wireguard.loadbalancer.server.port=51820
restart: unless-stopped
depends_on:
- traefik
mtg:
image: nineseconds/mtg
container_name: mtg
networks:
- 1frontend
volumes:
- /docker/mtg/mtg-config/config.toml:/config.toml
labels:
- traefik.enable=true
- traefik.tcp.routers.mtg.entrypoints=https
- traefik.tcp.routers.mtg.rule=HostSNI(`${MTG_TLS_HOST}`)
- traefik.tcp.routers.mtg.tls.passthrough=true
- traefik.tcp.services.mtg.loadbalancer.server.port=3128
restart: unless-stopped
depends_on:
- traefik
webhost:
image: nginx
container_name: webhost
networks:
- 1frontend
volumes:
- /docker/webhost/html:/usr/share/nginx/html:ro
labels:
- traefik.enable=true
- traefik.http.routers.webhost.entrypoints=https
- traefik.http.routers.webhost.rule=Host(`${WEBHOST_TLS_HOST}`)
- traefik.http.routers.webhost.tls=true
- traefik.http.routers.webhost.tls.certresolver=leresolver
restart: unless-stopped
depends_on:
- traefik
It creates independent network named 1frontend with CIDR 7 192.168.100.0/24, then Traefik container will appear first using this network as default for connections to all other containers. There are defined several major things like endpoints:
ACME 9 settings to make Let’s Encrypt 10 TLS certificates on-the-fly and Basic Authentication 11 configuration to protect Traefik dashboard.
The WireGuard container comprising several special features:
Overall, target is the following directories structure:
root@host:~# tree -a /docker
/docker
├── docker-compose.yaml
├── .env
├── mtg
│ └── mtg-config
│ └── template.toml
├── traefik
│ ├── acme.json
│ └── htpasswd
├── webhost
│ └── html
│ ├── index.html
│ └── ntwrk.png
└── wireguard
├── bird-config
│ ├── bird.conf
│ ├── exclusions.conf
│ ├── generated.conf
│ └── manual.conf
├── build
│ ├── Dockerfile
│ └── supervisord.conf
├── periodic-config
│ ├── 15min
│ │ └── make_routes
│ ├── daily
│ ├── hourly
│ ├── monthly
│ └── weekly
└── wireguard-config
├── bootstrap.sh
├── configurations
│ ├── peer1.conf
│ ├── peer2.conf
│ ├── publickey-server.wg0
│ └── server.conf
└── templates
├── peer.wg0.conf
└── server.wg0.conf
Environment file .env
Environment file .env is used to keep variables for Docker Compose file.
# Common
PUID=1001
PGID=1001
UMASK_SET=0022
TIMEZONE=Europe/Moscow
# Container Specific
TRAEFIK_LOG_LEVEL=INFO
TRAEFIK_ACME_MAIL=acme@ntwrk.today
TRAEFIK_DASHBOARD_HOST=dashboard.ntwrk.today
MTG_TLS_HOST=duckduckgo.com
WEBHOST_TLS_HOST=webhost.ntwrk.today
Traefik
Traefik Dashboard will be available at URI defined in .env file. The htpasswd file contains Basic Authentication credentials to protect Traefik Dashboard, the format is compatible with htpasswd 14 tool. In this guide we use ntwrk:_ntVVrk$T0d4Y that becomes:
root@host:~# cat /docker/traefik/htpasswd
ntwrk:$apr1$24xr5mbt$YdPRnBWP1oUQWyLq/BfS4.
Do not forget to use CAA 15 record for domain with ACME mail:
root@host:~# host -t caa ntwrk.today
ntwrk.today has CAA record 0 issuewild "letsencrypt.org"
ntwrk.today has CAA record 0 issue "letsencrypt.org"
ntwrk.today has CAA record 0 iodef "mailto:acme@ntwrk.today"
WireGuard Server
Let’s drill down to WireGuard files and review the Dockerfile first in build directory:
FROM alpine:edge
COPY build/supervisord.conf /etc/supervisord.conf
# You can get parser's source from https://github.com/unsacrificed/network-list-parser/
RUN echo 'https://dl-cdn.alpinelinux.org/alpine/edge/testing' >> /etc/apk/repositories && \
apk --update --no-cache add git wireguard-tools bird py3-setuptools supervisor && \
wget https://github.com/unsacrificed/network-list-parser/releases/download/v1.2/network-list-parser-linux-amd64-1.2.bin \
-O /usr/local/bin/parser && \
rm -f /etc/bird.conf && \
chmod a+x /usr/local/bin/parser && \
mkdir /etc/bird/
VOLUME /etc/periodic/
VOLUME /etc/bird
VOLUME /etc/wireguard/
EXPOSE 51820/udp
CMD ["supervisord","-c","/etc/supervisord.conf"]
The Alpine Linux 16 used to build this image, pushing supervisord.conf and downloading special parser 17 binary, it will also contain
- git
- wireguard tools
- bird
- supervisor and py3-setuptools
artifacts, 3 volumes, exposed 51820/UDP port and, finally, supervisor 18 daemon. As it was mentioned earlier, our image carries 3 services defined in supervisord.conf:
[supervisord]
nodaemon=true
user=root
# One-shot generate WireGuard configuration
# and start WireGuard server
[program:wireguard]
command=sh /etc/wireguard/bootstrap.sh
startsecs=0
autostart=true
autorestart=false
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes = 0
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes = 0
# Start BIRD 2.0 daemon
[program:bird]
command=bird -c /etc/bird/bird.conf
autorestart=true
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes = 0
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes = 0
# Start Crond daemon
[program:crond]
command=crond -l 2 -f
autostart=true
autorestart=true
stderr_logfile=/dev/stdout
stderr_logfile_maxbytes = 0
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes = 0
To create initial configuration for WireGuard it’s possible to use ephemeral container with mounted volume from configurations directory:
root@host:~# docker run --rm -ti -v /docker/wireguard/wireguard-config/configurations/:/mnt alpine:edge ash
Add wireguard-tools package:
/ # apk --update add wireguard-tools
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/20) Installing wireguard-tools-wg (1.0.20210223-r0)
... omitted for brevity ...
(20/20) Installing wireguard-tools (1.0.20210223-r0)
Executing busybox-1.31.1-r21.trigger
OK: 14 MiB in 34 packages
Alter the umask temporarily to ensure that access is only restricted to the owner. Then run generation of Private and Public keys:
/ # umask 077
/ # wg genkey | tee /mnt/server.conf | wg pubkey > /mnt/publickey-server.wg0
/ # ls -la /mnt/
total 16
drwxr-xr-x 2 root root 4096 Apr 18 15:39 .
drwxr-xr-x 1 root root 4096 Apr 18 15:38 ..
-rw------- 1 root root 45 Apr 18 15:39 publickey-server.wg0
-rw------- 1 root root 45 Apr 18 15:39 server.conf
exit will auto-remove ephemeral container:
/ # exit
Check files exist in configurations directory on host:
root@host:~# ls /docker/wireguard/wireguard-config/configurations/
publickey-server.wg0 server.conf
root@host:~# cat /docker/wireguard/wireguard-config/configurations/server.conf
+AgasB8xc0Ip0upAp/MdC+YFaczM3VD5t4hZSoEek2c=
Modify server.conf to make it compatible with simple templating engine used in our image:
root@host:~# sed -i '1 s/^/Server Configuration\n/' /docker/wireguard/wireguard-config/configurations/server.conf
root@host:~# echo 10.10.1.1/24 >> /docker/wireguard/wireguard-config/configurations/server.conf
root@host:~# cat /docker/wireguard/wireguard-config/configurations/server.conf
Server Configuration
+AgasB8xc0Ip0upAp/MdC+YFaczM3VD5t4hZSoEek2c=
10.10.1.1/24
The 1st line represents a comment, 2nd one is Private Key of WireGuard server, 3rd is for Address uses for VPN overlay on server side. Files peer1.conf and peer2.conf look exactly the same you expected, for example:
cpe1.ntwrk.today
tzIgY54n0Bx+1rIV2D/M8rbZyIxL4dTyYnto+J3U8Cg=
10.10.1.2/32, 192.168.101.0/24
cpe2.ntwrk.today
PhHXCXEos//WLpEisW4vBmwUuUOICiEInvDMjtnHTRs=
10.10.1.3/32, 192.168.102.0/24
The 1st line represents comment as well, 2nd is Public Key of the WireGuard client, 3rd is for AllowedIPs of client: the tunnel endpoint and the routed network behind it. Please recall topology diagram from Part 1 2.
During container startup the templating engine hidden by bootstrap.sh reads files in configurations directory and builds temporary wg0.conf for WireGuard server.
BIRD 2.0
Now we’re ready to review BIRD 2.0 configuration. Main configuration bird.conf file:
log syslog all;
ipv4 table master4;
ipv6 table master6;
protocol device { }
protocol kernel kernel4 {
ipv4 {
import none;
export none;
};
}
protocol kernel kernel6 {
ipv6 {
import none;
export none;
};
}
### Filters ###
protocol static static_bgp {
ipv4;
include "/etc/bird/manual.conf";
include "/etc/bird/generated.conf";
}
### BGP Templates ###
template bgp branch_Peer {
bfd;
connect retry time 10;
startup hold time 30;
hold time 60;
graceful restart;
router id 10.10.1.1;
local as 65001;
ipv4 {
import all;
export all;
};
}
template bgp roadwarrior_Peer {
connect retry time 10;
startup hold time 30;
hold time 60;
router id 10.10.1.1;
local as 65001;
ipv4 {
import all;
export all;
};
}
### Branch Peers ###
protocol bgp branch_Router_BGP_1 from branch_Peer {
neighbor 10.10.1.2 as 65002;
}
protocol bgp branch_Router_BGP_2 from branch_Peer {
neighbor 10.10.1.3 as 65003;
}
protocol bfd branch_Router_BFD {
neighbor 10.10.1.2;
neighbor 10.10.1.3;
}
### Road Warriors ###
protocol bgp roadwarrior_BGP_1 from roadwarrior_Peer {
neighbor 10.10.1.6 as 65006;
}
Templates for BGP 19 peers at branch premises and road warrior 20 BGP peers are defined there, it’s simplify the manner on how peers can be declared. The main difference, road warrior peers do not contain BFD 21 protocol due to the poor quality of channels used by mobile clients. Also IPv4 table will be populated by static files with prefixes. The static file manual.conf filled manually for some specific time-invariant prefixes, for example:
### Linkedin
route 108.174.0.0/20 via "wg0";
route 185.63.144.0/22 via "wg0";
route 13.64.0.0/11 via "wg0";
route 13.96.0.0/13 via "wg0";
route 13.104.0.0/14 via "wg0";
The generated.conf file contains the same prefixes format but refreshes every 15 minutes by abovementioned make_routes script you noted in the directories structure:
#!/usr/bin/env sh
echo "Running make_routes script"
rm -rf /tmp/z-i
git clone --depth=1 https://github.com/zapret-info/z-i.git /tmp/z-i
# You can get parser's source from https://github.com/unsacrificed/network-list-parser/
echo "Generating prefixes"
parser -src-file /tmp/z-i/dump.csv -prefix 'route ' -suffix ' via "wg0";' 2>/dev/null > /etc/bird/generated.conf
echo "Excluding certain prefixes"
while read line;
do
sed -i "/$line/d" /etc/bird/generated.conf;
done < /etc/bird/exclusions.conf
echo "Reloading BIRD"
birdc configure
rm -rf /tmp/z-i
Script creates a list of prefixes collected from Roskomnadzor blacklists 22 compiled into the Register of Internet Addresses filtered in Russian Federation 23 repository.
The exclusions.conf file is required to forcibly remove lines from generated.conf file and exclude such prefixes to be advertised to VPN overlay. Format example:
185.165.123
185.71.67
MTProto Proxy
MTProto 24 proxy is a useful piece in our solution and we chose the mtg 25 implementation. Generate new configuration with Fake TLS 26 secret to simulate duckduckgo.com 27:
root@host:~# mtg_secret=$(docker run --rm nineseconds/mtg generate-secret --hex duckduckgo.com); sed "s/__MTG_SECRET__/${mtg_secret}/" /docker/mtg/mtg-config/template.toml > /docker/mtg/mtg-config/config.toml
And put fronting domain name into MTG_TLS_HOST variable for .env file. Please pay attention on labels for mtg service in Docker Compose, you can find traefik.tcp.routers.mtg.tls.passthrough=true label which points Traefik not to terminate TLS session, this task will be done by mtg on its own, Traefik will route traffic based on SNI 28 field only.
Webhost
Nothing to mention here, simple stateless Web server, you can modify it on your own.
Run and Verify
It’s time to run entire stack:
root@host:/docker# docker-compose up -d
Creating network "1frontend" with driver "bridge"
Pulling traefik (traefik:latest)...
... omitted for brevity ...
Status: Downloaded newer image for traefik:latest
Building wireguard
Step 1/8 : FROM alpine:edge
edge: Pulling from library/alpine
... omitted for brevity ...
Status: Downloaded newer image for alpine:edge
---> b0da5d0678e7
Step 2/8 : COPY build/supervisord.conf /etc/supervisord.conf
---> 380a8326000a
Step 3/8 : RUN echo 'https://dl-cdn.alpinelinux.org/alpine/edge/testing' >> /etc/apk/repositories && apk --update --no-cache add git wireguard-tools bird py3-setuptools supervisor && wget https://github.com/unsacrificed/network-list-parser/releases/download/v1.2/network-list-parser-linux-amd64-1.2.bin -O /usr/local/bin/parser && rm -f /etc/bird.conf && chmod a+x /usr/local/bin/parser && mkdir /etc/bird/
---> Running in b98ba8d656b7
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
(1/43) Installing ncurses-terminfo-base (6.2_p20210418-r0)
... omitted for brevity ...
(43/43) Installing wireguard-tools (1.0.20210315-r0)
Executing busybox-1.33.0-r2.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 85 MiB in 57 packages
Connecting to github.com (140.82.114.3:443)
Connecting to github-releases.githubusercontent.com (185.199.109.154:443)
saving to '/usr/local/bin/parser'
parser 18% |****** | 441k 0:00:04 ETA
parser 100% |********************************| 2340k 0:00:00 ETA
'/usr/local/bin/parser' saved
Removing intermediate container b98ba8d656b7
---> babc2d07dffd
Step 4/8 : VOLUME /etc/periodic/
---> Running in 46a53f07ca4b
Removing intermediate container 46a53f07ca4b
---> ca0fba4d8936
Step 5/8 : VOLUME /etc/bird
---> Running in 89eafeb9b69f
Removing intermediate container 89eafeb9b69f
---> 9a41e203a241
Step 6/8 : VOLUME /etc/wireguard/
---> Running in 51c922497c62
Removing intermediate container 51c922497c62
---> 1337741ffc34
Step 7/8 : EXPOSE 51820/udp
---> Running in 6f65d8d4000d
Removing intermediate container 6f65d8d4000d
---> 15e4d0e21c34
Step 8/8 : CMD ["supervisord","-c","/etc/supervisord.conf"]
---> Running in 7ff10c3acf59
Removing intermediate container 7ff10c3acf59
---> 5559378ca728
Successfully built 5559378ca728
Successfully tagged docker_wireguard:latest
WARNING: Image for service wireguard was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling mtg (nineseconds/mtg:)...
... omitted for brevity ...
Status: Downloaded newer image for nineseconds/mtg:latest
Creating traefik ... done
Creating wireguard ... done
Creating mtg ... done
In a few seconds after we can check Traefik status by login into Traefik dashboard and verify services are up and running:
This information can also be collected through API 29 we pre-enabled in the Traefik configuration. The entry points:
curl -su 'ntwrk:_ntVVrk$T0d4Y' https://dashboard.ntwrk.today/api/entrypoints | jq '.[] | "\(.name) \(.address)"' -r | column -t
http :80/tcp
https :443/tcp
wireguard :443/udp
And routers table where the columns are Entry Points, TLS, Rule, Name, Status:
for proto in http tcp udp; do curl -su 'ntwrk:_ntVVrk$T0d4Y' https://dashboard.ntwrk.today/api/$proto/routers | jq '.[] | "\(.entryPoints[]) \(.tls) \(.rule) \(.name) \(.status)"' -r; done | column -t | sort -k4
http null hostregexp(`{host:.+}`) http-catchall@docker enabled
https {"passthrough":true} HostSNI(`duckduckgo.com`) mtg@docker enabled
https {"certResolver":"leresolver"} Host(`dashboard.ntwrk.today`) traefik@docker enabled
https {"certResolver":"leresolver"} Host(`webhost.ntwrk.today`) webhost@docker enabled
wireguard null null wireguard@docker enabled
We must also verify status of WireGuard and BIRD inside the wireguard container. Since CPEs are still not configured, the WireGuard peers are shown as not connected yet:
root@host:~# docker exec wireguard wg show
interface: wg0
public key: jyQ252wXhlkEpPdtHpC6DnTegKTxofapTENQyagJuhA=
private key: (hidden)
listening port: 51820
peer: tzIgY54n0Bx+1rIV2D/M8rbZyIxL4dTyYnto+J3U8Cg=
allowed ips: 10.10.1.2/32, 192.168.101.0/24
persistent keepalive: every 25 seconds
peer: PhHXCXEos//WLpEisW4vBmwUuUOICiEInvDMjtnHTRs=
allowed ips: 10.10.1.3/32, 192.168.102.0/24
persistent keepalive: every 25 seconds
Now checking the BIRD peers which are also not established due to peers are not reachable over the WireGuard tunnels:
root@host:~# docker exec wireguard birdc show proto
BIRD 2.0.8 ready.
Name Proto Table State Since Info
device1 Device --- up 14:29:02.682
kernel4 Kernel master4 up 14:29:02.682
kernel6 Kernel master6 up 14:29:02.682
static_bgp Static master4 up 14:29:02.682
branch_Router_BGP_1 BGP --- start 14:29:02.682 Connect Socket: Host is unreachable
branch_Router_BGP_2 BGP --- start 14:29:02.682 Connect Socket: Host is unreachable
branch_Router_BFD BFD --- up 14:29:02.682
roadwarrior_BGP_1 BGP --- start 14:29:02.682 Active Socket: Host is unreachable
The last thing we need to check is that routing for networks behind CPEs was automatically added once wg0 interface brought up:
root@main:~# docker exec wireguard ip route
default via 192.168.100.1 dev eth0
10.10.1.0/24 dev wg0 proto kernel scope link src 10.10.1.1
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.3
192.168.101.0/24 dev wg0 scope link
192.168.102.0/24 dev wg0 scope link
Part 3 explains CPEs and Device Endpoint configurations.
References
1. Virtual Private Network ↩
2. Build your own VPN node with Traefik v2, MTProto Proxy, WireGuard and BIRD 2.0 / Part 1 ↩
3. SD-WAN ↩
4. Customer Premises Equipment ↩
5. Docker Compose ↩
6. VPN box repository ↩
7. Classless Inter-Domain Routing ↩
8. WireGuard: fast, modern, secure VPN tunnel ↩
9. Automatic Certificate Management Environment ↩
10. Let’s Encrypt: free, automated, and open certificate authority ↩
11. The Basic HTTP Authentication Scheme ↩
12. BIRD Internet Routing Daemon 2.0 ↩
13. cron: time-based job scheduler in Unix-like OS ↩
14. htpasswd: manage user files for Basic Authentication ↩
15. DNS Certification Authority Authorization ↩
16. Alpine Linux ↩
17. network-list-parser: parse, normalize and aggregate list of IPv4 networks/addresses ↩
18. Supervisor: a process control system ↩
19. Border Gateway Protocol ↩
20. Road Warrior ↩
21. Bidirectional Forwarding Detection ↩
22. Unified register of resources which are forbidden in the Russian Federation ↩
23. Register of Internet Addresses filtered in Russian Federation ↩
24. MTProto Mobile Protocol ↩
25. mtg: MTProto proxy for Telegram ↩
26. MTProto Fake TLS ↩
27. DuckDuckGo Internet Search Engine ↩
28. Server Name Indication ↩
29. Application Programming Interface ↩