summaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'README.md')
-rw-r--r--README.md142
1 files changed, 90 insertions, 52 deletions
diff --git a/README.md b/README.md
index cdee62f..a99c0a8 100644
--- a/README.md
+++ b/README.md
@@ -11,11 +11,37 @@ This is an online version of [mat2](https://0xacab.org/jvoisin/mat2).
11Keep in mind that this is a beta version, don't rely on it for anything 11Keep in mind that this is a beta version, don't rely on it for anything
12serious, yet. 12serious, yet.
13 13
14# Introduction
15
16Mat2-web offers a Restful API which enables consumers to upload their files
17which will be cleaned using [mat2](https://0xacab.org/jvoisin/mat2) and can
18be downloaded afterward.
19
20Therefore this project has two components:
21
22
23
24* Restful API [Mat2-web](https://0xacab.org/jvoisin/mat2-web)
25* Single Page Application Frontend [mat2-quasar-frontend](https://0xacab.org/jfriedli/mat2-quasar-frontend)
26
27**To setup the application you'll need to deploy both parts.**
28
29There are several ways for deployment:
30
311) [Manually](#manually)
322) [Ansible](#deploy-via-ansible)
333) [Containers](#container)
34
35# Table Of Contents
36
37[[_TOC_]]
38
14# Demo instance 39# Demo instance
15 40
16There is a demo instance deployed a [mat2-web.dustri.org](https://mat2-web.dustri.org). 41There is a demo instance deployed a [matweb.info](https://matweb.info).
17Please don't upload any sensitive files to it. 42Please don't upload any sensitive files to it.
18 43
44
19# Vue Frontend 45# Vue Frontend
20![Frontend GIF Preview](https://0xacab.org/jfriedli/mat2-quasar-frontend/raw/2dd5de537088d67fe4167bf5b2e1f5dacf2fa537/mat-frontend.gif?inline=true) 46![Frontend GIF Preview](https://0xacab.org/jfriedli/mat2-quasar-frontend/raw/2dd5de537088d67fe4167bf5b2e1f5dacf2fa537/mat-frontend.gif?inline=true)
21There is a SPA Frontend available at https://0xacab.org/jfriedli/mat2-quasar-frontend. It consumes 47There is a SPA Frontend available at https://0xacab.org/jfriedli/mat2-quasar-frontend. It consumes
@@ -24,6 +50,8 @@ To set it up checkout the [Readme](https://0xacab.org/jfriedli/mat2-quasar-front
24 50
25# How to deploy it? 51# How to deploy it?
26 52
53## Manually
54
27mat2 is available in [Debian stable](https://packages.debian.org/stable/mat2). 55mat2 is available in [Debian stable](https://packages.debian.org/stable/mat2).
28 56
29``` 57```
@@ -47,17 +75,7 @@ Nginx is the recommended web engine, but you can also use Apache if you prefer,
47by copying [this file](https://0xacab.org/jvoisin/mat2-web/tree/master/config/apache2.config) 75by copying [this file](https://0xacab.org/jvoisin/mat2-web/tree/master/config/apache2.config)
48to your `/etc/apache2/sites-enabled/mat2-web` file. 76to your `/etc/apache2/sites-enabled/mat2-web` file.
49 77
50Configure the following environment variables: 78Make sure you configured the necessary [environment variables](#configuration).
51
52 - `MAT2_ALLOW_ORIGIN_WHITELIST=https://myhost1.org https://myhost2.org`
53 Note that you can add multiple hosts from which you want to accept API requests. These need to be separated by
54a space. **IMPORTANT:** The default value if the variable is not set is: `Access-Control-Allow-Origin: *`
55 - `MAT2_MAX_FILES_BULK_DOWNLOAD=10` Max number of files that can be grouped for a bulk download.
56 Note: Each file has a max file size of 16mb
57
58 - `MAT2_MAX_FILE_AGE_FOR_REMOVAL=900` Seconds a file in the upload folder is kept.
59 After that it will be deleted. Default `15 * 60`
60 - `MAT2_WEB_DOWNLOAD_FOLDER` Define the upload folder path. Defaults to: `./uploads/`
61 79
62Finally, restart uWSGI and your web server: 80Finally, restart uWSGI and your web server:
63 81
@@ -71,7 +89,7 @@ It should now be working.
71Note for reverse proxies: Include the Host header to ensure all generated urls are correct. 89Note for reverse proxies: Include the Host header to ensure all generated urls are correct.
72e.g. for Nginx: `proxy_set_header Host $host;` 90e.g. for Nginx: `proxy_set_header Host $host;`
73 91
74# Deploy via Ansible 92## Deploy via Ansible
75 93
76If you happen to be using [Ansible](https://www.ansible.com/), there's an 94If you happen to be using [Ansible](https://www.ansible.com/), there's an
77Ansible role to deploy mat2-web on Debian, thanks to the amazing 95Ansible role to deploy mat2-web on Debian, thanks to the amazing
@@ -85,16 +103,61 @@ collector cronjob to remove leftover files. Besides, it can create a
85the uploads folder, to ensure that the uploaded files won't be recoverable 103the uploads folder, to ensure that the uploaded files won't be recoverable
86between reboots. 104between reboots.
87 105
88# Development 106## Container
89Install docker and docker-compose and then run `docker-compose up` to setup 107
90the docker dev environment. Mat2-web is now accessible on your host machine at `localhost:5000`. 108There are two Dockerfiles present in this repository.
91Every code change triggers a restart of the app. 109The file called `Dockerfile.development` is used for development
92If you want to add/remove dependencies you have to rebuild the container. 110and `Dockerfile.production` is used for production deployments.
111
112You can find the automated container builds in the registry of this
113repository: https://0xacab.org/jvoisin/mat2-web/container_registry
114
115Make sure you configured the necessary [environment variables](#configuration).
116
117
118### Building the production image using Docker
119Build command: `docker build -f Dockerfile.production -t mat-web .`
120
121Run it: `docker run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json mat-web:latest`
122
123This does mount the upload folder as tmpfs and servers the app on `localhost:8181`.
124
125### Building the production image using Podman
126Build: `podman build -f Dockerfile.production -t matweb-podman .`
127
128Run: `podman run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json matweb-podman:latest`
129
130
131# Configuration
132
133The default settings from `main.py` may be overridden by adding a `config.py`
134file and add custom values for the relevant flask config variables. E.g.:
135
136```
137MAX_CONTENT_LENGTH = 32 * 1024 * 1024 # 32MB
138```
139## Configurable Environment Variables
140
141| Env Variable | Default Value | Explanation |
142|-------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
143| MAT2_ALLOW_ORIGIN_WHITELIST | * | Define which hosts are included in the Access-Control-Allow-Origin header. Note that you can add multiple hosts from which you want to accept API requests e.g. https://myhost1.org https://myhost2.org. These need to be separated by a space. **IMPORTANT**: The default value if the variable is not set is: `Access-Control-Allow-Origin: *` |
144| MAT2_MAX_FILES_BULK_DOWNLOAD | 10 | Max number of files that can be grouped for a bulk download |
145| MAT2_MAX_FILE_AGE_FOR_REMOVAL | 900 | Seconds a file in the upload folder is kept. After that it will be deleted |
146| MAT2_WEB_DOWNLOAD_FOLDER | ./uploads/ | Define the upload folder path. |
147## Custom templates
148
149You can override the default templates from `templates/` by putting replacements
150into the directory path that's configured in `app.config['CUSTOM_TEMPLATES_DIR']`
151(default `custom_templates/`).
93 152
94# RESTful API 153# RESTful API
95 154
96If you go to https://api.matweb.info/apidocs/ you can find a Swagger documentation. 155If you go to https://api.matweb.info/apidocs/ you can find a Swagger documentation.
97 156
157<p>
158<details>
159<summary>Click this for API Endpoints explanation</summary>
160
98## Upload Endpoint 161## Upload Endpoint
99 162
100**Endpoint:** `/api/upload` 163**Endpoint:** `/api/upload`
@@ -191,49 +254,24 @@ The `key` parameter is the key from a previously uploaded file.
191} 254}
192``` 255```
193 256
194# Docker 257</details>
195 258</p>
196There are two Dockerfiles present in this repository. The file called `Dockerfile.development` is used for development
197and `Dockerfile.production` is used for production deployments.
198
199You can find the automated docker builds in the registry of this
200repository: https://0xacab.org/jvoisin/mat2-web/container_registry
201
202### Building the production image
203Build command: `docker build -f Dockerfile.production -t mat-web .`
204
205Run it: `docker run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json mat-web:latest`
206
207This does mount the upload folder as tmpfs and servers the app on `localhost:8181`.
208
209##### Podman
210Build: `podman build -f Dockerfile.production -t matweb-podman .`
211
212Run: `podman run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json matweb-podman:latest`
213
214# Configuration
215 259
216The default settings from `main.py` may be overridden by adding a `config.py`
217file and add custom values for the relevant flask config variables. E.g.:
218
219```
220MAX_CONTENT_LENGTH = 32 * 1024 * 1024 # 32MB
221```
222 260
261# Development / Contribute
262Install docker and docker-compose and then run `docker-compose up` to setup
263the docker dev environment. Mat2-web is now accessible on your host machine at `localhost:5000`.
264Every code change triggers a restart of the app.
265If you want to add/remove dependencies you have to rebuild the container.
223See [Flask configuration docs](http://exploreflask.com/en/latest/configuration.html) 266See [Flask configuration docs](http://exploreflask.com/en/latest/configuration.html)
224for further information. 267for further information.
225 268
226# Custom templates
227
228You can override the default templates from `templates/` by putting replacements
229into the directory path that's configured in `app.config['CUSTOM_TEMPLATES_DIR']`
230(default `custom_templates/`).
231 269
232# Threat model 270# Threat model
233 271
234- An attacker in possession of the very same file that a user wants to clean, 272- An attacker in possession of the very same file that a user wants to clean,
235 along with its names, can perform a denial of service by continually 273 along with its names, can't perform a denial of service by continually
236 requesting this file, and getting it before the user. 274 requesting this file.
237- An attacker in possession of only the name of a file that a user wants to 275- An attacker in possession of only the name of a file that a user wants to
238 clean can't perform a denial of service attack, since the path to download 276 clean can't perform a denial of service attack, since the path to download
239 the cleaned file is not only dependent of the name, but also the content. 277 the cleaned file is not only dependent of the name, but also the content.