diff options
| author | jfriedli | 2020-07-14 15:05:48 -0700 |
|---|---|---|
| committer | jfriedli | 2020-07-14 15:05:48 -0700 |
| commit | 79d4a66dd43543f8404f40c03ccfd1fb6b681018 (patch) | |
| tree | 8ade56ceabfb207eafab043ff5f598e930223268 /README.md | |
| parent | 2ba8d74fa7e104ddfaac365c1efe1bb99c632ed3 (diff) | |
Update README.mdv0.4.0
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 142 |
1 files changed, 90 insertions, 52 deletions
| @@ -11,11 +11,37 @@ This is an online version of [mat2](https://0xacab.org/jvoisin/mat2). | |||
| 11 | Keep in mind that this is a beta version, don't rely on it for anything | 11 | Keep in mind that this is a beta version, don't rely on it for anything |
| 12 | serious, yet. | 12 | serious, yet. |
| 13 | 13 | ||
| 14 | # Introduction | ||
| 15 | |||
| 16 | Mat2-web offers a Restful API which enables consumers to upload their files | ||
| 17 | which will be cleaned using [mat2](https://0xacab.org/jvoisin/mat2) and can | ||
| 18 | be downloaded afterward. | ||
| 19 | |||
| 20 | Therefore this project has two components: | ||
| 21 | |||
| 22 | |||
| 23 | |||
| 24 | * Restful API [Mat2-web](https://0xacab.org/jvoisin/mat2-web) | ||
| 25 | * Single Page Application Frontend [mat2-quasar-frontend](https://0xacab.org/jfriedli/mat2-quasar-frontend) | ||
| 26 | |||
| 27 | **To setup the application you'll need to deploy both parts.** | ||
| 28 | |||
| 29 | There are several ways for deployment: | ||
| 30 | |||
| 31 | 1) [Manually](#manually) | ||
| 32 | 2) [Ansible](#deploy-via-ansible) | ||
| 33 | 3) [Containers](#container) | ||
| 34 | |||
| 35 | # Table Of Contents | ||
| 36 | |||
| 37 | [[_TOC_]] | ||
| 38 | |||
| 14 | # Demo instance | 39 | # Demo instance |
| 15 | 40 | ||
| 16 | There is a demo instance deployed a [mat2-web.dustri.org](https://mat2-web.dustri.org). | 41 | There is a demo instance deployed a [matweb.info](https://matweb.info). |
| 17 | Please don't upload any sensitive files to it. | 42 | Please don't upload any sensitive files to it. |
| 18 | 43 | ||
| 44 | |||
| 19 | # Vue Frontend | 45 | # Vue Frontend |
| 20 |  | 46 |  |
| 21 | There is a SPA Frontend available at https://0xacab.org/jfriedli/mat2-quasar-frontend. It consumes | 47 | There is a SPA Frontend available at https://0xacab.org/jfriedli/mat2-quasar-frontend. It consumes |
| @@ -24,6 +50,8 @@ To set it up checkout the [Readme](https://0xacab.org/jfriedli/mat2-quasar-front | |||
| 24 | 50 | ||
| 25 | # How to deploy it? | 51 | # How to deploy it? |
| 26 | 52 | ||
| 53 | ## Manually | ||
| 54 | |||
| 27 | mat2 is available in [Debian stable](https://packages.debian.org/stable/mat2). | 55 | mat2 is available in [Debian stable](https://packages.debian.org/stable/mat2). |
| 28 | 56 | ||
| 29 | ``` | 57 | ``` |
| @@ -47,17 +75,7 @@ Nginx is the recommended web engine, but you can also use Apache if you prefer, | |||
| 47 | by copying [this file](https://0xacab.org/jvoisin/mat2-web/tree/master/config/apache2.config) | 75 | by copying [this file](https://0xacab.org/jvoisin/mat2-web/tree/master/config/apache2.config) |
| 48 | to your `/etc/apache2/sites-enabled/mat2-web` file. | 76 | to your `/etc/apache2/sites-enabled/mat2-web` file. |
| 49 | 77 | ||
| 50 | Configure the following environment variables: | 78 | Make sure you configured the necessary [environment variables](#configuration). |
| 51 | |||
| 52 | - `MAT2_ALLOW_ORIGIN_WHITELIST=https://myhost1.org https://myhost2.org` | ||
| 53 | Note that you can add multiple hosts from which you want to accept API requests. These need to be separated by | ||
| 54 | a space. **IMPORTANT:** The default value if the variable is not set is: `Access-Control-Allow-Origin: *` | ||
| 55 | - `MAT2_MAX_FILES_BULK_DOWNLOAD=10` Max number of files that can be grouped for a bulk download. | ||
| 56 | Note: Each file has a max file size of 16mb | ||
| 57 | |||
| 58 | - `MAT2_MAX_FILE_AGE_FOR_REMOVAL=900` Seconds a file in the upload folder is kept. | ||
| 59 | After that it will be deleted. Default `15 * 60` | ||
| 60 | - `MAT2_WEB_DOWNLOAD_FOLDER` Define the upload folder path. Defaults to: `./uploads/` | ||
| 61 | 79 | ||
| 62 | Finally, restart uWSGI and your web server: | 80 | Finally, restart uWSGI and your web server: |
| 63 | 81 | ||
| @@ -71,7 +89,7 @@ It should now be working. | |||
| 71 | Note for reverse proxies: Include the Host header to ensure all generated urls are correct. | 89 | Note for reverse proxies: Include the Host header to ensure all generated urls are correct. |
| 72 | e.g. for Nginx: `proxy_set_header Host $host;` | 90 | e.g. for Nginx: `proxy_set_header Host $host;` |
| 73 | 91 | ||
| 74 | # Deploy via Ansible | 92 | ## Deploy via Ansible |
| 75 | 93 | ||
| 76 | If you happen to be using [Ansible](https://www.ansible.com/), there's an | 94 | If you happen to be using [Ansible](https://www.ansible.com/), there's an |
| 77 | Ansible role to deploy mat2-web on Debian, thanks to the amazing | 95 | Ansible role to deploy mat2-web on Debian, thanks to the amazing |
| @@ -85,16 +103,61 @@ collector cronjob to remove leftover files. Besides, it can create a | |||
| 85 | the uploads folder, to ensure that the uploaded files won't be recoverable | 103 | the uploads folder, to ensure that the uploaded files won't be recoverable |
| 86 | between reboots. | 104 | between reboots. |
| 87 | 105 | ||
| 88 | # Development | 106 | ## Container |
| 89 | Install docker and docker-compose and then run `docker-compose up` to setup | 107 | |
| 90 | the docker dev environment. Mat2-web is now accessible on your host machine at `localhost:5000`. | 108 | There are two Dockerfiles present in this repository. |
| 91 | Every code change triggers a restart of the app. | 109 | The file called `Dockerfile.development` is used for development |
| 92 | If you want to add/remove dependencies you have to rebuild the container. | 110 | and `Dockerfile.production` is used for production deployments. |
| 111 | |||
| 112 | You can find the automated container builds in the registry of this | ||
| 113 | repository: https://0xacab.org/jvoisin/mat2-web/container_registry | ||
| 114 | |||
| 115 | Make sure you configured the necessary [environment variables](#configuration). | ||
| 116 | |||
| 117 | |||
| 118 | ### Building the production image using Docker | ||
| 119 | Build command: `docker build -f Dockerfile.production -t mat-web .` | ||
| 120 | |||
| 121 | Run it: `docker run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json mat-web:latest` | ||
| 122 | |||
| 123 | This does mount the upload folder as tmpfs and servers the app on `localhost:8181`. | ||
| 124 | |||
| 125 | ### Building the production image using Podman | ||
| 126 | Build: `podman build -f Dockerfile.production -t matweb-podman .` | ||
| 127 | |||
| 128 | Run: `podman run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json matweb-podman:latest` | ||
| 129 | |||
| 130 | |||
| 131 | # Configuration | ||
| 132 | |||
| 133 | The default settings from `main.py` may be overridden by adding a `config.py` | ||
| 134 | file and add custom values for the relevant flask config variables. E.g.: | ||
| 135 | |||
| 136 | ``` | ||
| 137 | MAX_CONTENT_LENGTH = 32 * 1024 * 1024 # 32MB | ||
| 138 | ``` | ||
| 139 | ## Configurable Environment Variables | ||
| 140 | |||
| 141 | | Env Variable | Default Value | Explanation | | ||
| 142 | |-------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| 143 | | MAT2_ALLOW_ORIGIN_WHITELIST | * | Define which hosts are included in the Access-Control-Allow-Origin header. Note that you can add multiple hosts from which you want to accept API requests e.g. https://myhost1.org https://myhost2.org. These need to be separated by a space. **IMPORTANT**: The default value if the variable is not set is: `Access-Control-Allow-Origin: *` | | ||
| 144 | | MAT2_MAX_FILES_BULK_DOWNLOAD | 10 | Max number of files that can be grouped for a bulk download | | ||
| 145 | | MAT2_MAX_FILE_AGE_FOR_REMOVAL | 900 | Seconds a file in the upload folder is kept. After that it will be deleted | | ||
| 146 | | MAT2_WEB_DOWNLOAD_FOLDER | ./uploads/ | Define the upload folder path. | | ||
| 147 | ## Custom templates | ||
| 148 | |||
| 149 | You can override the default templates from `templates/` by putting replacements | ||
| 150 | into the directory path that's configured in `app.config['CUSTOM_TEMPLATES_DIR']` | ||
| 151 | (default `custom_templates/`). | ||
| 93 | 152 | ||
| 94 | # RESTful API | 153 | # RESTful API |
| 95 | 154 | ||
| 96 | If you go to https://api.matweb.info/apidocs/ you can find a Swagger documentation. | 155 | If you go to https://api.matweb.info/apidocs/ you can find a Swagger documentation. |
| 97 | 156 | ||
| 157 | <p> | ||
| 158 | <details> | ||
| 159 | <summary>Click this for API Endpoints explanation</summary> | ||
| 160 | |||
| 98 | ## Upload Endpoint | 161 | ## Upload Endpoint |
| 99 | 162 | ||
| 100 | **Endpoint:** `/api/upload` | 163 | **Endpoint:** `/api/upload` |
| @@ -191,49 +254,24 @@ The `key` parameter is the key from a previously uploaded file. | |||
| 191 | } | 254 | } |
| 192 | ``` | 255 | ``` |
| 193 | 256 | ||
| 194 | # Docker | 257 | </details> |
| 195 | 258 | </p> | |
| 196 | There are two Dockerfiles present in this repository. The file called `Dockerfile.development` is used for development | ||
| 197 | and `Dockerfile.production` is used for production deployments. | ||
| 198 | |||
| 199 | You can find the automated docker builds in the registry of this | ||
| 200 | repository: https://0xacab.org/jvoisin/mat2-web/container_registry | ||
| 201 | |||
| 202 | ### Building the production image | ||
| 203 | Build command: `docker build -f Dockerfile.production -t mat-web .` | ||
| 204 | |||
| 205 | Run it: `docker run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json mat-web:latest` | ||
| 206 | |||
| 207 | This does mount the upload folder as tmpfs and servers the app on `localhost:8181`. | ||
| 208 | |||
| 209 | ##### Podman | ||
| 210 | Build: `podman build -f Dockerfile.production -t matweb-podman .` | ||
| 211 | |||
| 212 | Run: `podman run -ti -p8181:8080 --read-only --tmpfs /tmp --tmpfs /run/uwsgi --tmpfs=/app/upload --security-opt=no-new-privileges --security-opt=seccomp=./config/seccomp.json matweb-podman:latest` | ||
| 213 | |||
| 214 | # Configuration | ||
| 215 | 259 | ||
| 216 | The default settings from `main.py` may be overridden by adding a `config.py` | ||
| 217 | file and add custom values for the relevant flask config variables. E.g.: | ||
| 218 | |||
| 219 | ``` | ||
| 220 | MAX_CONTENT_LENGTH = 32 * 1024 * 1024 # 32MB | ||
| 221 | ``` | ||
| 222 | 260 | ||
| 261 | # Development / Contribute | ||
| 262 | Install docker and docker-compose and then run `docker-compose up` to setup | ||
| 263 | the docker dev environment. Mat2-web is now accessible on your host machine at `localhost:5000`. | ||
| 264 | Every code change triggers a restart of the app. | ||
| 265 | If you want to add/remove dependencies you have to rebuild the container. | ||
| 223 | See [Flask configuration docs](http://exploreflask.com/en/latest/configuration.html) | 266 | See [Flask configuration docs](http://exploreflask.com/en/latest/configuration.html) |
| 224 | for further information. | 267 | for further information. |
| 225 | 268 | ||
| 226 | # Custom templates | ||
| 227 | |||
| 228 | You can override the default templates from `templates/` by putting replacements | ||
| 229 | into the directory path that's configured in `app.config['CUSTOM_TEMPLATES_DIR']` | ||
| 230 | (default `custom_templates/`). | ||
| 231 | 269 | ||
| 232 | # Threat model | 270 | # Threat model |
| 233 | 271 | ||
| 234 | - An attacker in possession of the very same file that a user wants to clean, | 272 | - An attacker in possession of the very same file that a user wants to clean, |
| 235 | along with its names, can perform a denial of service by continually | 273 | along with its names, can't perform a denial of service by continually |
| 236 | requesting this file, and getting it before the user. | 274 | requesting this file. |
| 237 | - An attacker in possession of only the name of a file that a user wants to | 275 | - An attacker in possession of only the name of a file that a user wants to |
| 238 | clean can't perform a denial of service attack, since the path to download | 276 | clean can't perform a denial of service attack, since the path to download |
| 239 | the cleaned file is not only dependent of the name, but also the content. | 277 | the cleaned file is not only dependent of the name, but also the content. |
