kore-doc

The kore documentation found under https://docs.kore.io/
Commits | Files | Refs | README | git clone https://git.kore.io/kore-doc.git

commit 55eb7972ea5c8a52e738958bb7d7109248c3d012
parent 05b61abd0aed9a46b28437133f15655233a3df56
Author: Joris Vink <joris@coders.se>
Date:   Fri, 18 Mar 2022 16:20:33 +0100

Merge branch 'master' of nightfall.coders.se:/home/kore/kore-doc

Diffstat:
SUMMARY.md | 1+
TODO | 37+++++++++++++++++++++++++++++++++++++
api/http.md | 64++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/json.md | 31+++++++++++++++++++++++--------
api/python.md | 50++++++++++++++++++++++++++++++++++++++++++--------
applications/acme.md | 6+++---
applications/building.md | 4+++-
applications/integrated_tools.md | 8++++----
applications/koreconf.md | 87+++++++++++++++++++++++++++++++++++++++----------------------------------------
applications/privsep.md | 38++++++++++++++++++++++++++++++++++++++
applications/routes.md | 33++++++++++++++++++++-------------
applications/running.md | 9+++++++++
book.json | 4++--
install.md | 54++++++++++++++++++++++++------------------------------
14 files changed, 313 insertions(+), 113 deletions(-)

diff --git a/SUMMARY.md b/SUMMARY.md @@ -8,6 +8,7 @@ * [build.conf](applications/buildconf_reference.md) * [Running](applications/running.md) * [Configuration](applications/koreconf.md) + * [Privilege Separation](applications/privsep.md) * [Routes](applications/routes.md) * [Filemaps](applications/filemap.md) * [Automatic HTTPs](applications/acme.md) diff --git a/TODO b/TODO @@ -0,0 +1,37 @@ +* decide on wether or not this is kore5 or 4.2.0 + - done its 4.2.0 + +* incorporate notls patch again from @tutus + - fixed via TLS_BACKEND + +* update docs for new http argument get API +* update docs regarding new privsep configuration + done +* update docs for new routing configuration and hooks + done + +* http_state_create() changed +* tls_dhparam optional now (we ship with one) + +* document new http_response*() apis + done + +* document new route decorator for Python (@kore.route) +* document new python req.connection.x509dict + (should probably add validity timestamps to here too) + ignore, fixed later in kore5 + +* document new req.body_digest + no, hidden for now. +* document deployment targets for python + done +* document new curlopt keyword for kore.httpclient + done +* document new kodev commands + done +* document new worker startup sequence + done +* document new worker log options + done +* document json api changes + doneish diff --git a/api/http.md b/api/http.md @@ -8,6 +8,8 @@ This page contains all available public functions related to understanding and r * [http\_response\_header](#http_response_header) * [http\_response\_stream](#http_response_stream) * [http\_response\_fileref](#http_response_fileref) +* [http\_response\_close](#http_response_close) +* [http\_response\_json](#http_response_json) * [http\_request\_header](#http_request_header) * [http\_file\_lookup](#http_file_lookup) * [http\_file\_read](#http_file_read) @@ -142,6 +144,68 @@ Nothing --- +# http\_response\_close {#http_response_close} + +### Synopsis + +``` +void http_response_close(struct http_request *req, int code, const void *data, size_t length) +``` + +### Description + +Creates an HTTP response for an HTTP request much like _http\_response\(\)_. + +This function however will force the connection to be closed immediately +after the response has been sent to the server. + +It will automatically add a "connection" header with "close" as its value. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to respond to. | +| status | The HTTP status code to include in the response. | +| data | The data to be sent in the response body. | +| length | The length of the data to be sent. | + +### Returns + +Nothing + +--- + +# http\_response\_json {#http_response_json} + +### Synopsis + +``` +void http_response_json(struct http_request *req, int code, struct kore_json_item *json) +``` + +### Description + +Creates an HTTP response for an HTTP request much like _http\_response\(\)_. + +This function takes a Kore JSON item that is to be sent as a response. + +It will automatically add a "content-type" header with "application/json" +as its value. + +The caller *must* not free the provided JSON item nor *use* it after +calling this function as this function will free it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to respond to. | +| status | The JSON data to be sent that was previously created. +| json | The data to be sent in the response body. | + +### Returns + +Nothing + +--- + # http\_request\_header {#http_request_header} ### Synopsis diff --git a/api/json.md b/api/json.md @@ -7,6 +7,7 @@ Kore provides a built-in JSON parser for your applications. * [kore\_json\_init](#init) * [kore\_json\_cleanup](#cleanup) * [kore\_json\_parse](#parse) + * [kore\_json\_find\_object](#findobject) * [kore\_json\_find\_array](#findarray) * [kore\_json\_find\_string](#findstring) @@ -19,6 +20,7 @@ Kore provides a built-in JSON parser for your applications. * [kore\_json\_create\_number](#createnumber) * [kore\_json\_create\_literal](#createliteral) +* [kore\_json\_errno](#errno) * [kore\_json\_strerror](#strerror) ## Examples @@ -30,7 +32,7 @@ See the included [example](https://github.com/jorisvink/kore/tree/master/example # kore\_json\_init {#init} ### Synopsis ``` -void kore_json_init(struct kore_json *json, const u_int8_t *data, size_t len); +void kore_json_init(struct kore_json *json, const void *data, size_t len); ``` ### Description Initializes a **kore_json** context. This must be called before any other @@ -394,17 +396,30 @@ if (kore_json_create_literal(parent, "istrue", KORE_JSON_FALSE) == NULL) ``` --- -# kore\_json\_strerror {#strerror} +# kore\_json\_errno {#errno} ### Synopsis ``` -const char *kore_json_strerror(struct kore_json *json); +int kore_json_errno(void); ``` ### Description -Returns a human readable string for the error that occurred in the context. +Returns a the last error code that occurred. -| Parameter | Description | -| -- | -- | -| json | A kore\_json context in which an error occurred. | +### Returns +The last error code that has occurred. + +### Example + +```c +kore_log(LOG_INFO, "parsing error '%d'", kore_json_errno()); +``` +--- +# kore\_json\_strerror {#strerror} +### Synopsis +``` +const char *kore_json_strerror(void); +``` +### Description +Returns a human readable string for last error that occurred. ### Returns A pointer to a human readable string that describes the error. @@ -413,5 +428,5 @@ This pointer should not be freed by the caller. ### Example ```c -kore_log(LOG_INFO, "parsing error '%s'", kore_json_strerror(&json)); +kore_log(LOG_INFO, "parsing error '%s'", kore_json_strerror()); ``` diff --git a/api/python.md b/api/python.md @@ -81,6 +81,7 @@ koreapp = MyApp() * [setname](#setname) * [coroname](#coroname) * [corotrace](#corotrace) + * [privsep](#privsep) * [Http module](#httpmodule) @@ -123,8 +124,6 @@ set via the Python code. | Configuration option | Description | | --- | --- | -| root | The root path in which the Kore server runs (either via chroot or chdir). If not set, the current working directory. | -| runas | The user the worker processes will run as. If not set, the current user. | | workers | The number of worker processes to use. If not set, the number of CPU cores in the system. | | worker\_max\_connections | The maximum number of active connections a worker process holds before refusing to accept more. | | worker\_rlimit\_nofiles | The maximum number of open file descriptor per worker. | @@ -137,10 +136,6 @@ set via the Python code. | tls\_cipher | OpenSSL ciphersuite list to use. Defaults to a very sane list with only AEAD ciphers and ephemeral key exchanges. | | tls\_dhparam | Path to DH parameters for the server to use. | | rand\_file | Path to a 2048 byte file containing entropy used to seed the PRNG. | -| keymgr\_runas | The user the keymgr process will run as. If not set, the current user. | -| keymgr\_root | The root path for the keymgr process. If not set, inherited from the root option. | -| acme\_runas | The user the acme process will run as. If not set, the current user. | -| acme\_root | The root path for the acme process. If not set, inherited from the root option. | | acme\_email | An email address used for account registration. | | acme\_provider | A URL to the directory for an ACME provider. Defaults to Let's Encrypt. | | pledge | OpenBSD only, pledge categories for the worker processes. | @@ -156,7 +151,8 @@ set via the Python code. | http\_body\_disk\_path | A path where the temporary body files are written if the http\_body\_disk\_offload setting is enabled. | | http\_server\_version | Allows you to override the Kore server header. | | http\_pretty\_error | If set to "yes" will display HTML based HTTP error codes. Defaults to "no". | -| deployment | The deployment type of the application. Either "production" or "development". The production setting will cause Kore to chroot and drop privileges and run in the background. The development setting will run Kore in the foreground and only chdir into the root setting. | +| logfile | Set the logfile to which Kore will write all worker output. | +| deployment | The deployment type of the application. Either "production", "development" or "docker". The production setting will cause Kore to chroot and drop privileges and run in the background. The development setting will run Kore in the foreground and only chdir into the root setting. The docker setting will cause Kore to run in the foreground but still apply priviledge separation. | Any configuration options should be set in your App.configure method. @@ -857,6 +853,7 @@ Addiotnally, optional keyword parameters can be specified: | tlsverify | Should the peer its x509 certificate be verified (default: True). | | cabundle | A path to a PEM bundle containing the CAs you would like to use to verify the peer. | | unix | Use this unix socket path instead (On Linux if prefixed with '@' this becomes an abstract socket path. | +| curlopt | A dictionary containing additional curl options and their settings. | Kore internally will re-use connections to the same hosts if they are available, as a developer you can go ahead and create multiple httpclient @@ -878,15 +875,22 @@ functions on it to perform the requested action (these calls return a coroutine | body | For put, post and patch, add the given bytes as the HTTP body. | | headers | A dictionary containing the headers to be added to the request. | | return\_headers | Set to True to return the headers as well (default: False). | +| curlopt | A dictionary containing additional curl options and their settings. | Unless the return\_headers keyword is True, these functions return a tuple containing the status and HTTP body. If return\_headers is True, the tuple returned will be status, headers, HTTP body. +Any curlopt entries given inside of the action calls will override +the options set on the httpclient context. + ### Example ```python -client = kore.httpclient("https://kore.io") +client = kore.httpclient("https://kore.io", + curlopt={ + kore.CURLOPT_TIMEOUT: 120 + }) # Do a normal GET request. status, body = await client.get() @@ -897,6 +901,9 @@ status, body = await client.post( body=mybody.encode("utf-8"), headers={ "x-header": "from-client" + }, + curlopt={ + kore.CURLOPT_TIMEOUT: 60 } ) @@ -1106,6 +1113,33 @@ kore.corotrace(True) --- +# privsep {#privsep} + +### Synopsis + +```python +kore.privsep("name", root="/var/chroot/kore", runas="_kore") +``` + +### Description + +Configuration privilege separation for Kore processes. + +This allows you to set the root directory of each process and what +user it will run as. + +### Returns + +Nothing + +### Example + +```python +kore.privsep("acme", root="/var/chroot/acme", runas="acme") +kore.privsep("worker", root="/var/chroot/kore", runas="kore") +kore.privsep("keymgr", root="/var/chroot/keymgr", runas="keymgr") +``` +--- # HTTP {#httpmodule} diff --git a/applications/acme.md b/applications/acme.md @@ -34,13 +34,13 @@ There are a few ACME related configuration options. | Configuration option | Description | | --- | --- | -| acme\_runas | The user the acme process will run as. If not set, the current user. | -| acme\_root | The root path for the acme process. If not set, inherited from the root option. | | acme\_email | An email address used for account registration. | | acme\_provider | A URL to the directory for an ACME provider. Defaults to Let's Encrypt. | The default ACME provider is "https://acme-v02.api.letsencrypt.org/directory". +To control privilege separation for ACME use the privsep configuration option. + ## ACME architecture When ACME is enabled, Kore will create a new acme process that stands @@ -53,7 +53,7 @@ all your private keys (even the ACME account key is only held by keymgr). ## ACME files All certificates and private keys are stored under the directory that -was configured via the **keymgr_root** configuration option. +was configured via the **acme privsep root**. The RSA account key is stored as **account-key.pem** in the **keymgr_root** directory while certificates and matching domain keys are stored under diff --git a/applications/building.md b/applications/building.md @@ -28,7 +28,9 @@ The example below configures / to serve the static asset index.html. ``` domain * { - route / asset_serve_index_html + route / { + handler asset_serve_index_html + } } ``` diff --git a/applications/integrated_tools.md b/applications/integrated_tools.md @@ -8,6 +8,9 @@ kodev build kodev run kodev flavor kodev info +kodev gen +kodev cflags +kodev ldflags ``` ### Creating an application @@ -95,7 +98,4 @@ The **kodev** tool will pickup the following environment variables if set: > The directory where the .o files will be placed. **KODEV_OUTPUT** -> The directory where the resulting binary is placed. - - - +> The directory where the resulting output is placed. diff --git a/applications/koreconf.md b/applications/koreconf.md @@ -9,10 +9,6 @@ Therefore it is an integral part of Kore as a whole. A server context sets up a one or more listeners for the Kore server. These listeners can either be ipv4/ipv6 addresses or unix sockets. -The old bind and bind_unix configuration options have been migrated since -Kore 4.0.0 to these server contexts. Note that the bind_unix option has been -renamed to unix. - You can also turn off TLS in a server context by specifying the **tls no** option inside of a context. @@ -32,45 +28,48 @@ server notls { } ``` +# Domain context + +A domain context attaches to a server context and defines any available +routes inside of that domain and how they are routed. + +Example: + +``` +domain kore.io { + attach tls + + route / { + handler index + methods get + } +} +``` + +# Route context + +A route context is specified inside of a domain context and will tell Kore +how to handle requests to the given route, what methods are allowed and +how to validate any parameters for that route. + +Several different routing contexts can be given for the same path as long +as the methods differ. + +Example: + +``` +route / { + handler index + methods get +} + +route / { + handler index_post + methods post + validate post username v_user +} +``` + # Configuration options. -There are more options than what is listed below, specifically for validators, authentication blocks and domains. Please find those in https://github.com/jorisvink/kore/blob/master/conf/kore.conf.example. - ---- - -| Configuration option | Description | -| --- | --- | -| root | The root path in which the Kore server runs (either via chroot or chdir). If not set, the current working directory. | -| runas | The user the worker processes will run as. If not set, the current user. | -| workers | The number of worker processes to use. If not set, the number of CPU cores in the system. | -| worker\_max\_connections | The maximum number of active connections a worker process holds before refusing to accept more. | -| worker\_rlimit\_nofiles | The maximum number of open file descriptor per worker. | -| worker\_accept\_threshold | The maximum number of new connections to accept in a single event loop. | -| worker\_death\_policy | The death policy for a worker, "restart" by default. If set to "terminate" will cause the Kore server to shutdown on abnormal worker termination. | -| worker\_set\_affinity | Worker CPU affinity (0 or 1, default 1). | -| pidfile | The path to a file in which the server will write the PID for the parent process. | -| socket\_backlog | The number of pending connections. | -| tls\_version | The TLS version to use (default: both, 1.2 for TLSv1.2 only and 1.3 for TLSv1.3 only). | -| tls\_cipher | OpenSSL ciphersuite list to use. Defaults to a very sane list with only AEAD ciphers and ephemeral key exchanges. | -| tls\_dhparam | Path to DH parameters for the server to use. | -| rand\_file | Path to a 2048 byte file containing entropy used to seed the PRNG. | -| keymgr\_runas | The user the keymgr process will run as. If not set, the current user. | -| keymgr\_root | The root path for the keymgr process. If not set, inherited from the root option. | -| acme\_runas | The user the acme process will run as. If not set, the current user. | -| acme\_root | The root path for the acme process. If not set, inherited from the root option. | -| acme\_email | An email address used for account registration. | -| acme\_provider | A URL to the directory for an ACME provider. Defaults to Let's Encrypt. | -| pledge | OpenBSD only, pledge categories for the worker processes. | -| seccomp\_tracing | Linux only, seccomp violations will be logged and not cause the process to terminate. Either "yes" or "no". | -| filemap\_ext | The default extension for files in a filemap. | -| filemap\_index | The root file in a filemap. (eg index.html). | -| http\_keepalive\_time | The time an HTTP connection is kept-alive server side. Defaults to 20 seconds. | -| http\_media\_type | Add a new HTTP media type (in the form of "mediatype ext1 ext2 ext"). | -| http\_header\_max | The maximum number of bytes HTTP headers can consist of. If a request comes in with headers larger than this the connection is closed. Defaults to 4096 bytes. | -| http\_header\_timeout | The number of seconds after which Kore will close a connection if no HTTP headers were received. Defaults to 10. | -| http\_body\_max | The maximum number of bytes an HTTP body can consist of. If a request comes in with a body larger than this the connection is closed with a 413 response. Defaults to 1MB. | -| http\_body\_timeout | The number of seconds after which Kore will close a connection if no HTTP body was received in full. Defaults to 60. | -| http\_body\_disk\_offload | The number in bytes from which point Kore will offload incoming HTTP bodies onto a file on disk instead of keeping it in memory. Disabled by default. | -| http\_body\_disk\_path | A path where the temporary body files are written if the http\_body\_disk\_offload setting is enabled. | -| http\_server\_version | Allows you to override the Kore server header. | -| http\_pretty\_error | If set to "yes" will display HTML based HTTP error codes. Defaults to "no". | +Please see https://github.com/jorisvink/kore/blob/master/conf/kore.conf.example for all configuration options and what they do. diff --git a/applications/privsep.md b/applications/privsep.md @@ -0,0 +1,38 @@ +# Privilege separation + +Unless explicitly started to not do so, Kore will force each of +its processes to run under different users and chroots. + +There are 3 process types that can be configured for privsep: + + * worker + * keymgr + * acme + +An example on configuring the worker processes to run as user kore, +with a chroot under /var/chroot/kore. + +``` +privsep worker { + # The user the workers will run as. + runas kore + + # The root directory for the worker processes, if chroot isn't + # skipped, this is the directory it will chroot into. + # + # If not set, Kore will take the current working directory. + root /var/chroot/kore + + # We could configure this process to not chroot and only + # chdir into its root directory. + #skip chroot +} +``` + +### Random + +**Important** If you are running Kore chrooted and privilege separated (which +you **should** be doing production), Kore will require /dev/urandom to be +created under the chroot environment for both the keymgr and worker processes. + +Failing to do so will prevent your application from working. diff --git a/applications/routes.md b/applications/routes.md @@ -8,10 +8,17 @@ Routes can either contain a fixed string of be a regular expression. ``` domain * { - route / root_page - route /about/ about_page + route / { + handler root_page + } - route ^.*$ redirect + route /about/ { + handler about_page + } + + route ^.*$ { + handler redirect + } } ``` @@ -26,26 +33,26 @@ in the configuration **and** specify how they should be validated. Validation is not optional. -After having defined the route you would continue with a param block: - +You define how to validate the parameters inside of the route context. ``` # First we must define a validator (in the global config) validator v_number regex ^[0-9]$ -# Then inside the domain {} configuration we can add the parameter block -# on an existing route: -params qs:get / { - validate id v_number +domain * { + route / { + handler root_page + methos get post + validate qs:get id v_number + validate post id v_number + } } ``` -The params block syntax is as follows: +The validate syntax is as follows: ``` -params method route { - validate parameter validator -} +validate method parameter validator ``` Where **method** is the lowercase method (eg: post, get) and optionally diff --git a/applications/running.md b/applications/running.md @@ -15,6 +15,15 @@ This method is aimed at running in production. You can skip chroot\(\) and privdrops using -n and -r. +## Startup sequence + +When Kore starts it will spawn several new processes such as the worker +processes, the keymgr process and acme process (if required). + +Kore will check that each process starts properly within a certain +time frame. If a process fails to start Kore will log the last output +of this process and terminate itself. + ### Random **Important** If you are running Kore chrooted and privilege separated (which diff --git a/book.json b/book.json @@ -1,7 +1,7 @@ { "author": "Joris Vink", - "title": "Kore 4.1.0 documentation", - "description": "The documentation for Kore 4.1.0", + "title": "Kore 4.2.0 documentation", + "description": "The documentation for Kore 4.2.0", "plugins": [ "highlight" diff --git a/install.md b/install.md @@ -4,22 +4,15 @@ Kore has been tested to run on the following platforms: -* Linux 3.2 or newer -* OpenBSD 5.3 or newer -* FreeBSD 10 or newer -* macOS 10.10.x or newer +* Linux +* OpenBSD +* FreeBSD +* macOS -Basic compilation requires the following libraries: - -* OpenSSL 1.1.0, 1.1.1 / LibreSSL - -Get the 4.1.0 release tarball and signature from [https://kore.io/releases/4.1.0](https://kore.io/releases/4.1.0) and verify it using minisign: +Get the 4.2.0 release tarball and signature from [https://kore.io/releases/4.2.0](https://kore.io/releases/4.2.0) and verify it using minisign: ``` -minisign -Vm kore-4.0.0.tgz -P RWSxkEDc2y+whfKTmvhqs/YaFmEwblmvar7l6RXMjnu6o9tZW3LC0Hc9 - - - +minisign -Vm kore-4.2.0.tgz -P RWSxkEDc2y+whfKTmvhqs/YaFmEwblmvar7l6RXMjnu6o9tZW3LC0Hc9 ``` If verification is successful, build it. Do not build distributions that @@ -33,32 +26,33 @@ $ sudo make install Kore has several build flavors available: -* CURL=1 \(compiles in asynchronous curl support\) -* ACME=1 \(compiles in ACME support\) -* TASKS=1 \(compiles in task support\) -* PGSQL=1 \(compiles in pgsql support\) -* DEBUG=1 \(enables use of -d for debug\) -* NOHTTP=1 \(compiles Kore without HTTP support\) -* NOOPT=1 \(disable compiler optimizations\) -* JSONRPC=1 \(compiles in JSONRPC support\) -* PYTHON=1 \(compiles in Python support\) +* ACME=1 (compiles in ACME support) +* CURL=1 (compiles in asynchronous curl support) +* TASKS=1 (compiles in task support) +* PGSQL=1 (compiles in pgsql support) +* DEBUG=1 (enables use of -d for debug) +* NOHTTP=1 (compiles Kore without HTTP support) +* NOOPT=1 (disable compiler optimizations) +* JSONRPC=1 (compiles in JSONRPC support) +* PYTHON=1 (compiles in the Python support) +* TLS_BACKEND=none (compiles Kore without any TLS backend) These build flavors can be passed on the command line when building. Note that enabling these flavors may require additional libraries to be present on your system: -Requirements for asynchronous libcurl support \(optional\): - -* libcurl +Base requirements +* openssl 1.1.1 or libressl 3.x +(note: openssl 3.0.0 is currently *not* supported) -Requirements for background tasks \(optional\) +Requirement for asynchronous curl (optional) +* libcurl (7.64.0 or higher) +Requirements for background tasks (optional) * pthreads -Requirements for pgsql \(optional\) - +Requirements for pgsql (optional) * libpq -Requirements for python \(optional\) - +Requirements for python (optional) * Python 3.6+ For BSD-like systems you will need to install GNU Make.