kore-doc

The kore documentation found under https://docs.kore.io/
Commits | Files | Refs | README | git clone https://git.kore.io/kore-doc.git

commit 580865e6c01cc82b68843b04156452507cf214f6
Author: Joris Vink <joris@coders.se>
Date:   Fri, 29 Jun 2018 10:31:13 +0200

initial kore-doc commit.

Diffstat:
.gitignore | 17+++++++++++++++++
README.md | 31+++++++++++++++++++++++++++++++
SUMMARY.md | 21+++++++++++++++++++++
api/README.md | 4++++
api/buffers.md | 169+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/http.md | 582++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/memory.md | 122+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/miscellaneous.md | 125+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/pgsql.md | 314+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/pools.md | 80+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/tasks.md | 152+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
api/websockets.md | 103+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
applications/README.md | 4++++
applications/buildconf_reference.md | 37+++++++++++++++++++++++++++++++++++++
applications/building.md | 36++++++++++++++++++++++++++++++++++++
applications/integrated_tools.md | 74++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
applications/koreconf.md | 88+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
applications/running.md | 21+++++++++++++++++++++
applications/the-pyko-tool.md | 6++++++
arch.png | 0
chapter1.md | 68++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
examples.md | 5+++++
tutorials/tutorials.md | 6++++++
23 files changed, 2065 insertions(+), 0 deletions(-)

diff --git a/.gitignore b/.gitignore @@ -0,0 +1,16 @@ +# Node rules: +## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) +.grunt + +## Dependency directory +## Commenting this out is preferred by some people, see +## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git +node_modules + +# Book build output +_book + +# eBook build output +*.epub +*.mobi +*.pdf+ \ No newline at end of file diff --git a/README.md b/README.md @@ -0,0 +1,31 @@ +# Kore web framework + +Kore is a web application framework written in C that allows you to create blazing fast web applications. It focuses on security, stability and rapid development. + +The latest version is [3.0.0](https://github.com/jorisvink/kore/releases) and was released DD/MM/2018 \(release is upcoming\). + +This gitbook serves as the main documentation for the project. + +The documentation is correct for the latest 3.0.0 release but may not be correct for the latest master branch on github. + +# Features + +* Supports SNI +* Supports HTTP/1.1 +* Websocket support +* Privseps by default +* TLS enabled by default +* Optional background tasks +* Built-in parameter validation +* Optional asynchronous PostgreSQL support +* Optional support for page handlers in Python +* Private keys isolated in separate process \(RSA and ECDSA\) +* Default sane TLS ciphersuites \(PFS in all major browsers\) +* Modules can be reloaded on-the-fly, even while serving content +* Event driven \(epoll/kqueue\) architecture with per CPU worker processes +* Build your web application as a precompiled dynamic library or single binary + +# Architecture overview + +![](arch.png) + diff --git a/SUMMARY.md b/SUMMARY.md @@ -0,0 +1,21 @@ +# Summary + +* [Introduction](README.md) +* [Installation](chapter1.md) +* [Applications](applications/README.md) + * [The kodev tool](applications/integrated_tools.md) + * [Build framework](applications/building.md) + * [Running](applications/running.md) + * [build.conf](applications/buildconf_reference.md) + * [kore.conf](applications/koreconf.md) +* [API](api/README.md) + * [Buffers](api/buffers.md) + * [HTTP](api/http.md) + * [Pools](api/pools.md) + * [Memory](api/memory.md) + * [Miscellaneous](api/miscellaneous.md) + * [Tasks](api/tasks.md) + * [Pgsql](api/pgsql.md) + * [Websockets](api/websockets.md) +* [Examples](examples.md) + diff --git a/api/README.md b/api/README.md @@ -0,0 +1,3 @@ +# API + +Kore provides several APIs for you to do your bidding. You can find their available functions and descriptions in the table of contents to the left below "API".+ \ No newline at end of file diff --git a/api/buffers.md b/api/buffers.md @@ -0,0 +1,169 @@ +# Buffers + +The kore_buf interface provides a safe way of constructing data. Buffers automatically grow when more data is written to them and can either exist on the heap or on the stack. + +--- + +# kore_buf_alloc +### Synopsis +``` +struct kore_buf *kore_buf_alloc(size_t initial) +``` +### Description +Allocates a new *kore_buf* data structure on the heap and initializes it. + +| Parameter | Description | +| -- | -- | +| initial | The number of initial bytes to allocate for the buffer data. Can be 0. | + +### Returns +A pointer to a new *kore_buf* data structure. This must be freed by the caller using *kore_buf_free()*. + +--- + +# kore_buf_init +### Synopsis +``` +void kore_buf_init(struct kore_buf *buf, size_t initial) +``` +### Description +Initializes the given buffer. + +| Parameter | Description | +| -- | -- | +| buf | The buffer to initialize. | +| initial | The number of initial bytes to allocate for the buffer data. Can be 0. | + +### Returns +Nothing. + +--- + +# kore_buf_cleanup +### Synopsis +``` +void kore_buf_cleanup(struct kore_buf *buf) +``` +### Description +Cleanup the given buffer by releasing the buffer data and resetting its members. + +| Parameter | Description | +| -- | -- | +| buf | The buffer to be cleaned up. | + +### Returns +Nothing + +--- + +# kore_buf_free +### Synopsis +``` +void kore_buf_free(struct kore_buf *buf) +``` +### Description +Cleanup and free a previously allocated buffer. + +| Parameter | Description | +| -- | -- | +| buf | The buffer to be cleaned up and freed. | + +### Returns +Nothing + +--- + +# kore_buf_append +### Synopsis +``` +void kore_buf_append(struct kore_buf *buf, const void *data, size_t length) +``` +### Description +Append some data to a buffer. This function will automatically grow the buffer data if required before appending the given data. + +| Parameter | Description | +| -- | -- | +| buf | The buffer. | +| data | The data to be appended. | +| length | The length of the data to be appended .| + +### Returns +Nothing + +--- + +# kore_buf_appendv +### Synopsis +``` +void kore_buf_appendv(struct kore_buf *buf, const char *fmt, va_list args) +``` +### Description +Append data given in the form of a format and a variadic argument list. + +| Parameter | Description | +| -- | -- | +| buf | The buffer. | +| fmt | The format string. | +| args | Variadic argument list. | + + +### Returns +Nothing + +--- + +# kore_buf_appendf +### Synopsis +``` +void kore_buf_appendf(struct kore_buf *buf, const char *fmt, ...) +``` +### Description +Append data given in the form of a string format and list of arguments. Much like snprintf(). + +| Parameter | Description | +| -- | -- | +| buf | The buffer. | +| fmt | The format string. | +| ... | Any arguments for the format string. | + +### Returns +Nothing + +--- + +# kore_buf_stringify +### Synopsis +``` +char *kore_buf_stringify(struct kore_buf *buf, size_t *length) +``` +### Description +Returns the data in a buffer as a C string. The result is NUL-terminated before it is returning to the caller. The caller should not free this data. + +| Parameter | Description | +| -- | -- | +| buf | The buffer. | +| length | optional, if not NULL the length of the C string is placed here. | + +### Returns +A pointer to the C string. + +--- + +# kore_buf_release +### Synopsis +``` +u_int8_t *kore_buf_release(struct kore_buf *buf, size_t *length) +``` +### Description +Release the buffer data from a buffer. This causes the buffer to be cleaned up. + +| Parameter | Description | +| -- | -- | +| buf | The buffer. | +| length | The length of the returned data is placed here. | + +### Returns +A pointer to the original buffer data. + +--- + diff --git a/api/http.md b/api/http.md @@ -0,0 +1,582 @@ +# HTTP + +This page contains all available public functions related to understanding and responding to HTTP requests. + +--- + +# http\_response + +### Synopsis + +``` +void http_response(struct http_request *req, int status, const void *data, size_t length) +``` + +### Description + +Creates an HTTP response for an HTTP request. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to respond to. | +| status | The HTTP status code to include in the response. | +| data | The data to be sent in the response body. | +| length | The length of the data to be sent. | + +### Returns + +Nothing + +--- + +# http\_response\_stream + +### Synopsis + +``` +void http_response_stream(struct http_request *req, int status, void *base, size_t length, int (*cb)(struct netbuf *), void *arg) +``` + +### Description + +Creates an HTTP response for an HTTP request much like _http\_response\(\)_. + +However unlike that function this one does not copy the response body data but rather stream from it. It will call the given callback _cb_ when all data has been sent. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to respond to. | +| status | The HTTP status code to include in the response. | +| base | The base pointer of the data tobe sent in the response body. | +| length | The length of the data to be sent. | +| cb | A callback that is called when all data has been sent. | +| arg | A user supplied argument that is passed in the callback. | + +### Returns + +Nothing + +--- + +# http\_request\_header + +### Synopsis + +``` +int http_request_header(struct http_request *req, const char *header, const char **out) +``` + +### Description + +Attempts to find the given _header_ in an HTTP request and returns the value of the header in the _out_ parameter. The returned pointer should not be freed. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| header | The name of the header to find. | +| out | Pointer to where the pointer to the result is stored. | + +### Returns + +KORE\_RESULT\_OK if a result was set in _out_. +KORE\_RESULT\_ERROR if the header was not present in the request. + +--- + +# http\_file\_lookup + +### Synopsis + +``` +struct http_file *http_file_lookup(struct http_request *req, const char *name) +``` + +### Description + +Lookup a file that was uploaded as part of a multipart form submission. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the form field the file was sent as. | + +### Returns + +A pointer to an _http\_file_ data structure containing information about the uploaded file. + +Will return NULL if the file could not be found. + +### Note + +You must call _http\_populate\_multipart\_form\(\)_ before this function will return any result at all. + +--- + +# http\_file\_read + +### Synopsis + +``` +ssize_t http_file_read(struct http_file *file, void *buf, size_t length) +``` + +### Description + +Read file data from the given file up to _length_ size. + +| Parameter | Description | +| --- | --- | +| file | The file from which to read. | +| buf | Where the file data is copied into. | +| length | The maximum length that can be copied into _buf_. | + +### Returns + +Returns the number of bytes successfully read from the file or 0 on end of file or -1 on error. + +--- + +# http\_file\_rewind + +### Synopsis + +``` +void http_file_rewind(struct http_file *file) +``` + +### Description + +Sets the offset member of the given _file_ data structure back to 0 for sequential reads. + +| Parameter | Description | +| --- | --- | +| file | The file to rewind. | + +### Returns + +Nothing + +--- + +# http\_populate\_post + +### Synopsis + +``` +void http_populate_post(struct http_request *req) +``` + +### Description + +Processes an HTTP POST by taking the HTTP body and parsing it according to _application/x-www-form-urlencoded_. + +This function will automatically match any fields found against configured validators to check if they contained sensible date. If the validators fail the field is automatically removed. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to parse. | + +### Returns + +Nothing + +--- + +# http\_populate\_qs + +### Synopsis + +``` +void http_populate_qs(struct http_request *req) +``` + +### Description + +The same as _http\_populate\_post\(\)_ but for query string parameters instead. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to parse. | + +### Returns + +Nothing + +--- + +# http\_populate\_multipart\_form + +### Synopsis + +``` +void http_populate_multipart_form(struct http_request *req) +``` + +### Description + +Parses a multipart form that was received via a POST request. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request to parse. | + +### Returns + +Nothing + +--- + +# http\_body\_read + +### Synopsis + +``` +ssize_t http_body_read(struct http_request *req, void *out, size_t length) +``` + +### Description + +Attempts to read data from the HTTP body received in a request. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| out | Where the data is copied into. | +| length | The maximum number of bytes that will fit in _out_. | + +### Returns + +Returns the number of bytes copied or 0 on end of body or -1 on error. + +--- + +# http\_state\_run + +### Synopsis + +``` +int http_state_run(struct http_state *states, u_int8_t elm, struct http_request *req) +``` + +### Description + +Runs an HTTP state machine. + +| Parameter | Description | +| --- | --- | +| states | The HTTP state machine to be run. | +| elm | The number of elements in this state machine. | +| req | The HTTP request to be passed to the state machine functions. | + +### Returns + +* KORE\_RESULT\_OK +* KORE\_RESULT\_ERROR +* KORE\_RESULT\_RETRY + +### Note + +This should be called from a page handler. + +--- + +# http\_status\_text + +### Synopsis + +``` +const char *http_status_text(int status) +``` + +### Description + +Returns a pointer to a human readable string for the given HTTP status code. + +| Parameter | Description | +| --- | --- | +| status | The HTTP status code for which to find the string. | + +### Returns + +A pointer to a human readable string for the HTTP status code. + +--- + +# http\_method\_text + +### Synopsis + +``` +const char *http_method_text(int method) +``` + +### Description + +Returns a pointer to a human readable string for the given HTTP method. + +| Parameter | Description | +| --- | --- | +| method | The HTTP method for which to find the matching string. | + +### Returns + +A pointer to a human readable string for the HTTP method. + +--- + +# http\_argument\_get\_string + +### Synopsis + +``` +int http_argument_get_string(struct http_request *req, const char *name, char **out) +``` + +### Description + +Lookup an argument as a string. The caller should not free the result. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the pointer to the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_byte + +### Synopsis + +``` +int http_argument_get_byte(struct http_request *req, const char *name, u_int8_t *out) +``` + +### Description + +Lookup an argument as a byte. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_int16 + +### Synopsis + +``` +int http_argument_get_int16(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 16-bit signed integer. This function will check that the result fits in a 16-bit signed integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_uint16 + +### Synopsis + +``` +int http_argument_get_uint16(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 16-bit unsigned integer. This function will check that the result fits in a 16-bit unsigned integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_int32 + +### Synopsis + +``` +int http_argument_get_int32(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 32-bit signed integer. This function will check that the result fits in a 32-bit signed integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_uint32 + +### Synopsis + +``` +int http_argument_get_uint32(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 32-bit unsigned integer. This function will check that the result fits in a 32-bit unsigned integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_int64 + +### Synopsis + +``` +int http_argument_get_int64(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 64-bit signed integer. This function will check that the result fits in a 64-bit signed integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_uint64 + +### Synopsis + +``` +int http_argument_get_uint64(struct http_request *req, const char *name, int16_t *out) +``` + +### Description + +Lookup an argument as a 64-bit unsigned integer. This function will check that the result fits in a 64-bit unsigned integer before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_float + +### Synopsis + +``` +int http_argument_get_uint64(struct http_request *req, const char *name, float *out) +``` + +### Description + +Lookup an argument as a float. This function will check that the result fits in a float before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# http\_argument\_get\_double + +### Synopsis + +``` +int http_argument_get_uint64(struct http_request *req, const char *name, double *out) +``` + +### Description + +Lookup an argument as a double. This function will check that the result fits in a double before returning it. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| name | The name of the argument to find. | +| out | Where the result is stored. | + +### Returns + +KORE\_RESULT\_OK if the argument was found or KORE\_RESULT\_ERROR if it was not found. + +--- + +# Data structures + +### http\_request + +### http\_file + +### http\_header + +### http\_arg + +### http\_state + +--- + + + diff --git a/api/memory.md b/api/memory.md @@ -0,0 +1,122 @@ +# Memory + +The memory system in Kore for heap allocation is based on pools. At startup Kore will initialize at least 10 pools for commonly sized objects \(ranging from 8 to 8192 bytes\). + +Any data allocated via _kore\_malloc\(\)_, _kore\_strdup\(\)_ or _kore\_realloc\(\)_ comes from the common pools unless it is larger then 8192 in which case _calloc\(\)_ is used. + +--- + +# kore\_malloc + +### Synopsis + +``` +void *kore_malloc(size_t length) +``` + +### Description + +Allocates new data. + +| Parameter | Description | +| --- | --- | +| length | The length of the data to allocate. | + +### Returns + +A pointer to allocated memory that can hold _length_ bytes. The returned pointer is aligned for use with any data type. + +--- + +# kore\_calloc + +### Synopsis + +``` +void *kore_calloc(size_t memb, size_t len) +``` + +### Description + +Allocates new data and zeroes it out. + +| Parameter | Description | +| --- | --- | +| memb | Number of objects. | +| len | Size of each object. | + +### Returns + +A pointer to allocated memory that can hold _ memb \* len_ bytes. The returned pointer is aligned for use with any data type and the contents is zeroed out. + +--- + +# kore\_realloc + +### Synopsis + +``` +void *kore_realloc(void *ptr, size_t length) +``` + +### Description + +Grows or shrinks a existing allocated memory pointer. + +| Parameter | Description | +| --- | --- | +| ptr | The pointer to reallocate. | +| length | The new length of the data. | + +### Returns + +A pointer to the newly allocated memory. + +--- + +# kore\_free + +### Synopsis + +``` +void kore_free(void *ptr) +``` + +### Description + +Frees a previously allocated pointer. + +| Parameter | Description | +| --- | --- | +| ptr | The pointer to be freed. | + +### Returns + +Nothing + +--- + +# kore\_strdup + +### Synopsis + +``` +char *kore_strdup(const char *str) +``` + +### Description + +Duplicate an existing C string. + +| Parameter | Description | +| --- | --- | +| str | The C string to be duplicated. | + +### Returns + +A pointer to the duplicated C string. + +--- + + + diff --git a/api/miscellaneous.md b/api/miscellaneous.md @@ -0,0 +1,124 @@ +# Miscellaneous + +Kore provides several helper functions. + +--- + +# kore_log +### Synopsis +``` +void kore_log(int prio, const char *fmt, ...) +``` +### Description +Output a log message via syslog. + +Note that when running in the foreground these go to stdout. + +| Parameter | Description | +| -- | -- | +| prio | The priority of the log message (see syslog(3)). | +| fmt | The format string. | +| ... | List of arguments. | + +### Returns +Nothing + +--- + +# kore_strlcpy +### Synopsis +``` +size_t kore_strlcpy(char *dst, const char *src, const size_t length) +``` +### Description +Bounded C string copying. The result is always NUL-terminated. + +| Parameter | Description | +| -- | -- | +| dst | The destination buffer. | +| src | The source C string. | +| length | The maximum number of bytes that the destination buffer holds. | + +### Returns +The length of the original string. If this length is equal or larger then the destination buffer then truncation of the string has occurred. + +--- + +# kore_strtonum +### Synopsis +``` +long long kore_strtonum(const char *str, int base, long long min, long long max, int *err) +``` +### Description +Safely convert a C string holding a number into an integer. + +| Parameter | Description | +| -- | -- | +| str | The C string to convert. | +| base | The base on which to operate. | +| min | The minimum value the converted integer is allowed to have. | +| max | The maximum value the converted integer is allowed to have. | +| err | A pointer to an integer that holds the error value. | + +### Returns +Returns the converted integer. If the *err* parameter was set to KORE_RESULT_OK the conversion went OK otherwise there was an error and the returned value is considered garbage. + +--- + +# kore_strtonum64 +### Synopsis +``` +u_int64_t kore_strtonum64(const char *str, int sign, int *err) +``` +### Description +Safely convert a C string holding a number to a 64-bit integer. + +| Parameter | Description | +| -- | -- | +| str | The C string to convert. | +| sign | If the 64-bit integer is allowed to be signed or not. | +| err | A pointer to an integer that holds the error value. | + +### Returns +Returns the converted integer. If the *err* parameter was set to KORE_RESULT_OK the conversion went OK otherwise there was an error and the returned value is considered garbage. + +--- + +# kore_split_string +### Synopsis +``` +int kore_split_string(char *input, char *delim, char **out, size_t ele) +``` +### Description +Safely split a string into several elements. The *out* array is NULL terminated. + +| Parameter | Description | +| -- | -- | +| input | The string to be split up. | +| delim | The delimiter used for splitting. | +| out | Pointer to an array where the components are stored. | +| ele | The number of elements that can be held in *out*. | + +### Returns +The number of components. + +--- + +# kore_strip_chars +### Synopsis +``` +void kore_strip_chars(char *in, const char strip, char **out) +``` +### Description +Safely strip all occurences of a certain character from a C string. The result must be freed by the caller. + +| Parameter | Description | +| -- | -- | +| in | The input C string. | +| strip | The character to be stripped out of the string. | +| out | A pointer to where the new result is stored. | + +### Returns +Nothing + +---+ \ No newline at end of file diff --git a/api/pgsql.md b/api/pgsql.md @@ -0,0 +1,314 @@ +# Pgsql + +The pgsql API in Kore allows you to use Postgresql in a straight forward manner. It supports both synchronous and asynchronous queries. + +Note that you must have built Kore with PGSQL=1 in order to use this API. + +--- + +# kore\_pgsql\_init + +### Synopsis + +``` +void kore_pgsql_init(struct kore_pgsql *pgsql) +``` + +### Description + +Initialize a kore\_pgsql data structure for use. + +| Parameter | Description | +| --- | --- | +| pgsql | A pgsql data structure. | + +### Returns + +Nothing. + +--- + +# kore\_pgsql\_setup + +### Synopsis + +``` +int kore_pgsql_query_init(struct kore_pgsql *pgsql, const char *dbname, int flags) +``` + +### Description + +Setup a pgsql data structure for use. + +| Parameter | Description | +| --- | --- | +| pgsql | A pgsql data structure. | +| dbname | The name of the database connection to be used. | +| flags | Any additional flags. | + +| Flag | Meaning | +| --- | --- | +| KORE\_PGSQL\_ASYNC | Setup an asynchronous query. | +| KORE\_PGSQL\_SYNC | Setup a synchronous query. | + +### Returns + +KORE\_RESULT\_OK or KORE\_RESULT\_ERROR. If KORE\_RESULT\_ERROR was returned but the _pgsql-&gt;state_ is still set to _KORE\_PGSQL\_STATE\_INIT_ then you can try again. + +--- + +# kore\_pgsql\_bind\_request + +### Synopsis + +``` +void kore_pgsql_bind_request(struct kore_pgsql *pgsql, struct http_request *req) +``` + +### Description + +Bind a kore\_pgsql data structure to an HTTP request. This causes the HTTP request to be notified for any changes to the pgsql +data structure. + +| Parameter | Description | +| --- | --- | +| pgsql | A pgsql data structure. | +| req | The HTTP request to be tied to. | + +### Returns + +Nothing. + +--- + +# kore\_pgsql\_bind\_callback + +### Synopsis + +``` +void kore_pgsql_bind_callback(struct kore_pgsql *pgsql, void (*cb)(struct kore_pgsql *, void *), void *ar) +``` + +### Description + +Bind a kore\_pgsql data structure to a callback function. This function is called for any state change related to the +pgsql data structure. + +| Parameter | Description | +| --- | --- | +| pgsql | A pgsql data structure. | +| cb | Pointer to a C function. | + +### Returns + +Nothing. + +--- + +# kore\_pgsql\_query + +### Synopsis + +``` +int kore_pgsql_query(struct kore_pgsql *pgsql, const char *query) +``` + +### Description + +Fires of a query. + +| Parameter | Description | +| --- | --- | +| pgsql | A previously initialized pgsql data structure. | +| query | The query to be sent. | + +### Returns + +KORE\_RESULT\_OK or KORE\_RESULT\_ERROR. + +--- + +# kore\_pgsql\_register + +### Synopsis + +``` +int kore_pgsql_register(const char *dbname, const char *connstring) +``` + +### Description + +Register a database connection. + +| Parameter | Description | +| --- | --- | +| dbname | The name to give to this connection. | +| connstring | A postgresql connection string. | + +### Returns + +KORE\_RESULT\_OK or KORE\_RESULT\_ERROR if the dbname was already taken. + +--- + +# kore\_pgsql\_continue + +### Synopsis + +``` +void kore_pgsql_continue(struct http_request *req, struct kore_pgsql *pgsql) +``` + +### Description + +Continue the asynchronous state machine. Usually called from HTTP state machines or callback functions. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request we are dealing with. | +| pgsql | The pgsql we are dealing with. | + +### Returns + +Nothing + +--- + +# kore\_pgsql\_cleanup + +### Synopsis + +``` +void kore_pgsql_cleanup(struct kore_pgsql *pgsql) +``` + +### Description + +Cleanup a previously initialize pgsql data structure and relinquish the database connection it held. + +| Parameter | Description | +| --- | --- | +| | | + +### Returns + +Nothing + +--- + +# kore\_pgsql\_logerror + +### Synopsis + +``` +void kore_pgsql_logerror(struct kore_pgsql *pgsql) +``` + +### Description + +Log what error occurred to syslog with a LOG\_NOTICE priority. + +| Parameter | Description | +| --- | --- | +| pgsql | The pgsql data structure. | + +### Returns + +Nothing + +--- + +# kore\_pgsql\_ntuples + +### Synopsis + +``` +int kore_pgsql_ntuples(struct kore_pgsql *pgsql) +``` + +### Description + +Returns the number of tuples affected by a query. + +| Parameter | Description | +| --- | --- | +| pgsql | The pgsql data structure. | + +### Returns + +The number of tuples. + +--- + +# kore\_pgsql\_nfields + +### Synopsis + +``` +int kore_pgsql_nfields(struct kore_pgsql *pgsql) +``` + +### Description + +Returns the number of fields in the result. + +| Parameter | Description | +| --- | --- | +| pgsql | The pgsql data structure. | + +### Returns + +The number of fields. + +--- + +# kore\_pgsql\_getlength + +### Synopsis + +``` +int kore_pgsql_getlength(struct kore_pgsql *pgsql, int row, int col) +``` + +### Description + +Returns the length of the data at row, col. + +| Parameter | Description | +| --- | --- | +| pgsql | The pgsql data structure. | +| row | The row number. | +| col | The column number. | + +### Returns + +The length of the data at row, col. + +--- + +# kore\_pgsql\_getvalue + +### Synopsis + +``` +char *kore_pgsql_getvalue(struct kore_pgsql *pgsql, int row, int col) +``` + +### Description + +Returns the value of the data at row, col. + +| Parameter | Description | +| --- | --- | +| pgsql | The pgsql data structure. | +| row | The row number. | +| col | The column number. | + +### Returns + +A pointer to the data. This should not be freed by the caller. + +--- + + + diff --git a/api/pools.md b/api/pools.md @@ -0,0 +1,79 @@ +# Pools + +Kore provides a memory pool interface allowing you to create your own. + +Pools are automatically thread-safe if Kore was built with TASKS=1. + +--- + +# kore_pool_init +### Synopsis +``` +void kore_pool_init(struct kore_pool *pool, const char *name, size_t len, size_t elm) +``` +### Description +Initializes a new pool. + +| Parameter | Description | +| -- | -- | +| pool | A pointer to a pool. | +| name | The name to give to the pool. This name is shown in the logs if the pool is exhausted. | +| len | The size in bytes of each element. | +| elm | The number of initial elements to preallocate to the pool. | + +### Returns +Nothing + +--- + +# kore_pool_cleanup +### Synopsis +``` +void kore_pool_cleanup(struct kore_pool *pool) +``` +### Description +Deallocate and cleanup a pool. + +| Parameter | Description | +| -- | -- | +| pool | A pointer to a pool. | + +### Returns +Nothing + +--- + +# kore_pool_get +### Synopsis +``` +void *kore_pool_get(struct kore_pool *pool) +``` +### Description +Obtain a pointer to a free element from the given pool. If a pool runs out of free elements it is automatically grown in size. + +| Parameter | Description | +| -- | -- | +| pool | A pointer to a pool. | + +### Returns +Returns a pointer to an area that is large enough to hold the data length the pool was initializes with. + +--- + +# kore_pool_put +### Synopsis +``` +void kore_pool_put(struct kore_pool *pool, void *ptr) +``` +### Description +Returns the given pointer back to its pool. + +| Parameter | Description | +| -- | -- | +| pool | A pointer to a pool. | +| ptr | The pointer to be returned to the pool. | + +### Returns +Nothing + +---+ \ No newline at end of file diff --git a/api/tasks.md b/api/tasks.md @@ -0,0 +1,151 @@ +# Tasks + +The tasks API in Kore allows you to create background tasks which are scheduled over OS threads. + +Background tasks can communicate with the main thread via a socket pair. + +Kore must be built with TASKS=1 in order to use this API. + +--- + +# kore_task_create +### Synopsis +``` +void kore_task_create(struct kore_task *t, int (*entry)(struct kore_task *)) +``` +### Description +Creates a new task which when run will call the given callback. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | +| entry | The entry point of the task when run. | + +### Returns +Nothing + +--- +# kore_task_run +### Synopsis +``` +void kore_task_run(struct kore_task *t) +``` +### Description +Allows a task to be run. After this function returns the task will be scheduled to run. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | + +### Returns +Nothing + +--- +# kore_task_bind_request +### Synopsis +``` +void kore_task_bind_request(struct kore_task *t, struct http_request *req) +``` +### Description +Bind an HTTP request to a task. Binding means that the HTTP request will be scheduled to be called again by Kore if the task writes data on the task channel or when it completes. + +Using this in combination with the HTTP state machine allows you to build request handlers that use background tasks for heavy labor. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | +| req | The HTTP request to be bound to the task. | + +### Returns +Nothing + +--- +# kore_task_bind_callback +### Synopsis +``` +void kore_task_bind_callback(struct kore_task *t, void (*cb)(struct kore_task *)) +``` +### Description +Bind a callback to a task. Binding means that the callback will be called whenever the task writes data on the task channel or whenever it completes. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | +| cb | The callback to call. | + +### Returns +Nothing + +--- +# kore_task_destroy +### Synopsis +``` +void kore_task_destroy(struct kore_task *t) +``` +### Description +Destroys a task. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | + +### Returns +Nothing + +--- +# kore_task_finished +### Synopsis +``` +int kore_task_finished(struct kore_task *t) +``` +### Description +Check if a task has finished running. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | + +### Returns +Returns 1 if the task has finished running, otherwise 0. + +--- +# kore_task_channel_write +### Synopsis +``` +void kore_task_channel_write(struct kore_task *t, void *data, u_int32_t length) +``` +### Description +Write data on the task channel for the other end to read. This works both from the main thread and a task itself. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | +| data | The data to be written on the channel. | +| length | The length of the data to be written. | + +### Returns +Nothing + +--- + +# kore_task_channel_read +### Synopsis +``` +u_int32_t kore_task_channel_read(struct kore_task *t, void *out, u_int32_t length) +``` +### Description +Read data from the task channel. + +NOTE: This is a blocking operation. + +| Parameter | Description | +| -- | -- | +| t | A task data structure. | +| out | Where the data read will be written too. | +| length | The maximum number of bytes that *out* can hold. | + +### Returns +Returns the number of original bytes from the message, if this is larger then +the *length* parameter then truncation has occurred. + +---+ \ No newline at end of file diff --git a/api/websockets.md b/api/websockets.md @@ -0,0 +1,103 @@ +# Websockets + +Kore provides an easy to use interface for websockets. Once a connection has been upgraded to a websocket it will use user given callbacks for incoming websocket messages. + +The prototypes for the callbacks are as follows: + +``` +void onconnect(struct connection *c); +void onmessage(struct connection *c, u_int8_t op, const void *data, size_t length); +void ondisconnect(struct connection *); +``` + +--- + +# kore\_websocket\_handshake + +### Synopsis + +``` +void kore_websocket_handshake(struct http_request *req, const char *onconnect, + const char *onmessage, const char *ondisconnect) +``` + +### Description + +Performs an upgrade of an HTTP connection to a websocket connection. + +| Parameter | Description | +| --- | --- | +| req | The HTTP request. | +| onconnect | Name of the function that will be called when a new websocket client connects. | +| onmessage | Name of the function that will be called when a new websocket message arrives. | +| ondisconnect | Name of the function that will be called when a websocket client disconnects. | + +### Returns + +Nothing + +--- + +# kore\_websocket\_send + +### Synopsis + +``` +void kore_websocket_send(struct connection *c, u_int8_t op, const void *data, size_t length) +``` + +### Description + +Sends a websocket message to a client. + +| Parameter | Description | +| --- | --- | +| c | A client connection that has upgraded to websockets. | +| op | The websocket op type. | +| data | The data to be sent. | +| length | The length of the data to be sent. | + +| Websocket ops | +| --- | +| WEBSOCKET\_OP\_TEXT | +| WEBSOCKET\_OP\_BINARY | + +### Returns + +Nothing + +--- + +# kore\_websocket\_broadcast + +### Synopsis + +``` +void kore_websocket_broadcast(struct connection *src, u_int8_t op, const void *data, size_t length, int scope) +``` + +### Description + +Broadcasts data to all connected websocket clients. + +| Parameter | Description | +| --- | --- | +| src | Source client connection. | +| op | The websocket op type. | +| data | The data to be sent. | +| length | The length of the data to be sent. | +| scope | The scope of the broadcast \(worker only or all workers\). | + +| scope | +| --- | +| WEBSOCKET\_BROADCAST\_LOCAL | +| WEBSOCKET\_BROADCAST\_GLOBAL | + +### Returns + +Nothing + +--- + + + diff --git a/applications/README.md b/applications/README.md @@ -0,0 +1,3 @@ +# Applications + +The chapters in this section talk about Kore applications, how to create build and run them.+ \ No newline at end of file diff --git a/applications/buildconf_reference.md b/applications/buildconf_reference.md @@ -0,0 +1,36 @@ +# build.conf + +The **build.conf** file dictates build flavors, their CFLAGS and LDFLAGS and whether or not a single binary should be constructed or not. + +A default **build.conf** looks like this: + +``` +# integers build config +# You can switch flavors using: kore flavor [newflavor] + +# Set to yes if you wish to produce a single binary instead +# of a dynamic library. If you set this to yes you must also +# set kore_source together with kore_flavor and update ldflags +# to include the appropriate libraries you will be linking with. +#single_binary=no +#kore_source=/home/joris/src/kore +#kore_flavor= + +# The cflags below are shared between flavors +cflags=-Wall -Wmissing-declarations -Wshadow +cflags=-Wstrict-prototypes -Wmissing-prototypes +cflags=-Wpointer-arith -Wcast-qual -Wsign-compare + +dev { + # These cflags are added to the shared ones when + # you build the "dev" flavor. + cflags=-g +} + +#prod { +# You can specify additional CFLAGS here which are only +# included if you build with the "prod" flavor. +#} +``` + +There are global directives and per flavor directives as you can see in the example above. The cflags and ldflags directives are merged between global and the current active flavor.+ \ No newline at end of file diff --git a/applications/building.md b/applications/building.md @@ -0,0 +1,36 @@ +# Build framework + +Kore provides a flexible way of building your applications through the **conf/build.conf** file. + +This file contains build flavors that dictate CFLAGS and LDFLAGS. + +Flavors can be switched using _ kodev flavor_: + +``` +$ kodev flavor osx +changed build flavor to: osx +$ +``` + +### Sources + +Kore will automatically build any source files found under the **src** directory in your application. + +### Assets + +By default the build tool will automatically convert files found under **assets** into C files and compile them into your code allowing you to access these directly. + +### Single binaries + +Kore will normally produce a single dynamic library \(DSO\) file that contains your application logic. This DSO file is then loaded at run-time by the main Kore binary as instructed by your configuration file. + +If DSO's are not your cup-o-tea you can opt for building a single binary instead. This requires you to update your **conf/build.conf** file and add the following settings globally: + +``` +single_binary = yes +kore_source = /path/to/kore/source/ +kore_flavor = <flavors..> +``` + +This will cause _kore build_ to build the original Kore source code and link your application logic directly into it. It will also add in the configuration file and of course any assets that are under the **assets** directory. The result is a single dynamically linked binary that can be run. + diff --git a/applications/integrated_tools.md b/applications/integrated_tools.md @@ -0,0 +1,74 @@ +# The kodev tool + +Kore provides a development tool called **kodev** to help you build and create applications. + +``` +kodev create +kodev build +kodev run +kodev flavor +kodev info +``` + +### Creating an application + +Creating a new application is done via the _ kodev create_ tool. This tool will create a new directory and populated it with all required files to get hacking. + +The tool will automatically generate a self-signed X.509 certificate for development purposes. + +``` +$ kodev create hello +created hello/src/hello.c +created hello/conf/hello.conf +created hello/conf/build.conf +created hello/.gitignore +created hello/dh2048.pem +hello created successfully! +note: do NOT use the created DH parameters/certificates in production +$ +``` + +### Building an application + +You can build a Kore application using the _ kodev build_ tool. + +This tool will read your **conf/build.conf** file and build your application according to it. + +``` +$ kodev build +building hello (dev) +CFLAGS=-Wall -Wmissing-declarations -Wshadow -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wsign-compare -fPIC -I./src -I./src/includes -I/usr/local/include -I/opt/local/include -I/usr/local/opt/openssl/include -g +LDFLAGS=-dynamiclib -undefined suppress -flat_namespace +compiling hello.c +hello built successfully! +$ +``` + +### Running an application + +You can run a Kore application in the foreground using the _ kodev run_ tool. + +This tool will simply build the application \(if any building needs to happen\) and run it in the foreground. You can CTRL-C it to bring it back down. + +``` +$ kodev run +building hello (dev) +CFLAGS=-Wall -Wmissing-declarations -Wshadow -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wsign-compare -fPIC -I./src -I./src/includes -I/usr/local/include -I/opt/local/include -I/usr/local/opt/openssl/include -g +LDFLAGS=-dynamiclib -undefined suppress -flat_namespace +nothing to be done! +[parent]: running on https://127.0.0.1:8888 +[parent]: kore is starting up +[keymgr]: key manager started +[wrk 1]: worker 1 started (cpu#1) +^C[keymgr]: cleaning up keys +[keymgr]: parent gone, shutting down +[wrk 1]: parent gone, shutting down +[parent]: server shutting down +[parent]: waiting for workers to drain and shutdown +[parent]: worker 0 (21752)-> status 0 +[parent]: worker 1 (21753)-> status 0 +$ +``` + +See the [Running](/applications/running.md) section on how to start your Kore application in the background for production purposes. + diff --git a/applications/koreconf.md b/applications/koreconf.md @@ -0,0 +1,88 @@ +# kore.conf + +The configuration file of an application describes to Kore what modules to load, how validators work, what page handlers to map to which functions and more. + +Therefor it is an integral part of Kore as a whole. + +Below we will quickly go over all available quick toggle configuration options, their default settings and what they do. + +There are more options then what is listed below, specifically for validators, authentication blocks and domains. Please find those in https://github.com/jorisvink/kore/blob/master/conf/kore.conf.example. + +--- + +**socket_backlog** +> Maximum length to queue pending connections (see listen(2)). Must be set before any bind directives. + +**bind** +> Bind to a given IP address and port number. + +**chroot** +> The directory the worker processes will chroot() into. + +**runas** +> The username the worker processes drop privileges to. + +**workers** (default: 1) +> The number of worker processes to spawn and keep alive. + +**worker_max_connections** (default: 250) +> The number of active connections each worker will accept. + +**worker_rlimit_nofiles** (default: 1024) +> Limit of maximum open files per worker. + +**worker_accept_threshold** (default: disabled) +> Limit the number of new connections a worker can accept in a single event loop. +By default Kore will accept as many new connections it can up to worker_max_connections. + +**worker_set_affinity** (default: enabled) +> Workers bind themselves to a single CPU by default. Turn this off by setting this option to 0. + +**pidfile** (default: none) +> Store the pid of the parent process in this file. + +**http_header_max** (default: 4096) +> Maximum size of HTTP headers (in bytes). + +**http_body_max** (default: 1024000) +> Maximum size of an HTTP body (in bytes). +> +> If set to 0 disallows requests with a body all together. + +**http_body_disk_offload** (default: disabled) +> Number of bytes after which Kore will use a temporary file to hold the HTTP body instead of holding it in memory. +> +> If set to 0 no disk offloading will be done. This is turned off by default. + +**http_body_disk_path** (default: tmp_files) +> Path where Kore will store any temporary HTTP body files. + +**http_keepalive_time** (default: 20 seconds) +> Maximum seconds an HTTP connection can be kept open by the browser. +> +> Set to 0 to turn off keep-alive completely. + +**http_hsts_enable** (default: 31536000 seconds) +> If not 0 the age of the HSTS header that is included in all responses. + +**http_request_limit** (default: disabled) +> The number of HTTP requests Kore workers will process in one loop. + +**websocket_maxframe** (default: 16384) +> The maximum number of bytes per websocket frame. + +**websocket_timeout** (default: 120 seconds) +> The number of seconds a websocket connection is kept open without traffic. + +**task_threads** (default: 2) +> The number of OS threads to use for background tasks. + +**tls_version** (default: 1.2 only) +> The TLS versions allowed, by default this is set to only TLSv1.2. + +**tls_cipher** (default: A very sane set of ciphersuites prefering AEAD ciphers and ephemiral key exchanges over static RSA). +> The server TLS ciphersuites that are allowed. + +**tls_dhparam** (default: dh2048.pem) +> The DH parameters to load (**required**) + diff --git a/applications/running.md b/applications/running.md @@ -0,0 +1,21 @@ +# Running applications + +Starting Kore applications is done in one or two ways: + +* Using _kodev run_ +* Using _kore -c config_ + +The first method will keep the process in the foreground allowing you to shut it down using CTRL-C. + +This method is aimed when developing. + +The second method will read the configuration file passed on the command line, load in all required application modules and attempt to change root and drop privileges accordingly. + +This method is aimed at running in production. + +You can skip chroot\(\) and privdrops using -n and -r. + +### Halting applications + +When you wish to shutdown your application gracefully you can send a SIGQUIT or SIGTERM signal to the parent process. You can find the PID for this parent process in the pidfile you specified in your configuration. + diff --git a/applications/the-pyko-tool.md b/applications/the-pyko-tool.md @@ -0,0 +1,6 @@ +# The pyko tool + +The pyko tool that comes with Kore + + + diff --git a/arch.png b/arch.png Binary files differ. diff --git a/chapter1.md b/chapter1.md @@ -0,0 +1,68 @@ +# Installation + +### Building and installing + +Kore has been tested to run on the following platforms: + +* Linux 3.2 or newer +* OpenBSD 5.3 or newer +* FreeBSD 10 or newer +* OSX 10.10.x or newer + +Basic compilation requires the following libraries: + +* openssl + +Download the latest release tarball from [https://github.com/jorisvink/kore/releases](https://github.com/jorisvink/kore/releases) and build it: + +``` +$ cd kore +$ make +$ sudo make install +``` + +Kore has several build flavors available: + +* TASKS=1 \(compiles in task support\) +* PGSQL=1 \(compiles in pgsql support\) +* DEBUG=1 \(enables use of -d for debug\) +* NOTLS=1 \(compiles Kore without TLS\) +* NOHTTP=1 \(compiles Kore without HTTP support\) +* NOOPT=1 \(disable compiler optimizations\) +* JSONRPC=1 \(compiles in JSONRPC support\) +* PYTHON=1 \(compiles in Python support\) + +These build flavors can be passed on the command line when building. Note that enabling these flavors may require additional libraries to be present on your system.: + +Requirements for background tasks \(optional\) + +* pthreads + +Requirements for pgsql \(optional\) + +* libpq + +Requirements for python \(optional\) + +* Python 3.6+ + +For BSD-like systems you will need to install GNU make. + +### OSx + +Kore is available on Homebrew under OSx and can be installed with: + +``` +$ brew install kore +``` + +### Verification + +All releases can be verified with [minisign](https://jedisct1.github.io/minisign/) and the following public key: + +``` +RWSxkEDc2y+whfKTmvhqs/YaFmEwblmvar7l6RXMjnu6o9tZW3LC0Hc9 +``` + + + diff --git a/examples.md b/examples.md @@ -0,0 +1,5 @@ +# Examples + +Kore comes with a few example applications: + +You can find these examples over at https://github.com/jorisvink/kore/tree/master/examples. diff --git a/tutorials/tutorials.md b/tutorials/tutorials.md @@ -0,0 +1,5 @@ +# Tutorials + +To be completed ... sorry :-) + +See the [examples](https://jorisvink.gitbooks.io/kore-doc/content/examples.html) page for now.+ \ No newline at end of file