An Introduction to Gerbil

This is a quick introductory guide to Gerbil for seasoned schemers; it assumes familiarity with scheme and exposure to a couple of different implementations.

In the following $ is the shell prompt and > the gxi interpreter prompt.

Using Gerbil

The Gerbil interpreter is $GERBIL_HOME/bin/gxi, and the compiler is $GERBIL_HOME/bin/gxc.

If you want an interactive Gerbil shell just execute the interpreter directly by running gxi.

Hello world

Add $GERBIL_HOME/bin to your path and invoke the interpreter for the obligatory "hello world":

$ export PATH=$PATH:$GERBIL_HOME/bin
$ gxi
> (displayln "hello world")
hello world

The "hello world" script:

$ cat > hello.ss <<EOF
#!/usr/bin/env gxi

(def (main . args)
  (displayln "hello world"))
EOF
$ chmod +x hello.ss
$./hello.ss
hello world

The "hello world" executable:

$ cat > hello.ss <<EOF
package: example

(export main)
(def (main . args)
  (displayln "hello world"))
EOF
$ gxc -exe -o hello hello.ss
$ ./hello
hello world

And if you want a statically linked hello with no runtime dependencies to the Gerbil environment:

$ gxc -static -exe -o hello hello.ss
$ ./hello
hello world

Core Gerbil

Primitive forms

The standard Scheme primitive forms and macros are all supported.

Runtime bindings are defined with the short forms def and defvalues:

(def (say-hello who)
  (displayln "hello " who))

(defvalues (a b)
  (values 1 2))

For those who prefer the classic long forms, define and define-values are also available as in standard Scheme.

Procedures are defined with lambda and can have optional and keyword formal arguments:

(def (a-simple-function a b)
  (+ a b))
> (a-simple-function 1 2)
=> 3

(def (an-opt-lambda a (b 1))
  (+ a b))
> (an-opt-lambda 1)
=> 2

(def (a-keyword-lambda a b: (b 1))
  (+ a b))
> (a-keyword-lambda 1)
=> 2
> (a-keyword-lambda 1 b: 2)
=> 3

Multiple arity lambdas can be declared with case-lambda:

(def my-case-lambda
  (case-lambda
    ((a b) (+ a b))
    ((a) (+ a 1))))

; or the short definition form
(def* my-case-lambda
  ((a b) (+ a b))
  ((a) (+ a 1)))

Let bindings can have a short form for single arguments, as well as multiple value bindings mixed in:

> ((let (x 1) (lambda (y) (+ x 1))) 1)
=> 2
> (let ((values a b) (values 1 2)) (+ a b))
=> 3
> (let ((x 1)
        ((values y z) (values 2 3)))
    (+ x y z))
=> 6

Note that the _ identifier is reserved for bindings to mean the null binding; that is the expression value is ignored and no lexical binding is generated:

(def (a-function x . _) ; accepts 1 or more ignored args
 (+ x 1))
> (a-function 1 2)
=> 2

Apart from cons and list, pairs and lists can also be can be constructed with short-hand syntax using square brackets:

; cons a pair
> [1 . 2]
=> (1 . 2)
; cons a list
> [1 2 3]
=> (1 2 3)

The short-hand syntax also supports list splicing using using the ellipsis ...:

; splice nested list
> [1 2 [3 4 5] ... 6]
=> (1 2 3 4 5 6)

Bindings can be mutated with set! as usual.

> (def a #f)
> (set! a 'a)
> a
=> 'a

set! also expands with s-expressions as the target of mutation. When the head of the s-expression is a setf-macro it is invoked to expand the syntax. If the head is a plain identifier, as is the case in the example below, to expands an identifier-set! invocation.

> (def a-pair (cons 'a 'b))
> (set! (car a-pair) 'c)
> (car a-pair)
=> 'c

Finally, macros that mixin the setq-macro class, like the ones created by identifier-rules, can also be the target of mutation which leads to an expander application.

All the usual Scheme macros are available, with common syntactic forms described later in the guide.

Structs and Classes

Gerbil supports Object-oriented programming with structs and classes. Structs are index-based types with single inheritance, while classes are slot-based types with multiple inheritance.

Structs

Structs are defined with defstruct:

(defstruct point (x y))
(defstruct (point-3d point) (z))
> (make-point-3d 1 2 3)
=> #<point-3d>

For each struct defstruct defines a constructor, a type predicate, a runtime type descriptor, accessors and mutators, expansion time struct info and a match expander.

So:

> (def my-point (make-point-3d 1 2 3))
> (point-x my-point)
=> 1
> (point-y my-point)
=> 2
> (point-3d-z my-point)
=> 3
> (set! (point-3d-z my-point) 0)
> (point-3d-z my-point)
=> 0

Classes

Classes are defined with defclass with slot accessed fields and support multiple inheritance. For example:

(defclass A (a))
(defclass B (b))
(defclass (C A B) (c))
...
> (def my (make-C a: 1 b: 2 c: 3))
> (A? my)
=> #t
> (B? my)
=> #t
> (@ my a)
=> 1
> (@ my b)
=> 2
> (@ my c)
=> 3
> (set! (@ my c) 0)
> (@ my c)
=> 0

Methods

Gerbil supports single dispatch for methods associated with a struct and class type. Methods are defined with defmethod and invoked with curly brace {} s-expressions.

For instance:

(import :std/format)
(defmethod {print point}
  (lambda (self)
    (with ((point x y) self)
      (printf "{point x:~a y:~a}~n" x y))))
> {print my-point}
{point x:1 y:2}

(defmethod {print point-3d}
  (lambda (self)
    (with ((point-3d x y z) self)
      (printf "{point-3d x:~a y:~a z:~a}~n" x y z))))
> {print my-point}
{point-3d x:1 y:2 z:0}

If you want to dispatch to the next method in the hierarchy, then you can use the @next-method macro that is locally bound inside your method definition:

(defmethod {identify point}
  (lambda (self)
    (displayln 'point)))

(defmethod {identify point-3d}
  (lambda (self)
    (displayln 'point-3d)
    (@next-method self)))

> {identify my-point}
point-3d
point

Constructors

By default, the constructors generated for structs expect all the fields in indexed order, while the class constructor expects optional keywords for slots in the class. A custom constructor can be defined by specifying a constructor property designating a method at struct or class definition. For example:

(defstruct (point-3d point) (z)
  constructor: :init!)
(defmethod {:init! point-3d}
  (lambda (self x y (z 0))
    (set! (point-x self) x)
    (set! (point-y self) y)
    (set! (point-z self) z)))
> (def my-point (make-point-3d 1 2))
> (point-3d-z my-point)
=> 0

Mixing classes and structs

Structs can only extend other structs with single inheritance. In contrast, classes can freely mixin structs, as long as the mixins contain a compatible base struct.

For example, the following constructs a diamond hierarchy with a base struct:

(defstruct A (x))
(defclass (B A) ())
(defclass (C A) ())
(defclass (D B C) () constructor: :init!)
(defmethod {:init! D}
  (lambda (self x)
    (set! (A-x self) x)))
(def d (make-D 1))

> (A? d)
=> #t
> (A-x d)
=> 1
> (B? d)
=> #t
> (C? d)
=> #t
> (D? d)
=> #t
> (type-descriptor-mixin D::t)
=> (#<type #5 B> #<type #6 C> #<type #7 A>)

Pattern Matching

Gerbil uses pattern matching extensively, so a suitable match macro is provided by the language. The pattern language is similar to plt's match macro, with structs destuctured by the structure name. In addition, the square brackets destructure lists symmetrically to their construction.

For example:

(def (my-destructurer obj)
 (match obj
   ([a . b]
    (printf "a pair (~a . ~a)~n" a b)
    'pair)
   ((point-3d x y z)
    (printf "a 3d-point (~a ~a ~a)~n" x y z)
    'point-3d)
   ((point x y)
    (printf "a 2d point (~a ~a)~n" x y)
    'point-2d)
   (else 'something-else)))
> (my-destructurer [1 2 3])
a pair (1 . (2 3))
=> 'pair
> (my-destructurer (make-point 1 2))
a 2d point (1 2)
=> 'point-2d
> (my-destructurer (make-point-3d 1 2))
a 3d-point (2 0 0)
=> 'point-3d

Destructuring Binds

Gerbil's match provides a shorthand syntax for match lambdas:

(def car+cdr (match <> ([a . b] (values a b))))
> (car+cdr [1 2 3])
=> values 1 '(2 3)

It is also common to destructure-bind an object, thus a common destructuring-bind form with is provided. The form can bind a single object with short-hand notation or multiple objects with a let-style head:

(def (car+cdr obj)
  (with ([a . b] obj)
    (values a b)))

(def (car+cdrx2 lsta lstb)
  (with (([a-car . a-cdr] lsta)
         ([b-car . b-cdr] lstb))
     (values a-car a-cdr b-car b-cdr)))

Macros

Gerbil has pervasive macro facilities and is a macro-rich language. The full meta-syntactic tower is provided, with macro hygiene support with syntax-case and quote-syntax.

defrules

Most macros are simple and medium syntax-rules macros, and thus Gerbil provides a short form for defining syntax-rules macros:

(defrules macro-id (id ...)
 (head [guard] body) ...)
; equivalent:
(defsyntax macro-id
  (syntax-rules (id ...)
    (head [guard] body) ...))

defsyntax

More complicated macros are defined defsyntax and syntax-case directly. Here is an example that introduces an identifier hygienically:

(defsyntax (with-magic stx)
  (syntax-case stx ()
   ((macro expr body ...)
    (with-syntax ((magic-id (datum->syntax #'macro 'magic)))
      #'(let (magic-id expr) body ...)))))
> (with-magic 3 (+ magic 1))
=> 4

defsyntax-for-match

The match expander is also macro capable; you can define a match macro with defsyntax-for-match, which has the following form:

(defsyntax-for-match id match-macro [macro])

Both macros are procedures at phi+1, with the match-macro invoked when expanding a match pattern and the optional normal macro invoked at normal procedure application.

For example, the following defines a match macro for constructing and destructuring pairs tagged with 'foo:

(defsyntax-for-match foo
  (syntax-rules ()
    ((_ pat) (cons 'foo pat)))
  (syntax-rules ()
    ((_ val) (cons 'foo val))))

> (def my-foo (foo 1))
> my-foo
=> (foo . 1)
> (match my-foo ((foo x) x))
=> 1
> (def my-bar (cons 'bar 2))
> (match my-bar ((foo x) x) (else 'not-a-foo))
=> not-a-foo

begin-syntax

If you need to shift the phase of the expander to evaluate support code for macros, you can do so with begin-syntax:

(begin-syntax form ...)

For example, the following macro uses a utility function in the fender, which is defined at phi=+1:

(begin-syntax
  (def (identifier-or-keyword? stx)
    (or (identifier? stx)
        (stx-keyword? stx)))

  (def (identifiers-or-keywords? lst)
    (andmap identifier-or-keyword? lst)))

(defrules qlist ()
  ((_ (key val) ...)
   (identifiers-or-keywords? #'(key ...))
   [['key . val] ...]))

The full meta-syntactic tower is supported, so you can use the full language at phi=+1 and shift higher with a nested begin-syntax. You will have to import :gerbil/core at higher phases however, as the prelude only provides bindings for phi=+1.

Modules and Libraries

Modules are self-contained pieces of code. All identifiers used in the runtime of the module must be bound. They are either available from the prelude, imported from another module, or declared as extern to indicate runtime-provided identifiers.

Modules can be declared at the top level with the module special form, can be defined in a file, or can be part of a library. They can also be nested in another module.

Top Modules

Here is an example of a simple top module, which provides a function that usesdisplay-exception from the runtime as extern:

(module A
  (export with-display-exception)
  (extern (display-exception display-exception))
  (def (with-display-exception thunk)
    (with-catch (lambda (e) (display-exception e (current-error-port)) e)
      thunk)))
> (import A)
> (with-display-exception (lambda () (error 'it-is-an-error)))
it-is-an-error
=> #<error-exception #5>

Imports and Exports

Identifiers are imported from a module with the import special form, which must appear at a top context (either top-level or module scope). It has the following syntax:

(import <import-spec> ...)
import-spec:
 <module-path>
 (import-expander <import-spec> expander-args ...)
module-path:
 identifier            ; top or module scope module
 :identifier           ; identifier with ':' prefix, library module
 "path-to-module-file" ; file module, .ss extension optional

As we can see, import allows macros to manipulate the import set of some import source (a module or another expansion). They can be defined with defsyntax-for-import An example macro is only-in, provided by the prelude:

(import (only-in :std/text/json read-json))

Here we import form :std/text/json only the read-json procedure.

Modules define the set of exported identifiers with the export special form, which must appear at module scope. It has the following syntax:

(export <export-sec> ...)
export-spec:
 #t                                       ; export all defined identifiers
 identifier                               ; export a specific identifiers
 (rename: id name)                        ; export an identifier with a different name
 (import: <module-path>)                  ; re-export all imports from <module-path>
 (export-expander <export-spec> args ...) ; export macro

Similarly to import, export also supports macros, which can be defined with defsyntax-for-export. An usual export macro is except-out, provided by the prelude:

(export (except-out #t display-exception))

This form exports all defined symbols, except display-exception. It could be used by the example module A above to the same effect.

File Modules

Modules can be written directly in files, without a surrounding module form. For example, we can place our module A into a file A.ss

$ cat > A.ss <<EOF
(export with-display-exception)
(extern (display-exception display-exception))
(def (with-display-exception thunk)
  (with-catch (lambda (e) (display-exception e (current-error-port)) e)
    thunk))
EOF
> (import "A")

File modules take their name from the including file, so this module is named A and uses A# as the namespace prefix. You can be explicit about the namespace the module uses by having a namespace: id declaration at the top of the module.

You can compile file modules with gxc:

$ gxc -d . A.ss
$ gxi
> (import "A")  ; compiled form takes precedence

Library Modules

Library modules are imported with the :library-module-path form. For example, to use the json module from the Gerbil std library you need the following import statement:

(import :std/text/json)

The library module is defined in a file named json.ss in the Gerbil std library source tree. The module declares that it is part of the std/text package, which places compiler artefacts in the $GERBIL_HOME/lib/std/text directory. The namespace prefix for identifiers defined in the module is std/text/json#.

When writing a library module, you should choose an appropriate package for your code. The package is specified with a package: package-path declaration at the top of a module. It effects the namespace of the module and placement of compiled code.

By default library modules are looked up in the $GERBIL_HOME/lib and ~/.gerbil/lib directories. You can specify additional directories to be searched with the GERBIL_LOADPATH environment variable. You can also modify the load-path at runtime with add-load-path.

Building Libraries

This is best illustrated with an example, so let's package the A module above into a library.

For this, we designate example as the library package, and then give the module a more appropriate name. Here, we call it util with the expectation that the library and module will grow further:

$ mkdir example
$ cat > example/util.ss <<EOF
package: example
(export with-display-exception)
(extern (display-exception display-exception))
(def (with-display-exception thunk)
  (with-catch (lambda (e) (display-exception e (current-error-port)) e)
    thunk))
EOF

You can now compile the library module by invoking gxc and import it as :example/util:

$ gxc example/util.ss
$ gxi
> (import :example/util)

By default, the compiler will place compiled modules in ~/.gerbil/lib. If you want a separate directory structure for your library, you can specify a different directory with the -d option:

$ gxc -d your-library-path example/util.ss

Executable Modules

The gerbil compiler can also create executables that invoke the main function of a module. The generated executables only load runtime dependencies without linking and initializing the expander, resulting in significantly faster load times compared to wrapper scripts.

Note that by default, the executable is linked dynamically: the module is still compiled as a dynamic loadable module and must be available in the gerbil library path for the executable to work.

For example, suppose we have a module example/hello.ss that we want to compile as an executable module:

$ cat > example/hello.ss <<EOF
package: example
(export main)
(def (main . args)
  (displayln "hello world"))
EOF

The module must define and export a main function that gets invoked with the command line arguments after loading the gerbil runtime and module dependencies.

You can compile it to an executable with gxc with the following command:

$ gxc -exe -o hello example/hello.ss
$ ./hello
hello world

You can also compile the executable with static linkage, which links parts of the gerbil runtime and all dependent modules statically and allows the executable to work without a local gerbil environment:

$ gxc -static -exe -o hello example/hello.ss
$ ./hello
hello world

The disadvantage of static linkage is that the executables are bigger, and can only use the baseline parts of the gerbil runtime (gx-gambc0). That means that static executables can't use the expander or the compiler, as the meta levels of the gerbil runtime are not linked and initialized. They also take quite longer to compile.

The advantage is that static executables don't require a local Gerbil installation to work, which makes them suitable for binary distribution. They also load faster, as they don't have to do any dynamic module loading. Furthermore, because dependencies are compiled in together, you can apply declarations like (not safe) to the whole program, resulting in potentially significant performance gains. And as of Gerbil-v0.13-DEV-50-gaf81bba the compiler performs full program optimization with tree shaking, which provides further performance benefits.

Prelude Modules and Custom Languages

Every identifier accessible to a Gerbil module has to be defined somewhere, either as a concrete definition or an extern reference. The initial bindings in a module come from the prelude and the root context which is the parent context of every module.

The root context is a special context that contains the core macro expanders that provide the core language. The prelude context on the other hand, is an ordinary module that exports any number of bindings that form the language of the module.

When a prelude is not specified, the default prelude is the Gerbil core prelude. Any module however can designate a different prelude with the prelude: module directive, which allows us to design custom languages.

Apart from standard bindings, custom preludes can also override some special expander indirection hooks by exporting macros with these names:

  • %%ref can intercept and redefine ordinary identifier references.
  • %%app can intercept and redefine ordinary procedure application.
  • %%begin-module can intercept the expansion of a module body and provide custom full or partial expansion.

Language extensibility does not stop there however: prelude modules can also specify a custom surface syntax, by providing a module reader. The custom reader is invoked by using a #lang declaration at the beginning of the module file:

#lang prelude [package: pkg-id] [namespace: namespace-id]

When the expander sources a module that begins with a #lang declaration it imports the prelude and looks for a read-module-body export. The function, which must be defined for syntax, takes as input a the port containing the module body and returns a list of syntax objects which then become the body of the module. The module is then expanded with the usual expansion mechanism, including custom module body expansion as defined by the prelude.

Custom languages are a big topic and this only touches the surface; they are further explored in the Custom Language tutorial.

Implicit Package Declarations

As of Gerbil-v0.12-DEV-845-g39f54e4, you can elide the package: and prelude: declarations in your module and have them automatically deduced from the file system layout of your library/package. You can do so by creating a gerbil.pkg file in the root of your library, which contains a property list.

The package: property specifies the prefix package at the root of your hierarchy. The package of individual modules will extend this prefix to mirror the directory structure.

The prelude: property specifies an implicit custom prelude for s-expression based languages.

If the gerbil.pkg file is empty, then it is treated as an empty property list. This allows you to simply touch a gerbil.pkg at the root of your source hierarchy when you don't need a custom prelude and use a directory structure that mimics your logical package structure.

Standard Library

The gerbil standard library is located at src/std; it includes several common libraries (SRFIs, and usual scheme libraries like :std/pregexp, :std/sort, and :std/format), along with Gerbil-specific libraries. Here we provide examples and brief documentation for the more interesting of the Gerbil-specific libraries.

Optional Libraries

Some library modules are not built by default, because they have external library dependencies that may not be present in your system. The build configuration for the std library is specified in $GERBIL_HOME/src/std/build-features.ss.

If you have the required libraries (documented in build-features) in your system, you can enable building by setting the (enable feature #f) statement in build-features.ss to #t. You can then build the optional library modules by running $GERBIL_HOME/src/build.sh stdlib.

Additional Syntactic Sugar

The :std/sugar library provides, among other macros, a try syntactic form for handling exceptions in imperative style. For example:

> (try (error "my error")
   (catch (e) (display-exception e (current-error-port)))
   (finally (displayln "finally!")))
my error
finally!

The general syntax is

(try body ...
 [catch-clause] ...
 [finally-clause])

catch-clause:
 (catch pred => K)
 (catch (pred var) body ...)
 (catch (var) body ...)
 (catch _ body ...)
finally-clause:
 (finally body ...)

Generics

Gerbil supports generic multi-method dispatch, with the requisite runtime support and macros provided by :std/generic. For example, the following defines a generic method my-add that dispatches on numbers and strings:

(import :std/generic)
(defgeneric my-add
  (lambda args #f))
(defmethod (my-add (a <number>) (b <number>))
  (+ a b))
(defmethod (my-add (a <string>) (b <string>))
  (string-append a b))

The code defined a generic method with the defgeneric macro, providing a default method which is dispatched when there are no matching methods. Next, we defined the two methods, operating on numbers and strings. We can use the generic method as a procedure:

> (my-add 1 2)
=> 3
> (my-add "a" "b")
=> "ab"

We can similarly define methods for user-defined types as well. Here we define an implementation for instances of a struct A:

> (my-add (make-A 1) (make-A 2))
=> #f

(defstruct A (x))
(defmethod (my-add (a A) (b A))
  (make-A (+ (A-x a) (A-x b))))

> (my-add (make-A 1) (make-A 2))
=> #<A a: 3>

Inside the body of every method implementation, @next-method is bound to a procedure which dispatches to the next matching method. For example:

(defmethod (my-add (a <fixnum>) (b <fixnum>))
  (displayln "add fixnums")
  (@next-method a b))

Normally in the procedure body we would add with fx+, but for the shake of the example we display a message and let the generic number method to add.

> (my-add 1 2)
add fixnums
=> 3

Iteration

The :std/iter library provides support for iteration using the iterator protocol. The library also provides macros of the for family for iterating over sequences or other objects that implement the iteration protocol.

Iteration Syntax

The basic iteration macro is the imperative for comprehension. The syntax matches patterns to iterators in parallel, and invokes the body as long as none of the iterators have signalled end of iteration.

For example:

(import :std/iter)

> (for (x '(1 2 3))
   (displayln x))
1
2
3
> (for ((x '(1 2 3))
        (y '#(a b c d)))
  (displayln x " " y))
1 a
2 b
3 c

All patterns supported by the match macro can be matched in lieu of plain variable bindings. For instance:

> (for ([key . val] '((a . 1) (b . 2) (c . 3)))
    (displayln key " " val))
a 1
b 2
c 3

The iteration macro supports the usual suspects for generic iteration: lists, vectors, strings, hash-tables, input-ports, and ranges.

Simple filters can be specified with the when and unless keyword in the binding for:

> (for ([x . y] '((a . 0) (b . 1) (c . 2)) when (> y 0)) (displayln x))
b
c
> (for ([x . y] '((a . 0) (b . 1) (c . 2)) unless (> y 0)) (displayln x))
a

The variant for* performs multi-dimensional iteration, equivalent to nested fors:

> (for* ((x (in-range 2)) (y (in-range 2)))
   (displayln x y))
00
01
10
11

The values of an iteration can be collected in a list with for/collect:

> (for/collect ((x (in-naturals))
                (y '#(a b c d)))
    (cons x y))
=> ((1 . a) (2 . b) (3 . c) (4 . d))

Finally, the values of an iteration can be folded to produce a value; in this example we construct a reversed list out of an iterator with a folding cons:

> (for/fold (r []) (x (in-range 1 5))
    (cons x r))
=> (5 4 3 2 1)

Iteration Protocol

Iteration dispatch applies the generic method :iter in order to produce an iterator object. The default implementation calls the method :iter on the object. There are methods for iterating lists, hashes, input-ports, ranges etc.

The easiest way to implement an iterator is through a coroutine procedure. The procedure is coexecuted with the iteration loop, and produces values for the loop with yield:

(def (my-generator n)
 (lambda ()
   (let lp ((k 0))
     (when (< k n)
       (yield k)
       (lp (1+ k))))))

> (for (x (my-generator 3))
    (displayln x))
0
1
2

We can now use this generator to produce an iterator for a user-defined struct:

(defstruct A (x))
(defmethod {:iter A}
  (lambda (self)
    (:iter (my-generator (A-x self)))))
> (for (x (make-A 3))
    (displayln x))
0
1
2

Coroutines

The :std/coroutine library provides support for coroutines, running in a separate thread and yielding results with yield. The user creates the coroutine with coroutine, and receives results with continue which blocks the current thread and passes control to the coroutine until it yields a value or exits. After the coroutine procedure finishes, all further calls to continue will return the final result or deliver an exception.

For example:

(import :std/coroutine)
(def (my-coroutine)
  (yield 1)
  (yield 2)
  (yield 3))
(def cort (coroutine my-coroutine))
> (continue cort)
=> 1
> (continue cort)
=> 2
> (continue cort)
=> 3
> (continue cort)
=> #!void ; coroutine ended
> (continue cort)
=> #!void ; all

Event Programming

The :std/event library provides procedures and macros for event-driven programming.

wait and select

These are the low level primitives, which wait and multiplex on primitive selectors:

  • Threads, which signal when the thread terminates.
  • Pairs of a locked mutex with a condition variable, which signal when the condition signals after the mutex has been unlocked.
  • Naked i/o condvars, which are signaled by the runtime scheduler.

wait blocks the current thread until a selector is signalled, while select blocks on multiple selectors concurrently, using a thread for each selector. Both procedures accept an optional timeout and return the selector that was signalled or #f in the case of timeout.

(def (wait selector (timeout #f)) ...)
(def (select list-of-selectors (timeout #f)) ...)

For example:

(import :std/event)
(def my-thread (spawn (lambda () (thread-sleep! 10))))
> (wait my-thread 1)      ; or (select [my-thread] 1)
=> #f                    ; after a second elapses
> (wait my-thread)       ; or (select [my-thread])
=> my-thread             ; after the thread completes its sleep

sync

sync is the high-level synchronization primitive from PLT-Scheme, which works with high level events.

A valid argument for sync is any synchronizable object, automatically wrapped with wrap-evt:

  • events
  • primitive selectors
  • input ports
  • timeouts
  • any object that implements the :event method to return a synchronizable object

An event is

  • the primitive events never-evt (bottom) and always-evt (top)
  • an event object, constructed with make-event or wrap-evt
  • an event-set object, constructed with choice-evt
  • an event-handler object, constructed with handle-evt; it is an event tied with a continuation function which is tail invoked with the value of the event. Multiple continuations can be chained with handle-evt each receiving the values of the previous, starting with the value of the vent.

sync accepts an arbitrary of events as arguments, and returns when exactly one of them is ready. The value of sync is the value of the event: by default, timeouts have a value of #f and other events have usually the synchronizer as value.

For example:

(def my-thread (spawn (lambda () (thread-sleep! 10))))
> (sync 1 my-thread)
=> #f ; after a second elapses
> (sync my-thread)
=> my-thread ; after the thread completes its sleep

A more complicated example which utilizes handle-evt for loops:

(def my-thread (spawn (lambda () (thread-sleep! 10) 'done)))
> (let lp ((n 0))
    (sync (handle-evt 1
            (lambda (_) (displayln "timeout " n) (lp (fx1+ n))))
          (handle-evt my-thread
            (lambda (thr) (thread-join! thr)))))
timeout 0
timeout 1
timeout 2
timeout 3
timeout 4
timeout 5
=> 'done

Sync Macros

The library also offers a couple of macros, ! and !*, which simplify event driven programming. ! syncs a single event while !* syncs multiple events.

For example:

;; sync on my-thread
(! my-thread (displayln "my thread exited"))

;; rewrite the previous example loop:
(let lp ((n 0))
  (!* (1 (displayln "timeout " n) (lp (fx1+ n)))
      (my-thread
       (thread-join! my-thread))))

Actors

At the low-level Gerbil builds on Gambit's thread mailboxes to provide actor-oriented programming capabilities and remote interactor communication.

Messages

Gerbil's actors are threads, either in the current or a remote processes and communicate exchanging messages. Messages can be arbitrary objects, but usually actors communicate with structured messages:

(defstruct message (e source dest options))
(def (send dest value) ...)                       ; send raw message
(def (send-message dest value (options #f)) ...)  ; send structured message
(def (send-message/timeout dest value timeo) ...)

Actors process messages and events with two main macros, << and <-. They both synchronize on the thread's mailbox and pattern match incoming messages; the difference is that << matches on raw messages and <- matches on structured message values. Within a <- pattern body, the variables @message, @value, @source, @dest and @options are bound from the structured message. Within the pattern body, the -> can be used as shorthand syntax to send messages to @source.

For example, a simple echo actor:

(import :std/actor)
(def (my-echo)
 (let lp ()
   (<- (value
        (displayln @source " says " value)
        (-> value)
        (lp)))))
(def echo (spawn my-echo))
> (send-message echo 'hello)
#<thread #1 primordial> says hello
> (<- (value value))
=> 'hello

Protocols

Beyond structured messages, Gerbil provides protocols for structured interaction. Protocol messages can be one way messages (instances of !event), a remote call (instances of !call) or a value for a previous call (!value or !error).

Protocols are defined with defproto, which defines structures and macros for using the protocol interfaces, together with marshalling support. For example, let's define an echo protocol:

(defproto echo
  id: echo
  call: (hello what))
(def (my-echo)
  (let lp ()
    (<- ((!echo.hello what k)
          (displayln @source " says " what)
          (!!value what k)
          (lp)))))
(def echo (spawn my-echo))
> (!!echo.hello echo 'hello)
#<thread #1 primordial> says hello
=> 'hello

In the example, we define a protocol echo with a single call hello. The macro defines the structures and macros for using the interface:

(defstruct echo.hello (what))
(defsyntax-for-match !echo.hello ...)
(defrules !!echo.hello ...

The invocation (!!echo.hello echo 'hello) constructs a !call with value an instance of echo.hello and a gensymed continuation id. It then sends a message with the !call to the echo actor and waits for a !value or !error message matching the continuation. If it receive a !value it returns it, and if it receives an !error it signals an error.

In the actor, the (!echo.hello what k) matches a !call with the value matching (echo.hello what) and the continuation token bound to k. The actor displays its message, and then responds by sending a value with the !!value macro. An error could be signalled with the !!error macro.

The syntax for !!value and !!error is similar: the take an optional destination (which defaults to @source), a value or error message and the continuation token:

(!!value [@source] val token)
(!!error [@source] msg token)

Streams

In addition to calls and events, actors can also open streams. A stream is like a call, but it returns multiple values using !!yield until the stream's end or an error occurs.

For example, the following server generates a stream of numbers as specified by the argument:

(defproto simple-stream
  stream: (count N))

(def (my-simple-stream)
  (def (stream dest N k)
    (let lp ((n 0))
      (if (< n N)
        (begin
          (!!yield dest n k)
          (!!sync dest k)               ; request sync
          (<- ((!continue k)            ; flow control
               (lp (1+ n)))
              ((!close k)               ; stream closed
               (!!end dest k))
              ((!abort k)               ; stream aborted
               (void))))
        (!!end dest k))))

  (let lp ()
    (<- ((!simple-stream.count N k)
         (spawn stream @source N k)
         (lp)))))

(def my-stream (spawn my-simple-stream))

Yielded values can be processed as messages, or with the !!pipe macro which constructs a vector pipe and pipes the yielded values through it in a background thread.

Here is an example that uses a pipe:

> (let ((values inp close) (!!pipe (!!simple-stream.count my-stream 5)))
    (for (x inp)
      (displayln x)))
0
1
2
3
4

The pipe may be convenient, but it forgoes end-to-end back pressure and synchronization with !!sync. Here is the same example again, but with explicit processing of messages through the actor mailbox:

(let (k (!!simple-stream.count my-stream 5))
  (let lp ()
    (<- ((!yield x (eq? k))
         (displayln x)
         (lp))
        ((!sync (eq? k))
         (!!continue k)
         (lp))
        ((!end (eq? k))
         (void)))))

RPC

The interaction so far has been local. In order to interact with remote actors, Gerbil provides an rpc protocol and server for handling the necessary network interaction.

Using rpc is very simple: An rpc server can be constructed with start-rpc-server! which accepts an optional server address to bind and a wire protocol implementation with a keyword.

In one shell:

(def (my-echo rpcd)
  (rpc-register rpcd 'echo echo::proto)
  (let lp ()
    (<- ((!echo.hello what k)
          (displayln @source " says " what)
          (!!value what k)
          (lp)))))
(def serverd (start-rpc-server! "127.0.0.1:9999"))
(def echod (spawn my-echo serverd))

This starts an rpc server at port 9999 in the localhost. The echo actor binds itself under the id echo using the echo protocol echo::proto for marshalling and unmarshalling.

In a different shell, we can connect to our echo with a remote handle:

(def clientd (start-rpc-server!))
(def echod (rpc-connect clientd 'echo "127.0.0.1:9999" echo::proto))
> (!!echo.hello echod 'hello)
=> 'hello

If your actors are well-known (application scoped), then you can globally bind a protocol to the name with bind-protocol! and you don't need to specify the protocol in rpc-register and rpc-connect:

(bind-protocol! 'echo echo::proto)
(def clientd ...)
(def echod (rpc-connect clientd 'echo "127.0.0.1:9999"))

By default, a null rpc protocol is used which does no authentication or encryption. If you intend to expose your actors to the Internet you should use authentication and optionally encryption.

For authentication, you can generate a shared cookie with rpc-generate-cookie! and start your rpc-server using the rpc-cookie-proto:

$ mkdir ~/.gerbil
> (rpc-generate-cookie!)
; generates a cookie in ~/.gerbil/cookie
...
(def remoted
 (start-rpc-server! "127.0.0.1:9999"
    proto: (rpc-cookie-proto)))
...
(def locald
  (start-rpc-server! proto: (rpc-cookie-proto)))
...

If you also want to encrypt your communications, then use the rpc-cookie-cipher-proto as proto: argument for your rpc servers. On top of cookie authentication, this protocol performs a Diffie-Hellman key exchange and then encrypts all messages with AES/HMAC (it encrypts and then MACs).

HTTP requests

Gerbil provides a simple interface for making http(s) requests, inspired by python's requests library. Here is an example for how to use the library:

> (import :std/net/request)
> (def req (http-get "freegeoip.net/json/8.8.8.8"))
> (request-status req)
=> 200
> (request-text req)
=> "{\"ip\":\"8.8.8.8\",\"country_code\":\"US\",\"country_name\":\"United States\",\"region_code\":\"CA\",\"region_name\":\"California\",\"city\":\"Mountain View\",\"zip_code\":\"94040\",\"time_zone\":\"America/Los_Angeles\",\"latitude\":37.3845,\"longitude\":-122.0881,\"metro_code\":807}\n"
> (hash->list (request-json req))
=> ((country_name . "United States")
    (metro_code . 807)
    (longitude . -122.0881)
    (country_code . "US")
    (latitude . 37.3845)
    (time_zone . "America/Los_Angeles")
    (region_name . "California")
    (ip . "8.8.8.8")
    (zip_code . "94040")
    (city . "Mountain View")
    (region_code . "CA"))

JSON

Gerbil has library support for JSON with the :std/text/json library. The library provides the following procedures:

(def (read-json (port (current-input-port)) ...)
(def (string->json-object str) ...)
(def (write-json obj (port (current-output-port))) ...)
(def (json-object->string obj) ...)

The mapping of Scheme Objects to JSON objects is similar to other Scheme JSON libraries. The read-json procedure constructs primitive objects (strings, numbers, lists, symbol hashes). The write-json writes JSON objects with the JSON external data representation. The following is a convertible JSON object:

  • booleans, corresponding to true and false
  • #!void, corresponding to null
  • real numbers
  • strings
  • proper lists of JSON objects
  • vectors of JSON objects
  • hashes mapping symbols to JSON objects
  • any object that defines a :json method mapping the object to a JSON object.

XML

Gerbil supports XML and HTML with the :std/xml library. The library supports parsing and querying with Oleg's SXML/SSAX/SXPath and provides additional facilities for processing SXML.

Optionally, when configured so, the library can also use libxml2 to parse real world HTML (and plain old XML). The libxml2 dependent components are not built by default. You can build them by editing std/build-features.ss to set (enable libxml #t) and rerunning the std library build script as described earlier in the guide.

For example, here is a parse of the bing front page without scripts, style, and CDATA:

> (import :std/net/request :std/xml/libxml)
> (def req (http-get "https://www.bing.com"))
> (parse-html (request-text req) filter: '("script" "style" "CDATA"))
=> (*TOP* (html (@ (lang "el")
                (xml:lang "el")
                (xmlns "http://www.w3.org/1999/xhtml"))
             (head (meta (@ (content "text/html; charset=utf-8")
                            (http-equiv "content-type")))
                   (title "Bing")
                   (link (@ (rel "icon")
                            (sizes "any")
                            (mask "")
                            (href "/fd/s/a/hp/bing.svg")))
                   (meta (@ (name "theme-color") (content "#4F4F4F")))
                   (link (@ (href "/s/a/bing_p.ico") (rel "icon")))
                   (meta (@ (content "Το Bing σ"))))))

; so much for modern html!
; everything script, style, and CDATA in the page.

Web Applications

Gerbil offers two options to support web applications:

  • fastcgi with a rack-style interface in :std/web/rack.
  • an embedded http server in :std/net/httpd.

The rack/fastcgi server has been in the standard library since early releases of Gerbil and has a very simple interface familiar from other languages. It works with standard ports so it supports non-development versions of Gambit which don't have raw devices.

The embedded http server is a new development in Gerbil-v0.12-DEV, and utilizes raw devices. It is significantly faster and offers a low level interface oriented towards API programming.

Web programming with rack/fastcgi

This is the obligatory hello web example:

(import :std/web/rack)
(def (respond env data)
  (values 200 '((Content-Type . "text/plain")) "hello world\n"))
(start-rack-fastcgi-server! "127.0.0.1:9000" respond)

The fastcgi web handler is started with start-rack-fastcgi-server! from the std/web/rack library module. The procedure accepts an address where it will listen for fastcgi requests and a handler procedure. The handler accepts two arguments, a hashtable which contains the CGI request environment and the data attached to the request as a u8vector. The handler returns 3 values: the status code for the response, the HTTP headers as an associative list, and the content which can be a string, u8vector or an iterable yielding a stream of string or u8vector data.

Here is a more complex example that prints all request variables to the response:

(def (respond env data)
  (values 200 '((Content . "text/html")) (print-headers env)))

(def (print-headers env)
  (lambda ()
    (yield "<pre>\n")
    (for ((values key val) env)
      (yield (format "~a: ~a\n" key val)))
    (yield "</pre>\n")))

Web programming with the embedded httpd

Here is the hello world example using the embedded httpd:

(import :std/net/httpd)

;; start the httpd
(def httpd
  (start-http-server! "127.0.0.1:8080"))

;; define a handler
(def (hello-handler req res)
  (http-response-write res
                       200                                ; status
                       '(("Content-Type" . "text/plain")) ; headers
                       "hello world\n"))

;; register the handler
> (http-register-handler httpd "/hello" hello-handler)

Here, we start an httpd server, which is a background thread serving HTTP requests. We then register a handler for the /hello path, which will serve all requests for /hello and subpaths.

The handler is a function that accepts two arguments: a request and a response. This handler does not read the response body, and simply responds with hello world with a single http-response-write call.

We can see the handler at work with curl:

$ curl http://localhost:8080/hello
hello world

For more examples of httpd handlers, see the httpd tutorial.

Databases

Gerbil include support for SQL databases (SQLite, PostgreSQL, MySQL) and key-value stores (LevelDB, LMDB) with the :std/db package.

SQL Databases

The :std/db/dbi library provides the implementation of the database interface, while individual modules (:std/db/sqlite, :std/db/postgresql and :std/db/mysql) provide the drivers for particular databases.

Note that not all drivers are built by default, as some are FFI drivers (SQLite, MySQL), so you will need to enable them for your installation in $GERBIL_HOME/src/std/build-features.ss.

Here is an example of using the dbi interface with SQLite. First, the necessary imports and a connection to an in-memory database:

> (import :std/db/dbi :std/db/sqlite)
> (def db (sql-connect sqlite-open ":memory:"))

Then we create a simple table with sql-eval, which evaluates an SQL statement:

> (sql-eval db "CREATE TABLE Users (FirstName VARCHAR, LastName VARCHAR, Secret VARCHAR)")

Let's insert some data in our table, using a prepared statements:

> (def insert (sql-prepare db "INSERT INTO Users (FirstName, LastName, Secret) VALUES (?, ?, ?)"))
> (sql-txn-begin db)
> (sql-bind insert "John" "Smith" "very secret")
> (sql-exec insert)
> (sql-bind insert "Marc" "Thompson" "oh so secret")
> (sql-exec insert)
> (sql-txn-commit db)

And finally a query:

> (def users (sql-prepare db "SELECT FirstName, LastName FROM Users"))
> (sql-query users)
=> (#("John" "Smith") #("Marc" "Thompson"))

We can also iterate on the results of a query:

> (import :std/iter)
> (for (#(FirstName LastName) (in-sql-query users))
    (displayln "First name: " FirstName " Last name: " LastName))
First name: John Last name: Smith
First name: Marc Last name: Thompson

And we are done, we can close our database connection:

> (sql-close db)

Key-Value Stores

The :std/db/leveldb library provides support for LevelDB, while the :std/db/lmdb library provides support for LMDB. The libraries are not built by default, as they have foreign dependencies, so you need to enable them for your installation in $GERBIL_HOME/src/std/build-features.ss.

For example, here we use the LevelDB library for some simple operations:

> (import :std/db/leveldb
          :std/sugar)
> (def db (leveldb-open "/tmp/leveldb-test.db"))

;; let's put some values
> (leveldb-put db "abc" "this is the value of abc")
> (leveldb-put db "def" "this is the value of def")

;; we can retrieve them -- objects are always stored as u8vectors
> (displayln (bytes->string (leveldb-get db "abc")))
this is the value of abc

;; let's iterate and print the contents of the store
> (let (itor (leveldb-iterator db))
    (leveldb-iterator-seek-first itor)
    (while (leveldb-iterator-valid? itor)
      (displayln (bytes->string (leveldb-iterator-key itor))
                 " => "
                 (bytes->string (leveldb-iterator-value itor)))
      (leveldb-iterator-next itor))
    (leveldb-iterator-close itor))
abc => this is the value of abc
def => this is the value of def

;; we can do the same with a for loop
> (for ((values key val) (in-leveldb db))
    (displayln (bytes->string key)
               " => "
               (bytes->string val)))
abc => this is the value of abc
def => this is the value of def

;; Let's delete a value
> (leveldb-delete db "abc")
> (leveldb-get db "abc")
=> #f

;; we are done, let's close the db
(leveldb-close db)

The LMDB library is covered in in the Key-Value Store Server tutorial.