Calling External Libraries with the Wolfram Compiler


Many compiled languages (such as C, C++, Rust, Swift, Haskell, etc.) can compile C-compatible dynamic libraries. Functions in these libraries can be directly called in compiled Wolfram Language code, making it possible to write high-performance links between top-level Wolfram Language and dynamic libraries.

This tutorial contains a short example of a connection to a single OpenSSL function, as well as an extended example of an interface to a large piece of SQLite functionality.

Connecting to OpenSSL

OpenSSL exposes C-compatible libraries that implement cryptography functionality. This example demonstrates a connection to OpenSSL's cryptographically secure pseudorandom number generator.

OpenSSL exposes a function called RAND_bytes with the following type signature:

int RAND_bytes(unsigned char *buf, int num);

The first argument buf is the buffer into which it will write the random bytes. The second argument num is the number of bytes to write to buf.

This function declaration can be represented with the following LibraryFunctionDeclaration:

(The library path did not need to be specified because OpenSSL is already loaded by the Wolfram Language kernel by default.)

Compile a function using this declaration:

The previous function uses several concepts to work.

First, CreateTypeInstance is used to create a managed "CArray" with the given length. This is the buffer into which the random bytes will be written.

Second, LibraryFunction["RAND_bytes"] is used to call the OpenSSL function declared by the LibraryFunctionDeclaration.

Finally, CreateTypeInstance is used again to extract the elements of the buffer and copy them to a "NumericArray". FromRawPointer can also be used for this task.

With everything compiled, run the function to get 10 cryptographically secure pseudorandom bytes:

Connecting to SQLite

SQLite is a C-compatible library that implements an SQL database. In order to run this tutorial, SQLite must be downloaded as a paclet.

Evaluate this command to download SQLite:

The paclet can be uninstalled by running PacletUninstall.

SQLite's documented C interface contains a function called sqlite3_libversion_number with the following signature:

int sqlite3_libversion_number(void);
This C signature can be represented with the following LibraryFunctionDeclaration:
Use the Wolfram Compiler to compile a program using the library function:

This demonstrates calling a single function, but this same mechanism can be used to create elaborate links that can pass complex custom data structures to and from dynamic libraries.

This tutorial goes through all the steps required to create an efficient link between top-level Wolfram Language and SQLite. First, a system is created for opening and automatically closing SQLite databases. Then, queries are executed on those databases. Finally, the results are read back from the queries and converted into Wolfram Language expressions, suitable for use in further data analysis.

Opening and Closing Databases

The first step in connecting to an SQLite database is opening and closing it. This is done through two functions, which have the following signatures:

int sqlite3_open(
const char *filename, /* Database filename (UTF-8) */
sqlite3 **ppDb /* OUT: SQLite db handle */

int sqlite3_close(sqlite3*);
These can be translated into the following instances of LibraryFunctionDeclaration, along with an alias for the sqlite3* type:
With these, compile a simple function that opens and closes a database given a path:
Executing this function on the path to an SQLite database returns error code 0 (SQLITE_OK):
In the previous program, the database was manually closed once it was no longer needed. This can be simplified by using the "Managed" type to add automatic memory management so that the database is closed automatically when it goes out of scope:
Open an existing SQLite database and return it as a memory-managed object:

This "Managed" object has a shared reference count in both compiled and top-level code, and when its reference count goes to 0, it will execute the freeing code. This provides a memory-safe way of passing around SQLite database objects in the top level.

Simple Queries

Now that databases can be opened and (automatically) closed, queries can be run on them. SQLite's simplest querying interface is through sqlite3_exec:

int sqlite3_exec(
sqlite3*, /* An open database */
const char *sql, /* SQL to be evaluated */
int (*callback)(void*,int,char**,char**), /* Callback function */
void *, /* 1st argument to callback */
char **errmsg /* Error msg written here */
Ignoring the callbacks and error message interfaces, sqlite3_exec can be represented with the following LibraryFunctionDeclaration:
Compile a function that takes a database object (as it would be returned from the previously defined openDB function) and runs a query on it:
To try it, create a temporary file for the database:
Open the database:
Execute a query to create a table in the new database:

This returns error code 0, indicating that the table was created successfully.

Running the same query again returns error code 1, indicating that the query failed because the table already exists:

This execquery function can run arbitrary queries on the database, but it has no mechanism for reading the results of the query or returning them in a useful form. That is the subject of the next section.

Adding Reading Functionality

Traversing Query Results

In order to read from the query results, a slightly different interface is used from before. Instead of using sqlite3_exec, sqlite3_prepare is used to generate a query statement and sqlite3_step is used to step through the results. sqlite3_finalize is also needed in order to close the query statement. Their type signatures are:

int sqlite3_prepare(
sqlite3 *db, /* Database handle */
const char *zSql, /* SQL statement, UTF-8 encoded */
int nByte, /* Maximum length of zSql in bytes. */
sqlite3_stmt **ppStmt, /* OUT: Statement handle */
const char **pzTail /* OUT: Pointer to unused portion of zSql */

int sqlite3_step(sqlite3_stmt*);

int sqlite3_finalize(sqlite3_stmt *pStmt);
Convert these signatures into LibraryFunctionDeclarations, along with a TypeDeclaration for sqlite3_stmt pointers:
Declare a function that compiles a query and returns a memory-managed statement object:
Compile a function that takes a query statement object, steps through its results, and returns the number of results:
Open an example database and count the number of rows in the customers table:

Turning Rows into Expressions

The functions above make it possible to execute queries and traverse the results. However, more library functions are needed to read the data off each row and convert it into Wolfram Language expressions. In particular, these SQLite functions are used for fetching the number of columns, their names and their types:

int sqlite3_column_count(sqlite3_stmt *pStmt);

const char *sqlite3_column_name(sqlite3_stmt*, int N);

int sqlite3_column_type(sqlite3_stmt*, int iCol);
These can be represented with the following LibraryFunctionDeclarations:

These SQLite functions are used for extracting data of various types from each row:

double sqlite3_column_double(sqlite3_stmt*, int iCol);

sqlite3_int64 sqlite3_column_int64(sqlite3_stmt*, int iCol);

const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol);
They can be represented with the following additional LibraryFunctionDeclarations:
Next, declare a function that takes a statement object and a column number and returns the value of the column as an expression. sqlite3_column_type is used to get the type of the column and then calls the appropriate function to get the value:
Finally, declare a function that extracts all of the columns from a given row of the query results:

Compiling the Querying Function

Databases can now be opened (and automatically closed) with openDB, queries can be compiled with createStatement, and each row of the query results can be turned into expressions with rowAssociation. All that remains is to put it together.

Compile everything into a function that takes an SQLite database and a query string and returns the result as a list of associations:
Open an SQLite database:
Query it:
Perform a larger query and put the results in a Dataset:

Complete Code

The complete code in a single call to FunctionCompile: