Introduction

Many compiled languages (such as C, C++, Rust, Swift, Haskell, etc.) can compile C-compatible dynamic libraries. Functions in these libraries can be directly called in compiled Wolfram Language code, making it possible to write high-performance links between top-level Wolfram Language and dynamic libraries.

This tutorial contains a short example of a connection to a single OpenSSL function, as well as an extended example of an interface to a large piece of SQLite functionality.

Connecting to OpenSSL

OpenSSL exposes C-compatible libraries that implement cryptography functionality. This example demonstrates a connection to OpenSSL's cryptographically secure pseudorandom number generator.

OpenSSL exposes a function called RAND_bytes with the following type signature:

int RAND_bytes(unsigned char *buf, int num);

The first argument buf is the buffer into which it will write the random bytes. The second argument num is the number of bytes to write to buf.

(The library path did not need to be specified because OpenSSL is already loaded by the Wolfram Language kernel by default.)

The previous function uses several concepts to work.

First, CreateTypeInstance is used to create a managed "CArray" with the given length. This is the buffer into which the random bytes will be written.

Second, LibraryFunction["RAND_bytes"] is used to call the OpenSSL function declared by the LibraryFunctionDeclaration.

Finally, CreateTypeInstance is used again to extract the elements of the buffer and copy them to a "NumericArray". FromRawPointer can also be used for this task.

Connecting to SQLite

SQLite is a C-compatible library that implements an SQL database. In order to run this tutorial, SQLite must be downloaded as a paclet.

The paclet can be uninstalled by running PacletUninstall.

SQLite's documented C interface contains a function called sqlite3_libversion_number with the following signature:

int sqlite3_libversion_number(void);

This demonstrates calling a single function, but this same mechanism can be used to create elaborate links that can pass complex custom data structures to and from dynamic libraries.

This tutorial goes through all the steps required to create an efficient link between top-level Wolfram Language and SQLite. First, a system is created for opening and automatically closing SQLite databases. Then, queries are executed on those databases. Finally, the results are read back from the queries and converted into Wolfram Language expressions, suitable for use in further data analysis.

Opening and Closing Databases

The first step in connecting to an SQLite database is opening and closing it. This is done through two functions, which have the following signatures:

int sqlite3_open(
  const char *filename,   /* Database filename (UTF-8) */
  sqlite3 **ppDb          /* OUT: SQLite db handle */
);

int sqlite3_close(sqlite3*);

This "Managed" object has a shared reference count in both compiled and top-level code, and when its reference count goes to 0, it will execute the freeing code. This provides a memory-safe way of passing around SQLite database objects in the top level.

Simple Queries

Now that databases can be opened and (automatically) closed, queries can be run on them. SQLite's simplest querying interface is through sqlite3_exec:

int sqlite3_exec(
  sqlite3*,                                  /* An open database */
  const char *sql,                           /* SQL to be evaluated */
  int (*callback)(void*,int,char**,char**),  /* Callback function */
  void *,                                    /* 1st argument to callback */
  char **errmsg                              /* Error msg written here */
);

This returns error code 0, indicating that the table was created successfully.

This execquery function can run arbitrary queries on the database, but it has no mechanism for reading the results of the query or returning them in a useful form. That is the subject of the next section.

Adding Reading Functionality

Traversing Query Results

In order to read from the query results, a slightly different interface is used from before. Instead of using sqlite3_exec, sqlite3_prepare is used to generate a query statement and sqlite3_step is used to step through the results. sqlite3_finalize is also needed in order to close the query statement. Their type signatures are:

int sqlite3_prepare(
  sqlite3 *db,            /* Database handle */
  const char *zSql,       /* SQL statement, UTF-8 encoded */
  int nByte,              /* Maximum length of zSql in bytes. */
  sqlite3_stmt **ppStmt,  /* OUT: Statement handle */
  const char **pzTail     /* OUT: Pointer to unused portion of zSql */
);

int sqlite3_step(sqlite3_stmt*);

int sqlite3_finalize(sqlite3_stmt *pStmt);

Turning Rows into Expressions

The functions above make it possible to execute queries and traverse the results. However, more library functions are needed to read the data off each row and convert it into Wolfram Language expressions. In particular, these SQLite functions are used for fetching the number of columns, their names and their types:

int sqlite3_column_count(sqlite3_stmt *pStmt);

const char *sqlite3_column_name(sqlite3_stmt*, int N);

int sqlite3_column_type(sqlite3_stmt*, int iCol);

These SQLite functions are used for extracting data of various types from each row:

double sqlite3_column_double(sqlite3_stmt*, int iCol);

sqlite3_int64 sqlite3_column_int64(sqlite3_stmt*, int iCol);

const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol);

Compiling the Querying Function

Databases can now be opened (and automatically closed) with openDB, queries can be compiled with createStatement, and each row of the query results can be turned into expressions with rowAssociation. All that remains is to put it together.

Complete Code