使用 Wolfram 编译器调用外部库
引言
许多编译语言(如 C、C++、Rust、Swift、Haskell 等)可以编译与 C 兼容的动态库. 这些库中的函数可以在编译后的 Wolfram 语言代码中直接调用,从而可以编写顶级 Wolfram 语言和动态库之间的高性能链接.
本教程包含一个简短示例,是关于与单个 OpenSSL 函数的连接,以及一个扩展示例,介绍与一大块 SQLite 功能的接口.
连接到 OpenSSL
OpenSSL 把实现加密功能的 C 兼容库公开. 此示例将演示与 OpenSSL 的加密安全伪随机数生成器的连接.
OpenSSL 公开一个名为 RAND_bytes 的函数,其类型签名如下:
int RAND_bytes(unsigned char *buf, int num);
第一个参数 buf 是将随机字节写入其中的缓冲区. 第二个参数 num 是要写入 buf 的字节数.
dec = LibraryFunctionDeclaration["RAND_bytes", {"CArray"::["CChar"], "CInt"} -> "CInt"];(不需要指定库路径,因为 OpenSSL 已经由 Wolfram 语言内核默认加载.)
cf = FunctionCompile[dec,
Function[Typed[len, "CInt"], Module[{buf},
buf = CreateTypeInstance["Managed"::["CArray"::["CChar"]], len];
LibraryFunction["RAND_bytes"][buf, len];
CreateTypeInstance["NumericArray", buf, len]
]]
]首先,CreateTypeInstance 用于创建具有给定长度的托管 "CArray". 这是将写入随机字节的缓冲区.
其次,LibraryFunction["RAND_bytes"] 用于调用 LibraryFunctionDeclaration 声明的 OpenSSL 函数.
最后,CreateTypeInstance 再次用于提取缓冲区的元素,并将它们复制到 "NumericArray". FromRawPointer 也可用于此任务.
Normal@cf[10]连接到 SQLite
SQLite 是实现 SQL 数据库的 C 兼容库. 为了运行本教程,必须将 SQLite 作为数据包下载.
PacletInstall["LibSQLite"]可以通过运行 PacletUninstall 来卸载数据包.
SQLite 记录的 C 接口包含一个名为 sqlite3_libversion_number 的函数,其签名如下:
int sqlite3_libversion_number(void);
versionDec = LibraryFunctionDeclaration["sqlite3_libversion_number", "libsqlite3", {} -> "CInt"];cf = FunctionCompile[versionDec, Function[{},
LibraryFunction["sqlite3_libversion_number"][]
]]cf[]这里演示的是调用单个函数,但同样的机制可用于创建复杂的链接,这些链接可以将复杂的自定义数据结构传入和传出动态库.
本教程详细介绍在高级 Wolfram 语言和 SQLite 之间创建有效链接所需的所有步骤. 首先,创建一个系统,用于打开和自动关闭 SQLite 数据库. 然后,在这些数据库上执行查询. 最后,从查询中读回结果,并将其转换为适用于进一步数据分析的 Wolfram 语言表达式.
打开和关闭数据库
连接到 SQLite 数据库,第一步是打开和关闭它. 这是通过两个函数完成的,它们具有以下签名:
int sqlite3_open(
const char *filename, /* Database filename (UTF-8) */
sqlite3 **ppDb /* OUT: SQLite db handle */
);
int sqlite3_close(sqlite3*);
openCloseDecs = {
TypeDeclaration["Alias", "SQLiteDB", "OpaqueRawPointer"],
LibraryFunctionDeclaration["sqlite3_open", "libsqlite3", {"CString", "RawPointer"::["SQLiteDB"]} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_close", "libsqlite3", {"SQLiteDB"} -> "CInt"]
};openClose = FunctionCompile[openCloseDecs,
Function[Typed[path, "String"],
Module[{cstrPath, dbPointer, db}, cstrPath = Cast[path, "Managed"::["CString"]];
dbPointer = Typed[ToRawPointer[], "RawPointer"::["SQLiteDB"]];
LibraryFunction["sqlite3_open"][cstrPath, dbPointer];
db = FromRawPointer[dbPointer];
LibraryFunction["sqlite3_close"][db]
]
]
]openClose[FindFile["ExampleData/ecommerce-database.sqlite"]]openDB = FunctionCompile[openCloseDecs,
Function[Typed[path, "String"],
Module[{cstrPath, dbPointer},
cstrPath = Cast[path, "Managed"::["CString"]];
dbPointer = Typed[ToRawPointer[], "RawPointer"::["SQLiteDB"]];
LibraryFunction["sqlite3_open"][cstrPath, dbPointer];
CreateTypeInstance["Managed", FromRawPointer[dbPointer], LibraryFunction["sqlite3_close"][#];&]
]
]
]openDB[FindFile["ExampleData/ecommerce-database.sqlite"]]该 "Managed" 对象在编译代码和顶层代码中都有一个共享的引用计数,当引用计数变为 0 时,将执行释放代码. 这提供了一种在顶层传递 SQLite 数据库对象的内存安全方式.
简单查询
现在可以打开和(自动)关闭数据库,可以在上面运行查询. SQLite 最简单的查询接口是通过 sqlite3_exec:
int sqlite3_exec(
sqlite3*, /* An open database */
const char *sql, /* SQL to be evaluated */
int (*callback)(void*,int,char**,char**), /* Callback function */
void *, /* 1st argument to callback */
char **errmsg /* Error msg written here */
);
execDecs = Join[
openCloseDecs, {
LibraryFunctionDeclaration["sqlite3_exec", "libsqlite3", {
"SQLiteDB",
"CString",
"OpaqueRawPointer",
"OpaqueRawPointer",
"RawPointer"::["CString"]
} -> "CInt"]
}];execQuery = FunctionCompile[execDecs,
Function[{Typed[db, "Managed"::["SQLiteDB"]], Typed[query, "String"]},
Module[{cstrQuery},
cstrQuery = Cast[query, "Managed"::["CString"]];
LibraryFunction["sqlite3_exec"][
db,
cstrQuery,
Cast[0, "OpaqueRawPointer", "BitCast"],
Cast[0, "OpaqueRawPointer", "BitCast"],
Cast[0, "RawPointer"::["CString"], "BitCast"]
]
]
]
]dbFile = CreateFile[];db = openDB[dbFile]createTableQuery = "CREATE TABLE people (
id INTEGER PRIMARY KEY,
name TEXT
age INTEGER
);";execQuery[db, createTableQuery]execQuery[db, createTableQuery]这个 execquery 函数可以在数据库上运行任意查询,但它没有读取查询结果或以有用的形式返回它们的机制. 这将是下一节的主题.
添加读取功能
遍历查询结果
为了从查询结果中读取,使用了与之前略有不同的界面。 不是使用 sqlite3_exec,而是使用 sqlite3_prepare 来生成查询语句,并使用 sqlite3_step 逐步执行结果. 为了关闭查询语句,还需要使用 sqlite3_finalize. 它们的类型签名是:
int sqlite3_prepare(
sqlite3 *db, /* Database handle */
const char *zSql, /* SQL statement, UTF-8 encoded */
int nByte, /* Maximum length of zSql in bytes. */
sqlite3_stmt **ppStmt, /* OUT: Statement handle */
const char **pzTail /* OUT: Pointer to unused portion of zSql */
);
int sqlite3_step(sqlite3_stmt*);
int sqlite3_finalize(sqlite3_stmt *pStmt);
readingDecs = Join[
execDecs, {
TypeDeclaration["Alias", "SQLiteStatement", "OpaqueRawPointer"],
LibraryFunctionDeclaration["sqlite3_prepare", "libsqlite3", {
"SQLiteDB",
"CString",
"CInt",
"RawPointer"::["SQLiteStatement"],
"RawPointer"::["CString"]
} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_step", "libsqlite3", {"SQLiteStatement"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_finalize", "libsqlite3", {"SQLiteStatement"} -> "CInt"]
}];readingDecs2 = Join[
readingDecs, {
FunctionDeclaration[createStatement,
Typed[{"Managed"::["SQLiteDB"], "String"} -> "Managed"::["SQLiteStatement"]]@
Function[{db, query},
Module[{cstrQuery, statementPointer},
cstrQuery = Cast[query, "Managed"::["CString"]];
statementPointer = Typed[ToRawPointer[], "RawPointer"::["SQLiteStatement"]];
LibraryFunction["sqlite3_prepare"][
db,
cstrQuery,
Cast[StringLength[query], "CInt", "ReinterpretCast"],
statementPointer,
Cast[0, "RawPointer"::["CString"], "BitCast"]
];
CreateTypeInstance["Managed", FromRawPointer[statementPointer], LibraryFunction["sqlite3_finalize"][#];&]
]
]
]
}];countResults = FunctionCompile[readingDecs2,
Function[{Typed[db, "Managed"::["SQLiteDB"]], Typed[query, "String"]},
Module[{statement, n},
statement = createStatement[db, query];
n = 0;
While[LibraryFunction["sqlite3_step"][statement] == 100, n++];
n
]
]
]With[{db = openDB[FindFile["ExampleData/ecommerce-database.sqlite"]]},
countResults[db, "SELECT * FROM customers;"]]将行变成表达式
上面的函数使得执行查询和遍历结果成为可能. 但是,读取每一行数据并将其转换为 Wolfram 语言表达式需要更多的库函数. 特别地,以下这些 SQLite 函数用于获取列数、名称和类型:
int sqlite3_column_count(sqlite3_stmt *pStmt);
const char *sqlite3_column_name(sqlite3_stmt*, int N);
int sqlite3_column_type(sqlite3_stmt*, int iCol);
readingDecs3 = Join[
readingDecs2, {
LibraryFunctionDeclaration["sqlite3_column_count", "libsqlite3", {"SQLiteStatement"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_column_name", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CString"],
LibraryFunctionDeclaration["sqlite3_column_type", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CInt"]
}];double sqlite3_column_double(sqlite3_stmt*, int iCol);
sqlite3_int64 sqlite3_column_int64(sqlite3_stmt*, int iCol);
const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol);
readingDecs4 = Join[
readingDecs3, {
LibraryFunctionDeclaration["sqlite3_column_double", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CDouble"],
LibraryFunctionDeclaration["sqlite3_column_int64", "libsqlite3", {"SQLiteStatement", "CInt"} -> "Integer64"],
LibraryFunctionDeclaration["sqlite3_column_text", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CString"]
}];readingDecs5 = Join[
readingDecs4, {
FunctionDeclaration[columnExpression,
Typed[{"Managed"::["SQLiteStatement"], "MachineInteger"} -> "Expression"]@
Function[{statement, colIndex0},
With[{colIndex = Cast[colIndex0, "CInt", "ReinterpretCast"]},
Switch[LibraryFunction["sqlite3_column_type"][statement, colIndex],
Typed[1, "CInt"],
Cast[LibraryFunction["sqlite3_column_int64"][statement, colIndex], "Expression"],
Typed[2, "CInt"],
Cast[LibraryFunction["sqlite3_column_double"][statement, colIndex], "Expression"],
Typed[3, "CInt"],
Cast[CreateTypeInstance["String", LibraryFunction["sqlite3_column_text"][statement, colIndex]], "Expression"],
Typed[5, "CInt"],
InertExpression[Missing["NullValue"]],
_,
InertExpression[Missing["UnsupportedType"]]
]
]
]
]
}];readingDecs6 = Join[
readingDecs5, {
FunctionDeclaration[rowAssociation,
Typed[{"Managed"::["SQLiteStatement"]} -> "Expression"]@
Function[statement,
Module[{columnCount, columnRules},
columnCount = Cast[LibraryFunction["sqlite3_column_count"][statement], "MachineInteger", "ReinterpretCast"];
columnRules = (col |-> Module[{colName, colRule},
colName = LibraryFunction["sqlite3_column_name"][statement, Cast[col, "CInt", "ReinterpretCast"]];
colRule = CreateTypeInstance["String", colName] -> columnExpression[statement, col];
Cast[colRule, "Expression"]
]) /@ Cast[Range[0, columnCount - 1], "ListVector"::[_]];
InertExpression[Association]@@columnRules
]
]
]
}];编译查询函数
现在可以使用 openDB 打开(并自动关闭)数据库,可以使用 createStatement 编译查询,并且可以使用 rowAssociation 将查询结果的每一行转换为表达式. 剩下的就是把它放在一起.
queryDB = FunctionCompile[readingDecs6,
Function[{Typed[db, "Managed"::["SQLiteDB"]], Typed[query, "String"]},
Module[{statement, out},
statement = createStatement[db, query];
out = CreateDataStructure["DynamicArray"];
While[LibraryFunction["sqlite3_step"][statement] === Typed[100, "CInt"],
AppendTo[out, rowAssociation[statement]]
];
Normal[out]
]
]
]db = openDB[FindFile["ExampleData/ecommerce-database.sqlite"]]queryDB[db, "SELECT * FROM customers WHERE creditLimit > 200000;"]Dataset[queryDB[db, "SELECT * FROM customers;"]]完整代码
{openDB, queryDB} = FunctionCompile[{
TypeDeclaration["Alias", "SQLiteDB", "OpaqueRawPointer"],
TypeDeclaration["Alias", "SQLiteStatement", "OpaqueRawPointer"],
LibraryFunctionDeclaration["sqlite3_open", "libsqlite3", {"CString", "RawPointer"::["SQLiteDB"]} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_close", "libsqlite3", {"SQLiteDB"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_prepare", "libsqlite3", {
"SQLiteDB",
"CString",
"CInt",
"RawPointer"::["SQLiteStatement"],
"RawPointer"::["CString"]
} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_step", "libsqlite3", {"SQLiteStatement"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_finalize", "libsqlite3", {"SQLiteStatement"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_column_count", "libsqlite3", {"SQLiteStatement"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_column_name", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CString"],
LibraryFunctionDeclaration["sqlite3_column_type", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CInt"],
LibraryFunctionDeclaration["sqlite3_column_double", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CDouble"],
LibraryFunctionDeclaration["sqlite3_column_int64", "libsqlite3", {"SQLiteStatement", "CInt"} -> "Integer64"],
LibraryFunctionDeclaration["sqlite3_column_text", "libsqlite3", {"SQLiteStatement", "CInt"} -> "CString"],
FunctionDeclaration[createStatement,
Typed[{"Managed"::["SQLiteDB"], "String"} -> "Managed"::["SQLiteStatement"]]@
Function[{db, query},
Module[{cstrQuery, statementPointer},
cstrQuery = Cast[query, "Managed"::["CString"]];
statementPointer = Typed[ToRawPointer[], "RawPointer"::["SQLiteStatement"]];
LibraryFunction["sqlite3_prepare"][
db,
cstrQuery,
Cast[StringLength[query], "CInt", "ReinterpretCast"],
statementPointer,
Cast[0, "RawPointer"::["CString"], "BitCast"]
];
CreateTypeInstance["Managed", FromRawPointer[statementPointer], LibraryFunction["sqlite3_finalize"][#];&]
]
]
],
FunctionDeclaration[columnExpression,
Typed[{"Managed"::["SQLiteStatement"], "MachineInteger"} -> "Expression"]@
Function[{statement, colIndex0},
With[{colIndex = Cast[colIndex0, "CInt", "ReinterpretCast"]},
Switch[LibraryFunction["sqlite3_column_type"][statement, colIndex],
Typed[1, "CInt"],
Cast[LibraryFunction["sqlite3_column_int64"][statement, colIndex], "Expression"],
Typed[2, "CInt"],
Cast[LibraryFunction["sqlite3_column_double"][statement, colIndex], "Expression"],
Typed[3, "CInt"],
Cast[CreateTypeInstance["String", LibraryFunction["sqlite3_column_text"][statement, colIndex]], "Expression"],
Typed[5, "CInt"],
InertExpression[Missing["NullValue"]],
_,
InertExpression[Missing["UnsupportedType"]]
]
]
]
],
FunctionDeclaration[rowAssociation,
Typed[{"Managed"::["SQLiteStatement"]} -> "Expression"]@
Function[statement,
Module[{columnCount, columnRules},
columnCount = Cast[LibraryFunction["sqlite3_column_count"][statement], "MachineInteger", "ReinterpretCast"];
columnRules = (col |-> Module[{colName, colRule},
colName = LibraryFunction["sqlite3_column_name"][statement, Cast[col, "CInt", "ReinterpretCast"]];
colRule = CreateTypeInstance["String", colName] -> columnExpression[statement, col];
Cast[colRule, "Expression"]
]) /@ Cast[Range[0, columnCount - 1], "ListVector"::[_]];
InertExpression[Association]@@columnRules
]
]
]
}, {
Function[Typed[path, "String"],
Module[{cstrPath, dbPointer},
cstrPath = Cast[path, "Managed"::["CString"]];
dbPointer = Typed[ToRawPointer[], "RawPointer"::["SQLiteDB"]];
LibraryFunction["sqlite3_open"][cstrPath, dbPointer];
CreateTypeInstance["Managed", FromRawPointer[dbPointer], LibraryFunction["sqlite3_close"][#];&]
]
],
Function[{Typed[db, "Managed"::["SQLiteDB"]], Typed[query, "String"]},
Module[{statement, out},
statement = createStatement[db, query];
out = CreateDataStructure["DynamicArray"];
While[LibraryFunction["sqlite3_step"][statement] === Typed[100, "CInt"],
AppendTo[out, rowAssociation[statement]]
];
Normal[out]
]
]
}]