One of the unique aspects of the Wolfram Compiler is its tight coupling with the Wolfram Language. This seamless integration provides many interesting and innovative features. At the core of this is the ability of the Wolfram Compiler to work with a central property of the Wolfram Language: Wolfram expressions.

Wolfram expressions are the single uniform data type and are processed by the Wolfram Language interpreter. Expressions are a flexible and powerful way to work with programs and data; they form the core basis of the Wolfram Language.

On the other hand, the Wolfram Compiler works with many different data types. New data types can be defined and worked on. These types can be tailored to be specific to particular needs. They can be modelled on particular program patterns that match to specific libraries or algorithms.

Expression Serialization

To start exploring compiler support for Wolfram expressions, it is useful to look at the way that compiled functions are called from the Wolfram Language interpreter. When a compiled function is called, the input arguments are converted from expressions into the data required by the function. When the result comes back, it is converted from the data type into an expression. This is known as serialization, and its use for calling compiled code is shown below.

Built-in Serialization

Built-in compiler types all support serialization, in the obvious and sensible fashion. For example, this function takes an integer and returns a real:

The compiled function is called with an integer expression; this is converted into a machine 64-bit integer. The compiled code executes, returning a machine double-precision float. This result is then converted into a real expression and returned to the evaluator:

If the compiled code is called with something that it cannot convert, an error results:

This function takes a machine double-precision float and returns a complex:

This calls the function, which works as expected:

Note that if the function is called with something that is not a real expression, but which can be converted unambiguously into a machine float, it works:

Of course, things that cannot be converted will lead to errors:

Other built-in types such as packed arrays work:

Other conversions just work as expected. A list vector of strings is converted to an expression that is a list of strings:

Calling compiled code from the Wolfram evaluator involves deserializing inputs from expressions and serializing the result to an expression.

InertExpression

One expression serialization that is particularly simple is serialization of an expression.

This function takes an expression and returns it. It uses the "InertExpression" type, which is used to represent expressions in compiled code:

This demonstrates a call to the compiled function. It will accept anything from the Wolfram evaluator and return it:

Data Structures

When compiled code uses data structures, these pass in and out of compiled code easily.

This function creates a bit vector and returns it:

The result is just the same as a data structure that was created directly in interpreted code:

The bit vector can be passed back into compiled code and worked on there:

This seamless way that the Wolfram Language can work in interpreted and compiled code is very powerful. It allows core expensive functions to be compiled, but other functions to work in the more flexible interpreter.

Product Serialization

Product types can be declared with TypeDeclaration, and as seen previously, if they are given the "DefaultSerializable" abstract type, they will serialize into expressions very conveniently:

Compile a function that returns an instance of the product and another function that extracts some data:

Create an instance of the object. It becomes a data structure expression:

You can call different functions passing in the object:

This is using the same technology for embedding data types in an expression as is used in the data structure library.

Custom Serialization

It is possible to take control of expression serialization so that compiler types can be converted into and out of any particular form of expression.

For demonstration, here is a declaration of a product type called "Demo":

A function that passes the type in or out of compiled code cannot be compiled because the system does not know how to convert it into an expression:

Conversion functions can be added using declarations for CompilerCallback. First is a declaration to serialize into an expression. Note that "InertExpression" is the compiler type for an expression:

This is a function that tests its argument to see if it can be deserialized into a specific type. Note how the target type is passed in as a "TypeSpecifier" type wrapping the desired type (as is common for type processing functions). By passing in a specific type, the specific function is selected. More information on working with types like this is found in the section Programming With Types:

Finally, this is the function that actually does the conversion from an expression into the desired type. This will return an instance of the type. It should not be written using "TypeSpecifier":

Collect all the declarations together:

Now a function that returns an instance of "Demo" can be compiled:

Calling the compiled code returns the expression set up by the custom serialization:

This function takes an input of "Demo" and returns one of its fields:

It can work to take the expression returned previously:

It can also take a completely new expression. It does not matter whether the expression input was generated by compiled code or not:

Compiled Code and Expressions

Compiled code can work with Wolfram Language expressions in many ways. The previous section showed how conversion between general data types and expressions is important to call compiled code from the Wolfram Language interpreter. It also showed how this conversion code can be written using the compiler.

This section will cover more of the ways that compiled code can work with Wolfram Language expressions. One first thing to note is that in compiled code, expressions are handled with the "InertExpression" compiled type. The name is chosen to indicate that instances of the type are different from expressions in the Wolfram evaluator. In the evaluator, expressions automatically evaluate, but in compiled code they are inert and only evaluate when special action is taken.

Creating Expressions

Expressions can be passed into and returned from compiled code. They can also be created in compiled code. For example, using InertExpression, as in the following:

This calls the compiled code:

The expression created is the same as one created by evaluated code:

One thing to note about InertExpression is that its elements are different instances from any compiler variables with the same name. For example, the x in the InertExpression is not the same as the local variable x:

This is similar to the way that characters in a string are different from variables with the same name:

Note that InertExpression can only be used in compiled code:

In order to build more complicated expressions that use compiler variables, you can use an "InertExpression" as a head.

The following function takes an expression and uses it as a head for an expression with two arguments. Note how the arguments are not expressions themselves; they get converted:

Call the function:

The result is the same as if the expression was created by the evaluator:

The arguments to expression-building functions are regular compiler variables, so they do reflect local variables:

This is similar to how variables can be converted to strings:

A key aspect of instances of "InertExpression" is that they are inert; they do not evaluate. This makes compiled and evaluated code that works with expressions different. It also gives a lot of interesting opportunities.

The following function creates an expression and wraps it in Hold:

This calls the function:

Note how the expression does not evaluate:

Stripping the Hold with ReleaseHold causes it to evaluate:

If you want to make held expressions in the compiler, it might be useful to make a declaration that does this:

Compile a function that uses the declaration:

Run the compiled code:

Evaluating Expressions

In order to evaluate instances of "InertExpression", you can use InertEvaluate. The following function constructs an expression and evaluates it. The head of the result is returned:

If Plus is passed in as an argument, then the expression will evaluate, returning something different:

The expression may evaluate to itself, as in this example:

If your purpose in calling the evaluator is to incorporate evaluated code into compiled code, then this is also possible by using functions such as KernelFunction and KernelEvaluate, which are described in another section.

If you want to call the evaluator for printing type operations, this is not necessary because Print and Echo are supported for general data types, as described in the section on Input and Output.

However, if you want to call evaluated code but need to make some preparation of arguments, then InertEvaluate might be useful. The following compiled code can open a file and read from it:

This opens a sample data file:

This reads one number:

This reads two more:

It is important to recognize that when things are read from the file, they are returned as expressions. This is shown by using Information to see the return type:

General Functionality

Compiled versions of many Wolfram Language functions are supported for the "InertExpression" type:

You can explore which functions are supported using CompilerInformation:

Not all functions are supported, as in this example:

Typically, it is fairly easy to make a call with InertEvaluate to add what you need:

Expression Conversions

Many uses of expressions in compiled code are more useful if they can be converted into and out of other compiler data types. This is known as expression serialization and is very important for calling compiled code from the evaluator. It was covered in a previous section.

Inside compiled code, conversion between data types is carried out by the function Cast. When this involves the "InertExpression" type, it typically uses expression serialization.

Built-in Conversion

Most built-in types support expression serialization and therefore also support casting with expressions.

The following function takes an expression and casts it to an integer:

The function can be called:

If it is called with an expression that is not an integer, an error results:

This function casts in the opposite way: from an integer to an expression:

These functions look like they are doing the same thing, but they differ in their types:

Expressions are more general than compiler data types, and so conversions to expressions rarely fail, while conversions from expressions can fail (if the expression does not match the requirements of the type).

Data Structure Conversion

It is very straightforward to convert data structures to and from expressions.

This function creates a bit vector and casts it to an expression:

The result can be used as a regular bit vector data structure:

It is just the same as a data structure that was created directly in interpreted code:

Product Conversion

If you declare a product type and set it up to be default serializable, then casting will also work:

Compile a function that returns an instance of the product and functions that cast to and from the type:

Create an instance of the type:

Convert an expression to the type:

Convert an instance of the type to an expression:

As previously, these functions look like they are doing the same thing, but they differ in their types:

Custom Conversion

You can write your own conversion functions to convert between compiled types and expressions.

For demonstration, here is a declaration of a product type called "Demo":

This declaration implements a cast from "Demo" to an "InertExpression". It is very similar to the serialization example seen previously and could use the serialization code that was used previously:

Collect all the declarations together:

This function creates an instance of "Demo" and casts it to an expression. The result type of the function is "InertExpression":

Call the function. The result is the expression form of "Demo":

Wolfram Patterns

The Wolfram Compiler has quite interesting support for Wolfram patterns. This is a developing area with new functionality being added with every release. One of the motivations for the pattern-matcher support is to support DownValuesFunction.

DownValuesFunction

Compilable code can be written to use Wolfram Language functions written with :=, which is an input form for SetDelayed. An example is shown below:

This calls the function declaration:

To use this definition in compiled code, you can use DownValuesFunction:

This calls the compiled version:

It is also possible to use a function declaration:

A function that uses the declaration can be compiled and then used:

This calls the compiled version:

Here, timing information for the uncompiled version is collected:

This collects timing for the compiled version:

The compiled code is quite a lot faster.

Multiple Declarations

You can give different declarations, and they all get used for the declarations:

The different versions are all used by one declaration:

These different examples all call the appropriate code:

Note that all the versions of the function have to be able to compile to the type of any declaration. This version has one function declaration returning an argument and the other a string:

Compiling for a result of an integer fails, because in this case a string cannot be returned:

If the type of the result is a string in all cases, then compilation works. Here the first argument is a string:

The two different versions can both be called. This is the general case:

This is the case when the second argument is 2:

Literals

Function declarations have type information and this is used when processing patterns. So when the patterns contain literals these are set to match types as well as value. This function uses a literal 1, but gives the pattern a name x:

Any call a 1 as argument will match:

This carries over to compiled code. Here the literal will be an "Integer64":

Here the literal will be an "Integer8":

Here the literal will be an "InertExpression":

Here the literal is the symbol t, but a compilation for an "InertExpression" works fine:

Nested Patterns

Patterns that are nested inside other arguments are supported for "InertExpression". In this example, the pattern has some structure. The output is written to use IfCompiled to allow it to work for both compiled and uncompiled invocations:

This declares and compiles a call to the function:

Called with arguments that match the pattern gives the expected result:

If the pattern does not match, there is an error. This could be avoided by giving another definition of the function that covered the general case:

The function can also be called with interpreted code to get the same result:

Variable-Length Patterns

Patterns that have variable-length constructs such as RepeatedBlank are supported for "InertExpression" as in this example. The output is written to use IfCompiled to allow it to work for both compiled and uncompiled invocations:

This declares and compiles a call to the function:

The function can also be called with interpreted code to get the same result:

Pattern Constructs Supported

This is a summary of the pattern constructs which are supported.

Low Level Pattern Functionality

Pattern functionality in the Wolfram Compiler works by generating a function that represents the pattern and then compiling this. This is an interesting new development for the Wolfram Language.

It is possible to see the function that is generated with some low-level code. This gives some insight into how the functionality works and how to get the best from it:

This is a single definition, but with a literal check on one of the arguments:

The generated function has a Which with two branches. The last one is always for the case where no patterns match:

This single definition has no checks on its arguments:

Now one of the branches will never be called because the pattern will always be called with the number of arguments given in the type. The compiler will always remove the code for that by optimization, so there is no need to remove it:

In this example, there are two definitions:

Now there are three cases to be checked. The condition is given as a nested function to be compiled by the compiler:

This example uses Except:

Now the sense of one of the tests is reversed by the addition of Not: