Transforming XML
Introduction
Mathematica is uniquely suited for processing symbolic expressions because of its powerful pattern-matching abilities and large collection of built-in structural manipulation functions. This tutorial provides a few examples to illustrate the use of
Mathematica for processing XML data.
When you import an arbitrary XML document into
Mathematica, it is automatically converted into a symbolic XML expression. Symbolic XML is the format used for representing XML documents in
Mathematica syntax. The conversion to symbolic XML translates the XML document into a
Mathematica expression, while preserving its structure. The advantage of converting XML data into symbolic XML is that you can directly manipulate symbolic XML using any of
Mathematica's built-in functions.
Convert an XML string into symbolic XML.
| Out[452]= |  |
Use a simple transformation rule to remove the unwanted
element from the list.
| Out[453]= |  |
Use
ExportString to convert symbolic XML into native XML syntax, which was designed to be easy to read.
| Out[454]= |  |
Visualizing the XML Tree
Many XML tools display an XML document as a collapsible tree, where the nodes correspond to the elements of the document. This example shows how to produce a similar visualization using cell grouping in a
Mathematica notebook.
You will do this by recursively traversing the symbolic XML expression and creating a
CellGroupData expression that contains cells for each of
XMLElement object's attributes and children. Each nested
CellGroupData expression will be indented from the previous one. You start with the function to process an
XMLElement object.
Use the integer
m for indentation. When you map
onto the
XMLElement object's children, you pass a larger value for
m, increasing the indentation for the child elements.
A
CellGroupData expression contains a list of cells. In the definition, you have only created a cell for the
XMLElement x. However, you have then mapped
onto the attribute list. Since this returns a list, you need to use
Apply[ Sequence] to the result in order to merge that list into the
CellGroupData expression's list of cells. You then do the same thing to the children of the
XMLElement.
However, you have not yet defined
to work on attributes. The attributes of an
XMLElement object are stored in symbolic XML as rules. In most cases, the rule contains two strings: the key and the value. However, when namespaces are involved, the first element of the rule may be a list containing two strings: the namespace and the key. You will need to make two definitions to handle the attributes.
We will need one more definition in order to process simple symbolic XML expressions. The text nodes in an XML document are stored simply as
String objects in symbolic XML. Thus, you need a definition that handles
String objects.
Construct a simple notebook to visualize a basic XML document.
| Out[459]= |  |
| Out[460]= |  |
Since the default value of the option
is
None, you did not alter comments, processing instructions, or anything else that would be stored in an
XMLObject. Adding definitions for these is not difficult and would be a good exercise in processing symbolic XML.
Manipulating XML Data
XML applications are used for more than just document layout. XML is also an excellent format for storing structured data. Many commercial database vendors are now adding XML support to their products, allowing you to work with databases using XML as an intermediate format.
Mathematica's symbolic pattern-matching capabilities make it an ideal tool for extracting and manipulating information from XML documents. To illustrate this, manipulate an XML file containing data on major league baseball players.
Import this file into
Mathematica as symbolic XML.
Each player's information is stored in a
PlayerRecord element. Extract this with
Cases.
| Out[463]= |  |
The XML document contains records for 294 players. Inside each
PlayerRecord, there is a
TEAM element which specifies a player's team. By passing a slightly more sophisticated pattern to
Cases, you can extract a list of all players on the Yankees team.
| Out[465]= |  |
The variable
yankees now contains a list of symbolic XML expressions for all the Yankees players. Look at the first element of
yankees.
| Out[466]= |  |
The player's name is stored in the
PLAYER element of each
PlayerRecord element. Extract the name from one
PlayerRecord element.
| Out[467]= |  |
Use
Map to extract all the names from
yankees.
| Out[468]= |  |
Alternatively, use
Cases on
yankees with an appropriate pattern.
| Out[469]= |  |
Symbolic XML is a general-purpose format for expressing arbitrary XML data. In some cases, you may find it more useful to convert symbolic XML into a different type of
Mathematica expression. This type of conversion is easy to do using pattern matching.
Import an XML file containing data about baseball pitchers and translate the symbolic XML into a list of rules.
| Out[476]= |  |
All the information about the player is stored in a list of
Mathematica rules with
Pitcher as the head.
In addition to transforming the data into a different expression syntax, you can also modify the data and leave the overall expression in symbolic XML. This way you can alter our data, but still export it to an XML file for use with other applications. As an example, you will work with the salaries of American League hitters.
Delete any
PlayerRecord entries where the salary is not available.
Create a function to extract name-salary pairs. Then sort these pairs by salary and look at the top 10.
Out[479]//TableForm= |
| |  |
To illustrate changing data in symbolic XML, create a function that doubles players' salaries.
Out[482]//TableForm= |
| |  |
Comparing XSLT and Mathematica
In many situations, you need to transform a document from one XML format into another; one popular technique used for this purpose is XSLT transformations.
Mathematica's pattern-matching and transformation abilities allow you to do similar transformations by importing the original document and then manipulating the symbolic XML. This section gives examples of some basic XSLT transformations and shows how to do the equivalent transformations in
Mathematica.A Simple Template
In this example, XML dialect uses the
code tag to enclose program code. Typically, this is displayed in a monospace font. If you were to convert such a document to XHTML, you would probably want to use the
pre tag for code.
<xsl:template match="code">
<pre class="code">
<xsl:value-of select="."/>
</pre>
</xsl:template>
In
Mathematica, you can create a function to do the same.
Inserting Attribute Values
Now consider an XML application that uses the
termdef element to indicate the definition of a new term. You might want to anchor the definition with an element named
a so that you can link directly to that location in the document. Assuming you have templates to handle whatever string formatting is inside the
termdef element, you can use the following XSLT.
<xsl:template match="termdef">
<span class="termdef">
<a name="{@id}">[Definition:] </a>
<xsl:apply-templates/>
</span>
</xsl:template>
Notice that the
name attribute in the resultant XHTML gets the value of the
id attribute of the original
termdef element.
In
Mathematica, you can do the following.
Using Predicates
Consider a more complicated example, which will use XPath predicates. Assume you would like to match a
note element, but only if it either has a
role attribute set to
example or if it contains an
eg element as a child. Look at an XSLT template and then explain what it does.
<xsl:template match="note[@role='example' or child::eg]">
<div class="exampleOuter">
<div class="exampleHeader">Example</div>
<xsl:if test="*[1][self::p]">
<div class="exampleWrapper">
<xsl:apply-templates select="*[1]"/>
</div>
</xsl:if>
<div class="exampleInner">
<xsl:apply-templates select="eg"/>
</div>
<xsl:if test="*[position()>1 and self::p]">
<div class="exampleWrapper">
<xsl:apply-templates
select="*[position>1 and self::p]"/>
</div>
</xsl:if>
</div>
</xsl:template>
The first
xsl:if element checks to see if the first child element is a
p element. If it is, then
xsl:apply-templates is called on that child. This is similar to calling
Map across the results of
Cases. In the second
xsl:if element, you check if there are
p child elements beyond the first child. If so,
xsl:apply-templates is called on those. Here is the corresponding
Mathematica code.
| Out[508]= |  |
Traversing Upward
To select an ancestor or sibling, you have to realize that an XML document is just a stream of characters that follows a grammar. Tools for manipulating XML documents treat XML according to some model. In the case of XSLT (and its path-selection language, XPath), this model is that of a tree. Since
Mathematica is a list-based language, it treats XML as nested expression lists.
While these two models are similar, they have important differences. Most notably, in nested lists you do not inherently have any concept of the containing list. Technically, any transformation that can be done with axis types like ancestor can also be done without them. However, it is often convenient to traverse up the XML document.
Next is an example and a discussion of how to implement the same behavior in
Mathematica. Consider the following XML document.
| Out[18]= |  |
Assume you simply want to have a template that matches
bibref elements and replaces them with the text inside of the corresponding
bibl element. In XSLT, you would write the following template.
<xsl:template match="bibref">
<xsl:param name="ref">
<xsl:value-of select="@ref"/>
</xsl:param>
<xsl:value-of select="/bibliography/bibl[@id = $ref][1]"/>
</xsl:template>
The problem with using the same approach in
Mathematica is that once you have matched a
bibref element, you no longer have any information about the elements containing it. As a remedy, you will instead pass an expression containing the entire symbolic XML expression. Notice that the
bibref element in question can be obtained from the following.
| Out[510]= |  |
You can pass this expression wrapped in
Hold, so that you can easily obtain the
bibref element by calling
ReleaseHold. You can access ancestors by dropping indices from the
Part expression. However, you will need to write a pattern-matching function so that you can match these in definitions of functions.
| Out[512]= |  |
The
Mathematica transformation then becomes relatively simple.
| Out[515]= |  |
Converting a Notebook to HTML
Suppose you need to export a notebook in a specific XML format (one not listed under the dialog's
Save As Type: submenu). One option would be to export to ExpressionML and then use some external tool (e.g., XSLT rules) to transform to the desired form of XML. But often it is just as easy to perform the manipulation within
Mathematica, converting the notebook expression directly into symbolic XML and saving the latter. Anyone with a basic command of
Mathematica patterns and programming should be able to do this. Users coming from an XSLT background may even feel a sense of deja vu; since
Mathematica expressions are essentially trees, the techniques are much the same.
As an example, re-create an abridged version of the functionality.
Create an example notebook.
| Out[93]= |  |
Define a recursive function,
transform, to process the original notebook expression from top to bottom, similar to the templates of XSLT.
Establish a default definition to discard anything not explicitly matched by other patterns.
The definition uses
Sequence
since
transform will be applied recursively. The best "null'' result is one that can be dropped in the midst of a list of arguments without disrupting the syntax.
Start with the notebook expression itself.
- The argument pattern must be robust enough to accept all variants. (Even though the notebook options are discarded in this conversion, a BlankNullSequence (
) is included to allow for them.)
- The only thing done with the contents argument is to pass it back to transform .
- The third argument is always a List. Forgetting this is a common pitfall.
- Those familiar with HTML will notice that the head element has been dropped.
The same general theme is followed for the remaining definitions.
Discard cell-grouping information, since the HTML has no use for it.
Translate to
Mathematica sectional heads their HTML counterparts.
Since the contents of a
Mathematica Text style cell can be a simple string or a list wrapped with
if the text has substructure of its own, create a definition for both cases.
Simple strings should just be passed on.
Deal with (simple) font changes.
Here is the final product.
| Out[105]= |  |
Get output in a more human-readable form by using
ExportString.
| Out[106]= |  |
Verify that this is well-formed XML.
| Out[107]= |  |
The symbolic XML can be exported to a file, suitable for viewing with a web browser.
| Out[108]= |  |
An alternative is to apply a list of replacement rules using
ReplaceRepeated (
//.).
| Out[110]= |  |
The two methods produce identical results.
| Out[111]= |  |
Here is how the two methods differ.
- Since the recursion occurs implicitly via ReplaceRepeated, the latter implementation is cleaner in spots. In particular, contrast the handling of Text cells: the
rule can be separated from the Cell rule. The same could be accomplished for the recursive function, but at the cost of additional patterns for the various forms that contents might take (for example, 
versus 
and so on). ReplaceRepeated, by acting on all subexpressions, obviates this need.
Verifying Symbolic XML Syntax
You can use
XML`SymbolicXMLErrors to find errors in a symbolic XML expression. This function returns a part specification that you can use with functions like
Part or
Extract to access the problematic part of your symbolic XML expression.
Import an XML file containing data about the American League hitters.
Assume that any player whose salary is
is making $1,000,000. Perform the transformation, but make an easily overlooked mistake and use the string
as the third element, rather than a list containing that string.
Export produces errors when writing modified XML to a file. Use
SymbolicXMLErrors to find the problematic expressions.
| Out[539]= |  |
XMLElement::cntsList messages are the first indication that something is wrong. The output of
SymbolicXMLErrors, however, shows exactly where something went wrong.
ALerrors now contains a list of part specifications where the errors occurred. Examine the first error.
| Out[540]= |  |
| Out[541]= |  |
| Out[542]= |  |
This problem is easy enough to fix.
| Out[543]= |  |
The rest of the errors are of the same nature.
| Out[544]= |  |
Use
Map to fix the rest of the errors in the same way.
| Out[545]= |  |
| Out[546]= |  |
