WXF Format Description

WXF is a binary format for faithfully serializing Wolfram Language expressions in a form suitable for outside storage
or interchange with other programs. WXF can readily be interpreted using low-level native types available in many
programming languages, making it suitable as a format for reading and writing Wolfram Language expressions in other
programming languages.
The basic functions for converting between a Wolfram Language expression and its serialized form are BinarySerialize
and BinaryDeserialize. Support for reading and writing files with WXF data is built into Export and Import.

BinarySerialize[expr]        gives a binary representation of any expression expr in the WXF format
BinaryDeserialize[bytearray] recovers an expression from a binary representation in the WXF format
Import[file,"WXF"]           imports a WXF file and returns an expression
Export[file,expr,"WXF"]      serializes an arbitrary expression and saves it as a WXF file
ImportByteArray[ba,"WXF"]    imports data and returns an expression
ExportByteArray[expr,"WXF"]  generates a ByteArray object corresponding to expr exported in the WXF format

There are many ways to serialize and deserialize WXF in the Wolfram Language.

Basic Structure

Data in WXF form always contains a plain ASCII header followed by a string of bytes. The header specifies how the bytes
can be decoded and is separated by a colon from the string of bytes that represents a sequence of parts.

1.gif

The byte array continues by giving a sequence of parts, each starting with a token from the following list that
specifies the type of the part.

byte value character representation (ISO8859-1) type of part
102        "f"                                  function
67         "C"                                  signed 8-bit integer
106        "j"                                  signed 16-bit integer
105        "i"                                  signed 32-bit integer
76         "L"                                  signed 64-bit integer
114        "r"                                  IEEE double-precision real
83         "S"                                  string
66         "B"                                  binary string
115        "s"                                  symbol
73         "I"                                  big integer
82         "R"                                  big real
193        "Á"                                  packed array
194        "Â"                                  numeric array
65         "A"                                  association
58         ":"                                  delayed rule in association
45         "-"                                  rule in association

The exhaustive list of WXF tokens.
After the token comes, if necessary, a length specification, followed by the sequence of actual content elements for
the part.

Basic Examples

Give the bytes for the serialized form of a symbol:


Wolfram Language code: BinarySerialize[Plot]


Wolfram Language code: Normal[%]

View the bytes as characters in "ISO8859-1":


Wolfram Language code: FromCharacterCode[%]

Give the bytes for the serialized form of a string as characters:


Wolfram Language code: ByteArrayToString[BinarySerialize["hello!"], "ISO8859-1"]

Give the bytes for the serialized form of Range[10]:


Wolfram Language code: bytes = Normal[BinarySerialize[Range[10]]]

Create a function to label pieces of the array:


Wolfram Language code: label[data_, label_] := Labeled[Framed[Row[data, ","]], label]

The first two bytes belong to the header, the rest corresponds to the packed array:


Wolfram Language code: Row[{label[bytes[[1 ;; 2]], "header, \":\" "], label[bytes[[3 ;; ]], "packed array"]}]

The header with the version followed by the separator:


Wolfram Language code: FromCharacterCode[{56, 58}]

The packed array with the type, the rank and dimensions, and finally the data:


Wolfram Language code: Row[{label[bytes[[3 ;; 4]], "type of array"], label[bytes[[5 ;; 6]], " rank, dimension "], label
[bytes[[7 ;; ]], "data"]}]

Examples with Multiple Parts

The examples in the previous section essentially consisted of a single part. There was a single token from the token
list, followed by size information, followed by the data. The examples in this section contain multiple parts and
types. The first example is shown by the following interactive illustration:

2.gif


Wolfram Language code: bytes = BinarySerialize[Select[OddQ]]

The first two bytes are the header and colon separator:


Wolfram Language code: {FromCharacterCode[bytes[[1]]], FromCharacterCode[bytes[[2]]]}

The next byte is the token for a function, followed by the number of arguments:


Wolfram Language code: {FromCharacterCode[bytes[[3]]], bytes[[4]]}

The next byte is the token of the function's head, in this case symbol:


Wolfram Language code: FromCharacterCode[bytes[[5]]]

After that is the number of bytes in the UTF-8 representation of the symbol:


Wolfram Language code: bytes[[6]]


Therefore, the next 6 bytes are the UTF-8 representation of the symbol:


Wolfram Language code: ByteArrayToString[bytes[[7 ;; 12]]]

The complete head being read, the next byte is the token of the first argument:


Wolfram Language code: FromCharacterCode[bytes[[13]]]

As this argument is a symbol, the next byte will be its length, followed by its name:


Wolfram Language code: {bytes[[14]], ByteArrayToString[bytes[[15 ;; 18]]]}

The function only had one argument, so all the bytes must be read by now:


Wolfram Language code: Length[bytes] == 18

The second example is a list of three elements. The list is represented as a function of length 3, with head List
followed by three parts. The first parts introduce the format used to represent integers in a compact form using a
token and an "Integer8". The last part shows the representation of a ByteArray:

3.gif

Give the bytes for the serialized form of {1,-1,ByteArray[{1,2,3}]}:


Wolfram Language code: list = Normal[BinarySerialize[{1, -1, ByteArray[{1, 2, 3}]}]]

Ignoring the header, this expression is a function of three arguments:


Wolfram Language code: {FromCharacterCode[list[[3]]], list[[4]]}

The head is a symbol of four bytes, namely List:


Wolfram Language code: Row[{label[list[[{5}]], "s"], label[list[[{6}]], "length"], label[list[[7 ;; 10]], "List"]},
Spacer[.5]]

After the head comes the first argument, namely the 8-bit integer 1:


Wolfram Language code: {FromCharacterCode[list[[11]]], list[[12]]}

The next two bytes are the second argument, namely the 8-bit integer -1:


Wolfram Language code: {FromCharacterCode[list[[13]]], list[[14]]}

Use Mod to interpret the byte as a signed value, converting 255 to the expected -1:


Wolfram Language code: Mod[list[[14]], 256, -128]

The third and final argument is a binary string of length 3 with values {1,2,3}:


Wolfram Language code: Row[{label[list[[{15}]], "B"], label[list[[{16}]], "length"], label[list[[17 ;; 19]],
"values"]}, Spacer[.5]]

More on the Header

The header is a plain ASCII string of variable length, delimited by the character ":". For the current version (1.0) of
WXF, the first byte in the header is the character "8" (i.e. byte value 56). When the binary serialization is zip
compressed, this is indicated in the header by the character "C". The header is never compressed; the compression only
applies to the following string of bytes.
Serialize an expression to give a byte array:


Wolfram Language code: BinarySerialize["body"]

The first byte is the character 8:


Wolfram Language code: ByteArrayToString[%, "ISO8859-1"]

Compression is indicated by the second character of the header being "C":


Wolfram Language code: ByteArrayToString[BinarySerialize["body", PerformanceGoal -> "Size"], "ISO8859-1"]

Length Encoding (Varint)

WXF types fall into three categories:

  • Atomic types of fixed length like machine-precision Integer or Real.

  • Atomic types of variable length like String or Symbol.

  • Types with a variable number of subparts like general expressions (token type "f" for function) or Association.

In WXF, all integers representing a length or a size are serialized using the varint method. A varint is a
self-indicating variable-length format, where smaller integers require fewer bytes. Each byte except the last one has
its most significant bit (MSB) set. The MSB indicates if the following byte of the stream is also part of the varint,
acting as a continuation marker. The lower seven bits of each byte store the binary representation of the integer, with
the least significant group first.
Here is an example of constructing the varint representation of 500:
Base-2 digits of 500:


Wolfram Language code: bits = IntegerDigits[500, 2]

Group bits by 7, with the least significant bits first:


Wolfram Language code: grouped7 = Partition[Reverse[bits], UpTo[7]]

Order bits of each group, with the most significant bit first, and pad each group so that they have 8 bits each:


Wolfram Language code: grouped8 = Map[Composition[PadLeft[#, 8]&, Reverse], grouped7]

Set the first bit of each byte except the last one to get the varint binary form:


Wolfram Language code: varint = ReplacePart[grouped8, {i_, 1} /; i < Length[grouped8] :> 1]

Construct byte values from base-2 digits, to give the varint representation of 500:


Wolfram Language code: Map[FromDigits[#, 2]&, varint]

The reverse operation of decoding the varint encoded byte sequence {244,3} back to 500 is explained in the following
illustration, in which each initial varint bit is assigned a unique color:

4.gif

Strings, Symbols and Non-Machine Numbers

Strings, symbols and non-machine numbers are represented using the same format. The first byte is a token, then the
byte count encoded in the varint format, followed by a string of Unicode characters encoded as UTF-8 corresponding to
the string InputForm of the expression. A non-machine number can be either an arbitrary-precision real or an integer
that requires more than $SystemWordLength bits to represent.

type of atom            token representation
String                  "S"   the Unicode character sequence
Symbol                  "s"   the fully qualified name of the symbol, specifying the context, except for System`
                              symbols
Arbitrary-precision     "R"   the digit representation specifying the mantissa and eventually the precision and the
reals                         exponent
Big integers            "I"   the string of digits

Types based on InputForm.
Serialize the first 500 characters of Alice in Wonderland:


Wolfram Language code: bytes = BinarySerialize[StringTake[ExampleData[{"Text", "AliceInWonderland"}], 500]]

The first byte after the header is the token for a string:


Wolfram Language code: FromCharacterCode[bytes[[3]]]

The next two bytes are 500 in the varint encoding as seen in the preceding example:


Wolfram Language code: Normal[bytes[[4 ;; 5]]]

The remaining bytes are the string contents in UTF-8:


Wolfram Language code: ByteArrayToString[bytes[[6 ;; -1]]]

Serialize a non-machine integer:


Wolfram Language code: bytes = BinarySerialize[10 ^ 127]

Other than the token, the serialization is the same as for a string:


Wolfram Language code: {FromCharacterCode[bytes[[3]]], Normal[bytes[[4 ;; 5]]], ByteArrayToString[bytes[[6 ;; -1]]]}

A number with one fewer digit requires only a single byte to encode the length:


Wolfram Language code: bytes = BinarySerialize[10 ^ 126];


Wolfram Language code: {FromCharacterCode[bytes[[3]]], bytes[[4]], ByteArrayToString[bytes[[5 ;; -1]]]}

Machine Integers Serialization

Machine integers are identified by the smallest integer type from the following list that can represent the value,
followed by the two's complement representation of the integer. The byte ordering is always little endian.

token definition             type size
"C"   signed 8-bit integer   [5]
"j"   signed 16-bit integer  [8]
"i"   signed 32-bit integer  [11]
"L"   signed 64-bit integer  [14]

Integer tokens, their associated types and the number of bytes used by each representation.
The bytes corresponding to the serialization of 2^14:


Wolfram Language code: bin = BinarySerialize[2 ^ 14]

Skip the two-byte header and display the first token as a character:


Wolfram Language code: FromCharacterCode[bin[[3]]]

View the bytes after the token:


Wolfram Language code: Normal[bin[[4 ;; -1]]]

Convert the preceding pair of bytes to an integer:


Wolfram Language code: bin[[4]] + BitShiftLeft[bin[[5]], 8] == 2 ^ 14

Negative integers binary representation uses the two's complement method. Given a N-bit integer α, its two's complement
β is its complement with respect to 2^N: α+β=2^N. Negation of a number is performed by taking the two's complement.
The two's complement of the 8-bit integer 1:


Wolfram Language code: 2^8-1

The two's complement is the 8-bit binary representation of -1:


Wolfram Language code: Normal[ExportByteArray[-1, "Integer8"]]

Serialize a negative 16-bit integer:


Wolfram Language code: bytes = BinarySerialize[-10000]

The last two bytes are the integer value:


Wolfram Language code: bytes[[-2 ;; ]]//Normal

Convert the pair of bytes to its decimal form:


Wolfram Language code: bytes[[-2]] + BitShiftLeft[bytes[[-1]], 8]

The value is the 16-bit two's complement of 10000:


Wolfram Language code: 10000 + % == 2 ^ 16

Machine Reals Serialization

Machine reals are represented using the character "r" followed by the memory representation of a double floating-point
value in the IEEE 754 standard. As for machine integers, the byte ordering is always little endian.
Serialize a real number:


Wolfram Language code: bytes = BinarySerialize[4.]

Skip the two-byte header and display the first token as a byte and as a character:


Wolfram Language code: bytes[[3]]


Wolfram Language code: FromCharacterCode[bytes[[3]]]

The next bytes are the real value in the IEEE 754 standard:


Wolfram Language code: Normal[bytes[[4 ;; ]]]


Wolfram Language code: ExportByteArray[4., "Real64"] == bytes[[4 ;; ]]

Machine-precision complex numbers are serialized as a function of two machine-precision reals. The following
illustration highlights the Complex head followed by two real values:

17.gif

Serialize a machine complex number:


Wolfram Language code: bytes = Normal[BinarySerialize[4. + 4.I]]

The first bytes after the header are a function of length 2 with head Complex:


Wolfram Language code: Row[{label[{bytes[[3]]}, "f"], label[{bytes[[4]]}," length "],label[bytes[[5;;13]], "Complex"]}]

The next nine bytes are the real part and match the serialization of the real value 4. as shown in the previous
example:


Wolfram Language code: bytes[[14 ;; 22]]

The remaining bytes are the imaginary part, again the real value 4.:


Wolfram Language code: bytes[[23 ;; ]]

Function Serialization

Functions are represented in WXF by the character "f", followed by the expression length in the varint format. The
number of elements is equal to the length incremented by one, for the head. The head and the parts are arbitrary
serialized expressions. In particular, the head can also be a function: Select[OddQ][{1,2,3}] is a function of length 1
with head Select[OddQ], which itself is a function with head Select and length 1.
Serialize an expression, using Unevaluated to prevent it from evaluating:


Wolfram Language code: bytes = BinarySerialize[Unevaluated@Select[OddQ][{1, 2, 3}]]

The first two bytes correspond to a function of length 1:


Wolfram Language code: {FromCharacterCode[bytes[[3]]], bytes[[4]]}

The next 16 bytes are the serialization of the head Select[OddQ] shown previously:


Wolfram Language code: Normal[bytes[[5 ;; 20]]]

The remaining bytes are the argument, with the token and length first:


Wolfram Language code: {FromCharacterCode[bytes[[21]]], bytes[[22]]}

Followed by the head:


Wolfram Language code: {FromCharacterCode[bytes[[23]]], bytes[[24]], ByteArrayToString[bytes[[25 ;; 28]]]}

The three machine integers round out the expression:


Wolfram Language code: Normal[bytes[[29 ;; -1]]]

Associations Serialization

Associations are represented by the character "A", followed by the length and the rules.
An association's rules are represented by the character "-", and delayed rules by the character ":". It is immediately
followed by two arbitrary serialized expressions. The length of the association's rule is always two and thus is
omitted. The following illustration shows the serialization of a simple association:

18.gif

Serialize the association:


Wolfram Language code: bytes=Normal[BinarySerialize[<\|1->2, "a":>"b"\|>]]

The first bytes correspond to an association of two elements:


Wolfram Language code: {FromCharacterCode[bytes[[3]]], bytes[[4]]}

Then comes the rule that implicitly has two parts; the length is thus omitted:


Wolfram Language code: Row[{label[{FromCharacterCode[bytes[[5]]]}, "Rule"], label[bytes[[6 ;; 7]], "Key"], label[bytes
[[8 ;; 9]], "Value"]}, Spacer[.5]]

The delayed rule is the final part:


Wolfram Language code: Row[{label[{FromCharacterCode[bytes[[10]]]}, "RuleDelayed"], label[bytes[[11 ;; 13]], "Key"],
label[bytes[[14 ;; 16]], "Value"]}, Spacer[.5]]

Rules of the previous example were part of an association. Rule and RuleDelayed that are not part of an Association are
serialized as functions. The serialized form is less packed, as shown in the next example.

19.gif

Serialize a list of rules:


Wolfram Language code: bytes=Normal[BinarySerialize[{1->2, "a":>"b"}]]

The first bytes declare a list of two elements:


Wolfram Language code: Row[{label[{FromCharacterCode[bytes[[3]]]}, "f"], label[{bytes[[4]]}, "length"], label[bytes[[5
;; 10]], "List"]}, Spacer[.5]]

The first element is a function of length 2 with head Rule:


Wolfram Language code: Row[{label[{FromCharacterCode[bytes[[11]]]}, "f"], label[{bytes[[12]]}, " length "], label
[{FromCharacterCode[bytes[[13]]]}, " Symbol "], label[{bytes[[14]]}, " length "], label[bytes[[15 ;; 18]], "Rule"]}]

The rule's arguments remain the same as in the association case:


Wolfram Language code: bytes[[19 ;; 22]]

The second element is also a function of length 2, but its head is RuleDelayed:


Wolfram Language code: Row[{label[{FromCharacterCode[bytes[[23]]]}, "function"], label[{bytes[[24]]}, " length "],
label[{FromCharacterCode[bytes[[25]]]}, " Symbol "], label[{bytes[[26]]}, " length "], label[bytes[[27 ;; 37]],
"RuleDelayed"]}]

Similarly, the elements of the rule delayed remain unchanged:


Wolfram Language code: bytes[[38 ;; -1]]

The serialized length is roughly the size of the string FullForm:


Wolfram Language code: ToString[FullForm[{1 -> 2, "a" :> "b"}]]//StringLength

The association serializes to a more compact form:


Wolfram Language code: BinarySerialize[<|1 -> 2, "a" :> "b"|>]//Length

Binary Strings

Binary strings are represented by the token "B". They follow the same pattern as strings, but the byte sequence is
arbitrary rather than UTF-8 characters. A ByteArray is serialized as a binary string.
Serialize a byte array:


Wolfram Language code: bytes = Normal[BinarySerialize[ByteArray[{0, 42, 192}]]]

The first byte after the header is a binary string token, followed by the length of the binary data:


Wolfram Language code: Row[{label[{bytes[[3]]}, "B"], label[{bytes[[4]]}, "length"]}, Spacer[.1]]

The next bytes are the data:


Wolfram Language code: bytes[[5 ;; ]]

Decoding the bytes as UTF-8 does not always succeed:


Wolfram Language code: FromCharacterCode[bytes[[5 ;; ]], "UTF-8"];

[20]
The bytes can always be represented as a string using the "ISO8859-1" encoding:


Wolfram Language code: FromCharacterCode[bytes[[5 ;; ]], "ISO8859-1"]

Numeric Arrays

Arrays are multidimensional tables of machine-precision numeric values. Arrays are represented by the following
sequence: a token specifying the type of array, a token specifying the type of the values, the rank in the varint
format, the dimensions as a sequence of integers also in the varint format and finally, the data.
There are two types of arrays in the WXF format: packed arrays represented by the token "Á" (byte value 193) and
numeric arrays represented by the token "Â" (byte value 194). There are slight differences between the two, the major
one being the supported value type, as described in the following tables.

integer value value in hexadecimal representation type of array
0             00[16]                              array of 8-bit signed integers
1             01[16]                              array of 16-bit signed integers
2             02[16]                              array of 32-bit signed integers
3             03[16]                              array of 64-bit signed integers (64-bit system only)
34            22[16]                              array of IEEE single-precision real numbers (float)
35            23[16]                              array of IEEE double-precision real numbers (double)
51            33[16]                              array of IEEE single-precision complex numbers
52            34[16]                              array of IEEE double-precision complex numbers

The valid value type tokens for packed arrays.

integer value value in hexadecimal representation type of array
0             00[16]                              array of 8-bit signed integers
16            10[16]                              array of 8-bit unsigned integers
1             01[16]                              array of 16-bit signed integers
17            11[16]                              array of 16-bit unsigned integers
2             02[16]                              array of 32-bit signed integers
18            12[16]                              array of 32-bit unsigned integers
3             03[16]                              array of 64-bit signed integers
19            13[16]                              array of 64-bit unsigned integers
34            22[16]                              array of IEEE single-precision real numbers (float)
35            23[16]                              array of IEEE double-precision real numbers (double)
50            33[16]                              array of IEEE single-precision complex numbers
51            34[16]                              array of IEEE double-precision complex numbers

The valid value type tokens for numeric arrays.
The integer range supported by packed arrays varies with the system word length, $SystemWordLength, from -2^31 to 2^
31-1 on a 32-bit environment, and from -2^63 to 2^63-1 on a 64-bit environment.
Define a matrix:


Wolfram Language code: matrix = Table[1000i + j, {i, 2}, {j, 3}]

Serialize it as a packed array:


Wolfram Language code: bytes = Normal[BinarySerialize[Developer`ToPackedArray[matrix]]]

The first byte after the header is the packed array token:


Wolfram Language code: bytes[[3]]

The next byte indicates an array of 16-bit signed integers:


Wolfram Language code: bytes[[4]]

The following bytes are the rank and dimensions of the array:


Wolfram Language code: Row[{label[{bytes[[5]]}, "rank"], label[bytes[[6 ;; 7]], "dimensions"]}, Spacer[.1]]

The trailing bytes are the values:


Wolfram Language code: bytes[[8 ;; ]]

It is possible to reconstruct the decimal form of each 16-bit integer. First, group the pair of bytes:


Wolfram Language code: int16 = Partition[bytes[[8 ;; ]], 2]

Each pair is a little endian 16-bit long integer whose value is reconstructed using a bit shift operation:


Wolfram Language code: Map[First[#] + BitShiftLeft[Last[#], 8]&, int16]

The interactive illustration following shows the serialization of the previous matrix before packing. The sequence of
elements is significantly different, since it involves nested functions with head List. The inner lists have three
parts corresponding to the integer values. It is worth noting that the binary representation of the integer values is
similar to the one witnessed in the packed array case (little endian signed 16-bit integer).

21.gif

Array value type tokens are constructed as bit fields. The four least significant bits store the log of the size of the
numeric type in bytes, and the four most significant bits represent the numeric type. Note that for complex types, the
size refers to the whole number, so that, for example, the single-precision complex type is considered to have a size
of 8 bytes.

0000  ●
0001  ●
0010  ●
0011  ●
0100  ●

The four least significant bits of the value type token and the corresponding type sizes in bytes.

0000 integer
0001 unsigned integer
0010 real
0011 complex

The four most significant bits of the value type token and the corresponding numeric types.
It is possible to construct the bit field corresponding to an array of double-precision reals using the Wolfram
Language.
A double-precision real is 8 bytes long; the logarithm to base 2 is 3:


Wolfram Language code: Log[2, 8]

Find the bit representation:


Wolfram Language code: lsb = IntegerDigits[%, 2, 4]

View the bit field made from the concatenation of the numeric type corresponding to a real number and the type size:


Wolfram Language code: msb = {0, 0, 1, 0};


Wolfram Language code: Row[{label[msb, "real type"], label[lsb, " 8 byte long "]}]

Convert the preceding bit sequence to retrieve the expected byte value:


Wolfram Language code: FromDigits[Join[msb, lsb], 2] == 35