Neural Networks

The Wolfram Language has state-of-the-art capabilities for the construction, training and deployment of neural network machine learning systems. Many standard layer types are available and are assembled symbolically into a network, which can then immediately be trained and deployed on available CPUs and GPUs.

Automated Machine Learning

Classify automatic training and classification using neural networks and other methods

Predict automatic training and data prediction

FeatureExtraction automatic feature extraction from image, text, numeric, etc. data

LearnDistribution automatic learning of data distribution

ImageIdentify fully trained image identification for common objects

Prebuilt Material

NetModel complete pre-trained net models

ResourceData access to training data, networks, etc.

Net Representation

NetGraph symbolic representation of trained or untrained net graphs to be applied to data

NetChain symbolic representation of a simple chain of net layers

NetPort symbolic representation of a named input or output port for a layer

NetExtract extract properties and weights etc. from nets

NetInformation give summary and detailed information about any network

Net Operations

NetTrain train parameters in any net from examples

NetInitialize randomly initialize parameters for a network

NetPortGradient differentiate a net with respect to a port

NetStateObject store and reuse recurrent state in a net

NetTrainResultsObject represent what happened in net training

NetMeasurements measure the performance of a net on test data

TrainingProgressMeasurements measure performance metrics during training

LearningRate  ▪  TrainingStoppingCriterion

Basic Layers

LinearLayer trainable layer with dense connections computing

ElementwiseLayer apply a specified function to each element in a tensor

SoftmaxLayer layer globally normalizing elements to the unit interval

Loss Layers

MeanSquaredLossLayer  ▪  MeanAbsoluteLossLayer  ▪  CrossEntropyLossLayer  ▪  ContrastiveLossLayer  ▪  CTCLossLayer

Elementwise Computation Layers

ElementwiseLayer  ▪  ThreadingLayer  ▪  ConstantTimesLayer  ▪  ConstantPlusLayer

Structure Manipulation Layers

CatenateLayer  ▪  PrependLayer  ▪  AppendLayer  ▪  FlattenLayer  ▪  ReshapeLayer  ▪  ReplicateLayer  ▪  PaddingLayer  ▪  PartLayer  ▪  TransposeLayer  ▪  ExtractLayer

Array Operation Layers

ConstantArrayLayer embed a learned constant array into a NetGraph

SummationLayer  ▪  TotalLayer  ▪  AggregationLayer  ▪  DotLayer  ▪  OrderingLayer

Convolutional and Filtering Layers

ConvolutionLayer  ▪  DeconvolutionLayer  ▪  PoolingLayer  ▪  ResizeLayer  ▪  SpatialTransformationLayer

Recurrent Layers

BasicRecurrentLayer  ▪  GatedRecurrentLayer  ▪  LongShortTermMemoryLayer

Sequence-Handling Layers

EmbeddingLayer trainable layer for embedding integers into continuous vector spaces

SequenceLastLayer  ▪  SequenceReverseLayer  ▪  SequenceMostLayer  ▪  SequenceRestLayer  ▪  UnitVectorLayer

AttentionLayer trainable layer for finding parts of a sequence to attend to

Training Optimization Layers

ImageAugmentationLayer  ▪  BatchNormalizationLayer  ▪  DropoutLayer  ▪  LocalResponseNormalizationLayer  ▪  NormalizationLayer

Higher-Order Network Construction

NetMapOperator define a network that maps over a sequence

NetMapThreadOperator define a network that maps over multiple sequences

NetFoldOperator define a recurrent network that folds in elements of a sequence

NetPairEmbeddingOperator  ▪  NetNestOperator  ▪  NetBidirectionalOperator

Network Surgery

NetDrop  ▪  NetTake  ▪  NetAppend  ▪  NetPrepend  ▪  NetJoin

NetDelete  ▪  NetInsert  ▪  NetReplace  ▪  NetReplacePart

NetFlatten  ▪  NetRename

Weight Sharing

NetSharedArray represent an array shared between several layers

NetInsertSharedArrays convert all arrays in a net into shared arrays

Encoding & Decoding

NetEncoder convert images, categories, etc. to net-compatible numerical arrays

"Audio"  ▪  "AudioMelSpectrogram"  ▪  "AudioMFCC"  ▪  "AudioSpectrogram"  ▪  "AudioSTFT"  ▪  "Boolean"  ▪  "Characters"  ▪  "Class"  ▪  "Function"  ▪  "Image"  ▪  "Image3D"  ▪  "Tokens" "BPESubwordTokens" "UTF8"

NetDecoder interpret net-generated numerical arrays as images, probabilities, etc.

"Boolean"  ▪  "Characters"  ▪  "Class"  ▪  "CTCBeamSearch"  ▪  "Image"  ▪  "Function"  ▪  "Image3D"  ▪  "Tokens" "BPESubwordTokens"

Activation Functions

Ramp rectified linear (ReLU)

Tanh  ▪  LogisticSigmoid  ▪  Exp  ▪  Log  ▪  Sin  ▪  Cos  ▪  Sqrt  ▪  Abs

Importing & Exporting

"WLNet" Wolfram Language Net representation format

"MXNet" MXNet net representation format

Import  ▪  Export

Managing Data & Training

ClassifierMeasurements measure accuracy, recall, etc. of a classifier net

DeleteMissing remove missing data before training

LossFunction  ▪  TargetDevice  ▪  ValidationSet  ▪  TrainingProgressFunction  ▪  TrainingProgressCheckpointing  ▪  TrainingProgressReporting  ▪  TrainingStoppingCriterion  ▪  TrainingProgressMeasurements  ▪  LearningRate  ▪  LearningRateMultipliers  ▪  NetEvaluationMode

Reinforcement Learning Environments

"OpenAIGym", ... access to video games and many other test environments