TrainingUpdateSchedule

TrainingUpdateSchedule

is an option for NetTrain that specifies which arrays of the network can be updated at each step of the optimization process.

Details

  • With the default value of TrainingUpdateScheduleAutomatic, all arrays are updated at any step of the optimization process.
  • TrainingUpdateSchedule{group1,group2,,groupn} specifies which arrays will be updated for each respective optimization step, this schedule being repeated until the end of training.
  • In TrainingUpdateSchedule{group1,group2,,groupn}, each of the groupi can be of the following forms:
  • "layer"all arrays of a named layer or subnetwork
    nall arrays of the nth layer
    m;;nall arrays of layers m through n
    {layer,"array"}a particular array in a layer or subnetwork
    {part1,part2,}
  • arrays of a nested layer or subnetwork
  • spec1|spec2|any of the specified arrays
    _all arrays in the network
    specsrepeat the same specification for s consecutive steps
  • The hierarchical specification {part1,part2,} used by TrainingUpdateSchedule to refer to a subpart of a net is equivalent to that used by NetExtract and NetReplacePart.
  • Specifications of a subnet (e.g. a nested NetChain or NetGraph) apply to all layers and arrays within that subnet.
  • Any group of parameters not specified in TrainingUpdateSchedule is held constant during training.

Examples

open allclose all

Basic Examples  (1)

Train a NetGANOperator by alternating updates of the discriminator and updates of the generator:

Scope  (2)

Create and initialize a net with three layers:

Train this net by updating alternately the first and third layers, and collect the net arrays after each optimization iteration on a batch:

Show the evolution of the first array value through iterations and check that the arrays are alternately updated:

Create and initialize a NetGraph with named layers:

Train the net by updating a subpart of the NetGraph 10 times more than another:

Check how the arrays are updated:

Train the same NetGraph by updating arrays separately, one by one:

Possible Issues  (1)

When a shared array occurs at several places in the network, only a unique training update schedule will be applied to all the occurrences of the shared array.

Create a network with shared arrays:

Train with a TrainingUpdateSchedule that specifies that only the first layer should be updated and collects the value of the weights of the third layer after each update:

The shared weights are updated at each epoch:

The results are the same as training without any update schedule:

Introduced in 2020
 (12.1)