Coding ML with Julia
Last Updated: 01 Nov 2022When I took my machine learning classes at Michigan, we learned how to code our architectures using Julia. It’s a very performant language (compared to something like Python) and was built with math and data science in mind.
One of Julia’s machine learning libraries is called Flux. This post is a short reference for using the Flux package to setup and train deep learning networks. The offical website can be found here.
Basics
An efficient way to use one-hot encoding (better than regular arrays). This produces a one-hot vector of length 5, with a 1 at the second position.
onehot(2, 1:5)
Softmax normalizes a vector such that the sum of all entries is exactly equal to 1 (as it should be for probability distributions). Here, xs[2] is a vector representing the 2nd training example.
softmax(model(xs[2]))
And now setting up a neural network:
model = Dense(2,3,sigma)
L(x,y) = Flux.mse(model(x),y)
opt = ADAM()
Flux.train!(L, zip(xs, ys), opt)
Where:
xsis the feature vectorysis the label vectorL(x,y)is the loss function
Deep Learning
We can build deep learning networks (i.e. networks with more than 1 layer) by using the Chain function.
layer1 = Dense(2, 3, sigma)
layer2 = Dense(3, 2, identity)
layer3 = softmax
model = Chain(layer1, layer2, layer3)
L(x,y) = Flux.crossentropy(model(x), y)
opt = SGD(params(model))
Batching
Batching is a function that merges many data points into a single matrix. This is more efficient to train because matrix-matrix multiplication has been heavily optimized. To create mini-batches, we can call the batch function on a subset of our data.
databatch = (Flux.batch(xs), Flux.batch(ys))
Flux.train!(L, Iterators.repeated(databatch, 1000), opt)
Second line: Flux takes the model, and trains it on the dataset in databatch, and repeats 1000 times.
Callbacks
A function that we’d like Flux to call every so often. This is useful for capturing the loss or learning progress while we’re training a model. We can wrap our callback function with the throttle function to set how often (in seconds).
Flux.throttle(callback, 1)