This notebook quickly demonstrates how you can create biological realistic neural network models that can be used for deep learning.
To use Norse effectively, you need to know two things
How to build neuron models
How to apply neuron models
You can execute the code below by hitting above and pressing Live Code.
2.1. Building Models in Norse¶
Norse is designed to be as simple as possible to use; our neuron models integrate seamlessly with PyTorch - with a single caveat: state.
Neurons need to remember their history, similar to recurrent networks in PyTorch. Therefore, we cannot wrap models with the regular
Further details on state handling in Working with Norse
Our solution is to provide our own
norse.torch.SequentialState module, which makes it trivial to construct spiking networks:
import torch import norse model = norse.torch.SequentialState( norse.torch.LIF(), torch.nn.Linear(2, 1) )
Notice how similar this looks to PyTorch’s
Sequential. The only difference is that we have replaced one activation function with a leaky integrate-and-fire (LIF) model, which is a widely used simple biological neuron model.
2.2. Applying neuron models¶
If you are familiar with recurrent networks in PyTorch, you will know that they return not one, but two values: the actual model
output and the model
state (see e. g. PyTorch’s RNN module).
output, state = model(torch.ones(4, 2))
Because of their recurrent nature, neurons exist in time. A neuron needs time to emit action potentials/spikes, so we will always assume that the outer dimension of the input tensor is time.
Below, we show how you can take a tensor with 4 timesteps (the outer dimension) and apply it to 2 neurons (inner dimension). The linear layer should, then, reduce this to a single neuron simulated over 4 timesteps.
output, state = model(torch.ones(4, 2)) output
tensor([[0.0330], [0.0330], [0.0330], [0.0330]], grad_fn=<AddmmBackward0>)
Perfect! The output looks as we would expect: it’s a
(4,1) tensor. 4 timesteps, 1 output neuron.
2.2.1. Bonus: Applying neuron models with predefined state¶
Before we conclude, note that you can inject a different state if you wish to influence the neuron starting state. Below, we simply use the state we stored above in the
state variable. The page on Working with Norse will demonstrate how to configure this in depth.
output, next_state = model(torch.ones(4, 2), state=state)
This is, honestly, everything you need to get started. Provided you have knowledge with building and training models in PyTorch, you can keep working as you have done so far. If you are new to PyTorch, we highly recommend their suite of tutorials: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html