6. Parameter learning in Spiking Neural Networks#

Author: Christian Pehle

As we have already seen in Introduction to spiking systems Neuron models come with parameters, such as membrane time constants, which determine their dynamics. While those parameters are often treated as arbitrarily treated constants and therefore hyperparameters in a machine learning context, more recently it has become clear that it can be benefitial to treat them as parameters. In this notebook, we will first learn how to initialise a network of LIF neurons with distinct, but fixed time constants and voltage thresholds and then how to incorporate them into the optimisation. As we will see, this is largely facilitated by the pre-existing ways of treating parameters in PyTorch.

import torch
import norse
import matplotlib.pyplot as plt

6.1. Defining a Network of LIF Neurons with varying membrane time-constants#

A population of recurrently connected LIF neurons can be instantiated in Norse as follows:

from norse.torch.module import LIFRecurrentCell

m = LIFRecurrentCell(input_size=200, hidden_size=100)
m
LIFRecurrentCell(input_size=200, hidden_size=100, p=LIFParameters(tau_syn_inv=tensor(200.), tau_mem_inv=tensor(100.), v_leak=tensor(0.), v_th=tensor(1.), v_reset=tensor(0.), method='super', alpha=tensor(100.)), autapses=False, dt=0.001)

As you can see the LIFParameters are initialised to some default values. It is easy enough to instead sample these values from a given distribution. While we will use a random normal distribution here, a more appropriate choice would be for example a lognormal distribution, as this would guarantee that the inverse membrane time constant remains positive, which is an essential requirement. Here we just choose the standard deviation in such a way, that a negative value would be very unlikely.

import numpy as np

counts, bins = np.histogram(norse.torch.functional.lif.LIFParameters().tau_mem_inv + 20*torch.randn(100))
plt.hist(bins[:-1], bins, weights=counts, histtype='step')
plt.xlabel('$\\tau_{m}^{-1}$ [ms]')
Text(0.5, 0, '$\\tau_{m}^{-1}$ [ms]')
../_images/7d41cab57a466486df347536b708ec615af0e5f077a925881d9e0eab9106e768.png

The parameters of a PyTorch module can be accesses by invoking the parameters method. In the case of the LIFRecurrentCell those are the recurrent and input weight matrices. Note how both tensors have requires_grad=True and that the LIFParameters do not appear. This is because none of the tensors entering the LIFParameters have been registered as a PyTorch Parameter.

list(m.parameters())
[Parameter containing:
 tensor([[-5.8637e-03, -3.6807e-02,  3.9165e-01,  ...,  2.6109e-01,
          -4.6717e-02,  9.1689e-02],
         [-1.1323e-01,  3.6441e-04,  9.3518e-02,  ..., -8.9191e-02,
           1.2103e-01, -1.5315e-02],
         [ 4.7549e-03, -7.7586e-02, -9.1182e-02,  ..., -5.6045e-02,
          -1.8624e-01, -1.9790e-01],
         ...,
         [-1.0738e-03, -9.7985e-02, -7.8792e-02,  ..., -6.7082e-03,
          -2.6878e-02,  1.5208e-01],
         [ 5.4254e-02, -1.3804e-01,  6.2338e-02,  ..., -6.6062e-02,
          -4.3294e-02, -4.9237e-02],
         [-5.8116e-02, -2.4526e-01, -6.4269e-02,  ...,  4.8523e-02,
          -1.2775e-02,  4.6012e-02]], requires_grad=True),
 Parameter containing:
 tensor([[ 0.0000,  0.1463, -0.0487,  ..., -0.1102, -0.2006,  0.0317],
         [ 0.3007,  0.0000,  0.0090,  ..., -0.1080,  0.1953, -0.0950],
         [ 0.0480,  0.0395,  0.0000,  ...,  0.1212,  0.1496, -0.1600],
         ...,
         [-0.2192,  0.1611,  0.1901,  ...,  0.0000, -0.0111,  0.2407],
         [-0.2422, -0.1298, -0.0649,  ..., -0.2418,  0.0000,  0.0875],
         [ 0.1934,  0.0605, -0.0934,  ..., -0.0666, -0.2258,  0.0000]],
        requires_grad=True)]

The easiest way to rectify this situation is to define an additional torch.nn.Module and explicitely register the inverse membrane time constant and threshold as a parameter.

class ParametrizedLIFRecurrentCell(torch.nn.Module):
    def __init__(self, input_size, hidden_size):
        super(ParametrizedLIFRecurrentCell, self).__init__()
        self.tau_mem_inv = torch.nn.Parameter(norse.torch.functional.lif.LIFParameters().tau_mem_inv + 20*torch.randn(hidden_size))
        self.v_th = torch.nn.Parameter(0.5 + 0.1 * torch.randn(hidden_size))
        self.cell = norse.torch.module.LIFRecurrentCell(input_size=input_size, hidden_size=hidden_size, 
                                                        p = norse.torch.functional.lif.LIFParameters(
                                                            tau_mem_inv = self.tau_mem_inv,
                                                            v_th = self.v_th,
                                                            alpha = 100,
                                                        )
                                                        
        )

    def forward(self, x, s = None):
        return self.cell(x, s)
m = ParametrizedLIFRecurrentCell(200, 100)
m
ParametrizedLIFRecurrentCell(
  (cell): LIFRecurrentCell(
    input_size=200, hidden_size=100, p=LIFParameters(tau_syn_inv=tensor(200.), tau_mem_inv=Parameter containing:
    tensor([105.8002, 118.7476, 121.0143, 107.7886,  95.9332,  97.9559,  82.1780,
             83.9186,  76.6450, 105.7318, 114.1652,  57.4965,  71.4951,  83.8756,
             79.2064,  63.3554,  99.6375, 135.0398, 130.4913, 144.0811,  73.5309,
             88.7118,  85.7732, 102.1938, 105.8278,  93.2198, 105.9444,  89.0313,
             70.8726,  70.5813, 116.8266,  73.4906,  99.6221,  93.2961,  93.6535,
            100.9834, 114.6153, 133.8099,  81.9619, 134.1033,  86.2323,  81.2740,
             93.7436, 113.0415, 113.3560,  94.1462, 105.8760,  66.6123,  96.8165,
             90.9954,  70.0471, 120.2753,  91.1188, 111.8848,  77.3143, 111.0734,
             94.2069,  82.3439,  75.7089,  82.6979,  67.8745,  94.8562, 125.4359,
             90.9662, 109.8191,  99.0547, 127.8489,  86.2925, 110.9234,  92.9017,
             91.8363,  91.0217,  93.4295,  96.1113, 101.6119,  73.8234,  70.4884,
            109.9867, 120.7329,  68.9098,  69.2704,  93.0631, 107.8932,  79.4659,
             95.5474,  98.4740, 111.3480, 132.3869,  72.0627,  98.2057,  79.3738,
             86.8217, 109.3848,  67.9757, 122.7573,  98.4890,  90.8490, 118.1372,
             96.0169,  89.3513], requires_grad=True), v_leak=tensor(0.), v_th=Parameter containing:
    tensor([0.4245, 0.5003, 0.4772, 0.6213, 0.4540, 0.5343, 0.4711, 0.5739, 0.6938,
            0.2578, 0.4255, 0.4642, 0.4776, 0.5375, 0.7099, 0.3907, 0.3588, 0.4489,
            0.4324, 0.4364, 0.4488, 0.5066, 0.4606, 0.5203, 0.6279, 0.5174, 0.5062,
            0.6447, 0.6344, 0.3952, 0.3214, 0.4598, 0.4387, 0.4816, 0.7487, 0.4587,
            0.5737, 0.4371, 0.4841, 0.4286, 0.4360, 0.3613, 0.3182, 0.7134, 0.7182,
            0.5473, 0.6726, 0.6715, 0.3215, 0.4130, 0.5395, 0.4576, 0.5902, 0.3695,
            0.4416, 0.3895, 0.5763, 0.5153, 0.7317, 0.4261, 0.5278, 0.3722, 0.5055,
            0.4400, 0.4364, 0.4307, 0.4361, 0.3520, 0.3665, 0.3329, 0.5601, 0.5924,
            0.4776, 0.5682, 0.5736, 0.4400, 0.3915, 0.4977, 0.3708, 0.4486, 0.4725,
            0.6915, 0.4902, 0.5364, 0.3285, 0.6454, 0.5668, 0.4228, 0.4203, 0.5578,
            0.4720, 0.5132, 0.3799, 0.4843, 0.4477, 0.5774, 0.4645, 0.5535, 0.5356,
            0.5089], requires_grad=True), v_reset=tensor(0.), method='super', alpha=tensor(100)), autapses=False, dt=0.001
  )
)

The inverse membrane time constant and the threshold value now also appears as parameters:

list(m.parameters())
[Parameter containing:
 tensor([105.8002, 118.7476, 121.0143, 107.7886,  95.9332,  97.9559,  82.1780,
          83.9186,  76.6450, 105.7318, 114.1652,  57.4965,  71.4951,  83.8756,
          79.2064,  63.3554,  99.6375, 135.0398, 130.4913, 144.0811,  73.5309,
          88.7118,  85.7732, 102.1938, 105.8278,  93.2198, 105.9444,  89.0313,
          70.8726,  70.5813, 116.8266,  73.4906,  99.6221,  93.2961,  93.6535,
         100.9834, 114.6153, 133.8099,  81.9619, 134.1033,  86.2323,  81.2740,
          93.7436, 113.0415, 113.3560,  94.1462, 105.8760,  66.6123,  96.8165,
          90.9954,  70.0471, 120.2753,  91.1188, 111.8848,  77.3143, 111.0734,
          94.2069,  82.3439,  75.7089,  82.6979,  67.8745,  94.8562, 125.4359,
          90.9662, 109.8191,  99.0547, 127.8489,  86.2925, 110.9234,  92.9017,
          91.8363,  91.0217,  93.4295,  96.1113, 101.6119,  73.8234,  70.4884,
         109.9867, 120.7329,  68.9098,  69.2704,  93.0631, 107.8932,  79.4659,
          95.5474,  98.4740, 111.3480, 132.3869,  72.0627,  98.2057,  79.3738,
          86.8217, 109.3848,  67.9757, 122.7573,  98.4890,  90.8490, 118.1372,
          96.0169,  89.3513], requires_grad=True),
 Parameter containing:
 tensor([0.4245, 0.5003, 0.4772, 0.6213, 0.4540, 0.5343, 0.4711, 0.5739, 0.6938,
         0.2578, 0.4255, 0.4642, 0.4776, 0.5375, 0.7099, 0.3907, 0.3588, 0.4489,
         0.4324, 0.4364, 0.4488, 0.5066, 0.4606, 0.5203, 0.6279, 0.5174, 0.5062,
         0.6447, 0.6344, 0.3952, 0.3214, 0.4598, 0.4387, 0.4816, 0.7487, 0.4587,
         0.5737, 0.4371, 0.4841, 0.4286, 0.4360, 0.3613, 0.3182, 0.7134, 0.7182,
         0.5473, 0.6726, 0.6715, 0.3215, 0.4130, 0.5395, 0.4576, 0.5902, 0.3695,
         0.4416, 0.3895, 0.5763, 0.5153, 0.7317, 0.4261, 0.5278, 0.3722, 0.5055,
         0.4400, 0.4364, 0.4307, 0.4361, 0.3520, 0.3665, 0.3329, 0.5601, 0.5924,
         0.4776, 0.5682, 0.5736, 0.4400, 0.3915, 0.4977, 0.3708, 0.4486, 0.4725,
         0.6915, 0.4902, 0.5364, 0.3285, 0.6454, 0.5668, 0.4228, 0.4203, 0.5578,
         0.4720, 0.5132, 0.3799, 0.4843, 0.4477, 0.5774, 0.4645, 0.5535, 0.5356,
         0.5089], requires_grad=True),
 Parameter containing:
 tensor([[-0.0422, -0.0364, -0.3005,  ...,  0.0366,  0.2412,  0.0492],
         [ 0.0408,  0.2024,  0.0127,  ...,  0.0150,  0.1394, -0.0830],
         [-0.1600, -0.1036,  0.2151,  ...,  0.0462, -0.2395,  0.0550],
         ...,
         [-0.0165, -0.0117, -0.1041,  ...,  0.0317,  0.2117,  0.1845],
         [ 0.1241, -0.0686,  0.0417,  ...,  0.1071,  0.0264, -0.0164],
         [-0.1045, -0.2212, -0.2499,  ..., -0.0813,  0.1965, -0.2827]],
        requires_grad=True),
 Parameter containing:
 tensor([[ 0.0000, -0.0072, -0.1808,  ...,  0.1227,  0.0914,  0.1887],
         [-0.1343,  0.0000,  0.1680,  ...,  0.0641, -0.1959, -0.3788],
         [-0.1995,  0.1303,  0.0000,  ..., -0.1215, -0.1959,  0.0877],
         ...,
         [ 0.0973, -0.0732, -0.2094,  ...,  0.0000,  0.1461, -0.0790],
         [-0.1707, -0.1247,  0.0877,  ..., -0.1199,  0.0000, -0.0312],
         [-0.2795,  0.0166, -0.0023,  ..., -0.1528,  0.0421,  0.0000]],
        requires_grad=True)]

6.2. Toy Example: Training a small recurrent SNN on MNIST#

In order to demonstrate training on an example task, we turn to MNIST. The training and testing code here is adopted from the training notebook on MNIST and serves as an illustration. We do not necessarily expect there to be a great benefit in optimising the time-constants and threshold in this example.

6.2.1. Dataset Loading#

import torchvision

BATCH_SIZE = 256

transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize((0.1307,), (0.3081,)),
    ]
)

train_data = torchvision.datasets.MNIST(
    root=".",
    train=True,
    download=True,
    transform=transform,
)

train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=BATCH_SIZE,
    shuffle=True
)

test_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST(
        root=".",
        train=False,
        transform=transform,
    ),
    batch_size=BATCH_SIZE
)
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./MNIST/raw/train-images-idx3-ubyte.gz
  0%|          | 0/9912422 [00:00<?, ?it/s]
100%|██████████| 9912422/9912422 [00:00<00:00, 176659321.01it/s]
Extracting ./MNIST/raw/train-images-idx3-ubyte.gz to ./MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./MNIST/raw/train-labels-idx1-ubyte.gz
  0%|          | 0/28881 [00:00<?, ?it/s]
100%|██████████| 28881/28881 [00:00<00:00, 66704677.22it/s]

Extracting ./MNIST/raw/train-labels-idx1-ubyte.gz to ./MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./MNIST/raw/t10k-images-idx3-ubyte.gz
  0%|          | 0/1648877 [00:00<?, ?it/s]
100%|██████████| 1648877/1648877 [00:00<00:00, 119579690.44it/s]
Extracting ./MNIST/raw/t10k-images-idx3-ubyte.gz to ./MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./MNIST/raw/t10k-labels-idx1-ubyte.gz

  0%|          | 0/4542 [00:00<?, ?it/s]
100%|██████████| 4542/4542 [00:00<00:00, 15131476.38it/s]

Extracting ./MNIST/raw/t10k-labels-idx1-ubyte.gz to ./MNIST/raw

6.2.2. Model Defitinition#

For simplicity we are considering a simple network with a single hidden layer of recurrently connected neurons.

from norse.torch import LICell

class SNN(torch.nn.Module):
    def __init__(self, 
                 input_features,
                 hidden_features, 
                 output_features,
                 recurrent_cell
                ):
        super(SNN, self).__init__()
        self.cell = recurrent_cell
        self.fc_out = torch.nn.Linear(hidden_features, output_features, bias=False)
        self.out = LICell()
        self.input_features = input_features
                             
    def forward(self, x):
        seq_length, batch_size, _, _, _ = x.shape
        s1 = so = None
        voltages = []


        for ts in range(seq_length):
            z = x[ts, :, :, :].view(-1, self.input_features)
            z, s1 = self.cell(z, s1)
            z = self.fc_out(z)
            vo, so = self.out(z, so)
            voltages += [vo]
        
        return torch.stack(voltages)
class Model(torch.nn.Module):
    def __init__(self, encoder, snn, decoder):
        super(Model, self).__init__()
        self.encoder = encoder
        self.snn = snn
        self.decoder = decoder

    def forward(self, x):
        x = self.encoder(x)
        x = self.snn(x)
        log_p_y = self.decoder(x)
        return log_p_y

6.2.3. Training and Test loop#

Both the training and test loop are independent of the model or dataset that we are using, in practice you should however probably use a more sophisticated version as they don’t take care of parameter checkpoints or other concerns.

from tqdm.notebook import tqdm, trange

def train(model, device, train_loader, optimizer, epoch, max_epochs):
    model.train()
    losses = []

    for (data, target) in tqdm(train_loader, leave=False):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = torch.nn.functional.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        losses.append(loss.item())

    mean_loss = np.mean(losses)
    return losses, mean_loss
def test(model, device, test_loader, epoch):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += torch.nn.functional.nll_loss(
                output, target, reduction="sum"
            ).item()  # sum up batch loss
            pred = output.argmax(
                dim=1, keepdim=True
            )  # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    accuracy = 100.0 * correct / len(test_loader.dataset)

    return test_loss, accuracy
from norse.torch import ConstantCurrentLIFEncoder

def decode(x):
    x, _ = torch.max(x, 0)
    log_p_y = torch.nn.functional.log_softmax(x, dim=1)
    return log_p_y


T = 32
LR = 0.002
INPUT_FEATURES = 28*28
HIDDEN_FEATURES = 100
OUTPUT_FEATURES = 10
EPOCHS = 5

if torch.cuda.is_available():
    DEVICE = torch.device("cuda")
else:
    DEVICE = torch.device("cpu")
def run_training(model, optimizer, epochs = EPOCHS):
    training_losses = []
    mean_losses = []
    test_losses = []
    accuracies = []

    torch.autograd.set_detect_anomaly(True)

    for epoch in trange(epochs):
        training_loss, mean_loss = train(model, DEVICE, train_loader, optimizer, epoch, max_epochs=EPOCHS)
        test_loss, accuracy = test(model, DEVICE, test_loader, epoch)
        training_losses += training_loss
        mean_losses.append(mean_loss)
        test_losses.append(test_loss)
        accuracies.append(accuracy)

    print(f"final accuracy: {accuracies[-1]}")
    return model

6.2.4. Model Evaluation#

With all of this boilerplate out of the way we can define our final model and run training for a number of epochs (10 by default). If you are impatient you can decrease that number to 1-2. With the default values, the result will be a test accuracy of >95%, that is nothing to write home about.

model = Model(
    encoder=ConstantCurrentLIFEncoder(
      seq_length=T,
    ),
    snn=SNN(
      input_features=INPUT_FEATURES,
      hidden_features=HIDDEN_FEATURES,
      output_features=OUTPUT_FEATURES,
      recurrent_cell=ParametrizedLIFRecurrentCell(
            input_size=28*28, 
            hidden_size=100
      )
    ),
    decoder=decode
).to(DEVICE)

optimizer = torch.optim.Adam(model.parameters(), lr=LR)
tau_mem_inv_before = model.snn.cell.cell.p.tau_mem_inv.cpu().detach().numpy()
v_th_before = model.snn.cell.cell.p.v_th.cpu().detach().numpy()
model_after = run_training(model, optimizer, epochs=10)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[17], line 1
----> 1 model_after = run_training(model, optimizer, epochs=10)

Cell In[14], line 9, in run_training(model, optimizer, epochs)
      5 accuracies = []
      7 torch.autograd.set_detect_anomaly(True)
----> 9 for epoch in trange(epochs):
     10     training_loss, mean_loss = train(model, DEVICE, train_loader, optimizer, epoch, max_epochs=EPOCHS)
     11     test_loss, accuracy = test(model, DEVICE, test_loader, epoch)

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tqdm/notebook.py:312, in tnrange(*args, **kwargs)
    310 def tnrange(*args, **kwargs):
    311     """Shortcut for `tqdm.notebook.tqdm(range(*args), **kwargs)`."""
--> 312     return tqdm_notebook(range(*args), **kwargs)

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tqdm/notebook.py:234, in tqdm_notebook.__init__(self, *args, **kwargs)
    232 unit_scale = 1 if self.unit_scale is True else self.unit_scale or 1
    233 total = self.total * unit_scale if self.total else self.total
--> 234 self.container = self.status_printer(self.fp, total, self.desc, self.ncols)
    235 self.container.pbar = proxy(self)
    236 self.displayed = False

File /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/tqdm/notebook.py:108, in tqdm_notebook.status_printer(_, total, desc, ncols)
     99 # Fallback to text bar if there's no total
    100 # DEPRECATED: replaced with an 'info' style bar
    101 # if not total:
   (...)
    105 
    106 # Prepare IPython progress bar
    107 if IProgress is None:  # #187 #451 #558 #872
--> 108     raise ImportError(WARN_NOIPYW)
    109 if total:
    110     pbar = IProgress(min=0, max=total)

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
tau_mem_inv_after = model_after.snn.cell.cell.p.tau_mem_inv.cpu().detach().numpy()
v_th_after = model_after.snn.cell.cell.p.v_th.cpu().detach().numpy()

We can plot the distribution of inverse membrane timeconstants before and after the training:

counts, bins = np.histogram(tau_mem_inv_before)
fig, ax = plt.subplots(figsize=(9,5))
ax.hist(bins[:-1], bins, weights=counts, histtype='step', label='before')
counts, bins = np.histogram(tau_mem_inv_after)
ax.hist(bins[:-1], bins, weights=counts, histtype='step', label='after')
ax.set_xlabel('$\\tau_{m}^{-1}$ [ms]')
ax.legend()
<matplotlib.legend.Legend at 0x7fe665b8efd0>
../_images/1366a1eaf90906598ef36b52f1126e45718b99b438a02d870243b5620aa7a181.png

Similarly we can plot the distribution of membrane threshold voltages before and after optimisation:

counts, bins = np.histogram(v_th_before)
fig, ax = plt.subplots(figsize=(9,5))
ax.hist(bins[:-1], bins, weights=counts, histtype='step', label='before')
counts, bins = np.histogram(v_th_after)
ax.hist(bins[:-1], bins, weights=counts, histtype='step', label='after')
ax.set_xlabel('$v_{th}$ [a.u.]')
ax.legend()
<matplotlib.legend.Legend at 0x7fe665b8a8e0>
../_images/4665e6830daadb4361f43c003c9db7a729ddf68b6571ef31a3f148179e0252fc.png

6.3. Conclusions#

This concludes this tutorial, as we have seen incorporating neuron parameters into the optimisation in Norse mainly requires knowledge of the PyTorch internals. At the moment that optimisation is also only supported for the surrogate gradient implementation and not while using the discretised adjoint implementation. This is not a fundamental limitation though and support for neuron parameter gradients could be added.

Several publications have recently explored optimising neuron parameters to both demonstrate that the resulting time constant distributions more closely resembled ones measured in biological cells and advantages in ML applications. As always the challenge to practically demonstrating the advantage of incorporating neuron parameters into the optimisation is doing careful and controlled experiments and choosing the right tasks. While this notebook demonstrates the general technique it does take some shortcuts:

  • Optimisation works best if the scale of the values to be optimised is in an appropriate range. This is not the case here for the inverse time constant, which has a value of > 100. A more sophisticated approach would take this into account by for example generating this parameter from an appropriately scaled initial value.

  • Similarly we have chosen a rather naive initialisation of both the threshold and inverse membrane time constant. More careful initialisation would for example involve sampling from a distribution with guaranteed positive samples.

  • Finally we haven’t demonstrated any benefit of jointly optimising neuron parameters and synaptic weights.