2. Neural Networks
- Input xi, output yj, weight matrix Wij
- Supervised learning: adjust W to minimize error on dataset
yj=Wijxi(Matrix multiplication)yj=Wijxi+cj(+ Bias)ybj=Wijxbi+cj(+ Batch dimension)ybj=f(Wijxbi+cj)(+ Activation function)
zbj=Wijxbi+cj
ybj=f(zbj)=relu(zbj)
| def dense(x, w, c): |
| z = tf.matmul(w, x) + c |
| # or z = tf.einsum("ij,bi->bj", w, x) + c |
| |
| y = tf.nn.relu(z) |
| return y |
| |
| # x has shape [batch, in_units] |
| w = tf.Variable(tf.random.normal([x.shape[1], 10])) |
| c = tf.Variable(tf.zeros([10])) |
| |
| with tf.GradientTape() as tape: |
| y = dense(x, w, c) |
| dy_dw = tape.gradient(y, w) |
Keras - Layers as objects
| |
| |
| dense_layer = tf.keras.layers.Dense( |
| units=10, activation="relu" |
| ) |
| |
| |
| y = dense_layer(x) |
| |
| |
| v = dense_layer(u) |
Point-cloud Classification
| network = Sequential([ |
| Dense(units=64, activation="relu"), |
| Dense(units=64, activation="relu"), |
| Dense(units=40, activation="softmax") |
| ]) |
| |
| network.compile(loss="categorical_crossentropy", |
| optimizer="Adam") |
| |
| |
| |
| model.fit(train_inputs, train_outputs, |
| epochs=10, validation_split=0.1) |
| |
| predictions = model.predict(test_inputs) |
4. GA in TensorFlow
Geometric product: multiplication table
For e21=e22=1
1 |
e1 |
e2 |
e12 |
e1 |
1 |
e12 |
e2 |
e2 |
-e12 |
1 |
-e1 |
e12 |
-e2 |
e1 |
-1 |
Example: table[e1,e12]=e2
or with blade-indices (Scalar: 0, e1: 1, e2: 2, e12: 3): table[1,3]=2
Can be precomputed
Multivectors: basis blade coefficients as 1-Tensor
Eg.: a = 1+2ex+3ey+4exy→[1,2,3,4]=ai
Dense representation of mult. table as 3-tensor
Cijk
ab=y→aibjCijk=yk
| a = tf.constant([1, 2, 3, 4]) # shape [4] |
| b = tf.constant([3, 9, 7, 5]) # shape [4] |
| # c is precomputed and has shape [4, 4, 4] |
| y = tf.einsum("i,j,ijk->k", a, b, c) # y shape [4] |
Geometric product
Addition
Reminder: Automatic differentiation
y=ab,δy(a,b)δa
| a = tf.constant([1, 2, 3, 4]) # shape [4] |
| b = tf.constant([3, 9, 7, 5]) # shape [4] |
| with tf.GradientTape() as tape: |
| # c is precomputed and has shape [4, 4, 4] |
| y = tf.einsum("i,j,ijk->k", a, b, c) |
| |
| print(tape.gradient(y, a)) # = d/da_i sum(y_j), has shape [4] |
| print(tape.jacobian(y, a)) # = d/da_i y_j, shape [4, 4] |
| ga = tfga.GeometricAlgebra([1, 1, 1]) |
| |
| # Create geometric algebra tf.Tensor for vector blades |
| # (ie. e_0 + e_1 + e_2). |
| # Result: tf.Tensor [0, 1, 1, 1, 0, 0, 0, 0] |
| vector = ga.from_tensor_with_kind( |
| tf.ones(3), kind="vector" |
| ) |
| # 5 + 5 e_01 + 5 e_02 + 5 e_12 |
| # tf.Tensor [5, 0, 0, 0, 5, 5, 5, 0] |
| quaternion = ga.from_tensor_with_kind( |
| tf.fill(dims=4, value=5), |
| kind="even" |
| ) |
| ga.print(ga.geom_prod(vector, quaternion)) |
| |
| |
| |
| ga.print(ga.reversion(quaternion)) |
| |
| |
| |
| ga.print(ga.select_blades(quaternion, "10")) |
| |
| |
| |
| ga.print(ga.keep_blades(quaternion, "10")) |
| |
| |
| ga.print(ga.ext_prod(ga.e01, ga.e2)) |
| ga = tfga.GeometricAlgebra([1, 1, 1]) |
| a = ga.from_tensor(...) |
| b = ga.from_tensor(...) |
| |
| |
| |
| mv_a = ga(a) |
| mv_b = ga(b) |
| |
| |
| print(~mv_a) |
| |
| print(mv_a * mv_b, mv_a | mv_b, mv_a ^ mv_b) |
| |
| print(mv_a.tensor) |
Tensor vs MultiVector API
- Tensors: easy TensorFlow interop
- MultiVector: less verbose, no overhead
5. GA Neural Nets
Standard Dense Layer: ybj=f(Wijxbi+cj)
Wij∈RMN,cj∈RN
| dense_layer = Dense( |
| units=64, activation="relu" |
| ) |
GA Dense Layer: make parameters multivector-valued
Wij∈ClMN(p,q,r),cj∈ClN(p,q,r)
| |
| ga = tfga.GeometricAlgebra([1, 1, 1]) |
| |
| |
| even_indices = ga.get_kind_blade_indices("even") |
| |
| odd_indices = ga.get_kind_blade_indices("odd") |
| |
| ga_dense_layer = tfga.layers.GeometricProductDense( |
| algebra=ga, |
| units=64, activation="relu", |
| kernel_blade_indices=even_indices, |
| bias_blade_indices=odd_indices |
| ) |
Choices
- Activation function f: elementwise or acting on multivector (eg. eae12=cos(a)+e12sin(a))?
- Geometric product Wijxi or Sandwich product Wijxi˜Wij
- Matrix mult Wijxi or elementwise mult Wixi
- Add bias cj?
- Algebra / signature
- Subalgebra for Wij and cj
No activation, sandwich product, elementwise multiplication, no bias, quaternion weights
y(1)i=Wixi˜Wi
Two layers
y(2)i=Uiy(1)i˜Ui=UiWixi˜Wi˜Ui
Composition of transforms
Multi-layer reduces to one layer
Log input, Exp on final output, ReLU activation, geometric product, matrix multiplication, bias, quaternion weights
x(1)i=log(x(0)i)
x(n+1)j=relu(W(n)ijx(n)i+c(n)j)
yj=eW(N−1)ijx(N−1)i+c(N−1)j
log goes from Lie group to algebra
exp goes from Lie algebra to group
6.1 LieNet
Deep Learning on Lie Groups for Skeleton-based Action Recognition
Implementation in TFGA by Hugo Hadfield
| ga = GeometricAlgebra([1, 1, 1]) |
| |
| model = Sequential([ |
| TensorToGeometric(ga, blade_indices=[6, 5, 4, 0]), |
| RotMap(ga, use_bias=False), |
| LogMap(ga), |
| Flatten(), |
| ReLU(), |
| Dense(units=20), |
| Softmax(axis=-1) |
| ]) |
Train model on dataset (Pose-sequence → Action)
| model.compile( |
| optimizer="Adam", |
| loss="sparse_categorical_crossentropy", |
| metrics=["sparse_categorical_accuracy"] |
| ) |
| |
| model.fit( |
| x=inputs_train, y=labels_train, |
| validation_data=(inputs_test, labels_test), |
| epochs=100 |
| ) |
Results
- Relatively easy and quick to implement using TFGA
- Accuracy verified
- But: outperformed by simple Dense ReLU NN (~90% accuracy)
6.3 Lattice QFT
LQED(X)=⟨ℏ(∇ψ(X))γ2γ1γ3˜ψ(X)−eA(X)ψ(X)γ0˜ψ(X)−mψ(X)˜ψ(X)⟩0
Action S=∫XL(X)dX
- Many cells, many parallel calculations, perfect for TensorFlow
- TensorFlow Probability provides MCMC samplers (or variational inference) needed for lattice QFT
mψ(X)˜ψ(X)
| sta = tfga.GeometricAlgebra([1, -1, -1, -1]) |
| |
| def get_mass_term(psi, electron_mass): |
| return (electron_mass * |
| sta.geom_prod(psi, sta.reversion(psi) |
| ) |
eA(X)ψ(X)γ0˜ψ(X)
| def get_interaction_term(psi, a, electron_charge): |
| return sta.geom_prod( |
| electron_charge * a, |
| sta.geom_prod( |
| psi, |
| sta.geom_prod(sta.e0, sta.reversion(psi)) |
| ) |
| ) |
ℏ(∇ψ(X))γ2γ1γ3˜ψ(X)
| def get_momentum_term(psi, spacing, hbar): |
| dt_psi = finite_differences(psi, axis=0, spacing=spacing) |
| dx_psi = finite_differences(psi, axis=1, spacing=spacing) |
| dy_psi = finite_differences(psi, axis=2, spacing=spacing) |
| dz_psi = finite_differences(psi, axis=3, spacing=spacing) |
| d_psi = dt_psi + dx_psi + dy_psi + dz_psi |
| |
| return sta.geom_prod( |
| hbar * d_psi, |
| sta.geom_prod(sta.e213, sta.reversion(psi)) |
| ) |
Sum cells' L to get Action S
| def get_action(psi, a, electron_charge): |
| mass_term = get_mass_term(psi=psi, |
| electron_mass=electron_mass) |
| int_term = get_interaction_term(psi=psi, |
| a=a, electron_charge=electron_charge) |
| mom_term = get_momentum_term(psi=psi, |
| spacing=spacing, hbar=hbar) |
| |
| # Sum terms and get scalar part |
| lagrangians = (mom_term - mass_term - int_term)[..., 0] |
| |
| return tf.reduce_sum(lagrangians) |