martin/state_interpretation
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add RWKV, H3, Hyena

Browse Source
This commit is contained in:
Korbinian Pöppel
2023-08-05 17:33:32 +02:00
parent a71030547c
commit 7b15a413d4
22 changed files with 1794 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
config/models/h3.yaml Normal file
View File

@@ -0,0 +1,7 @@
_name_: h3
d_state: 64
head_dim: 1
mode: diag
measure: diag-lin
# lr: ${eval:"min(0.001, ${optimizer.lr})"}
lr: 0.001