Missing trending activations like GEGLU & Mish #3483
-
|
GEGLU is a top activation right now but I had to implement it myself in a recent project. Is there a specific reason for omitting it? I can submit a PR for those? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 5 replies
-
|
Flax imports most of its activation functions from Jax. You could try submitting an issue/PR to Jax. |
Beta Was this translation helpful? Give feedback.
-
|
I noticed PReLu was crudely implemented in Flax’s activation, opposed to importing from Jax. This seemed premeditated, given that it has learnable parameters, just like GEGLU. Any idea on why? |
Beta Was this translation helpful? Give feedback.
-
|
I think PReLU was implemented in Flax precisely because it has learnable parameters (and there is also no Jax implementation of it). In which case if GEGLU has learnable parameters, maybe we can add it to Flax instead. |
Beta Was this translation helpful? Give feedback.
-
|
Great, I shall submit a PR. |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks! Also I found the original discussion for PReLU for context. |
Beta Was this translation helpful? Give feedback.
-
|
And for Mish: |
Beta Was this translation helpful? Give feedback.
I think PReLU was implemented in Flax precisely because it has learnable parameters (and there is also no Jax implementation of it). In which case if GEGLU has learnable parameters, maybe we can add it to Flax instead.