transformer_config

Classes

TransformerConfig

Configuration for a Transformer model.

Module Contents

class transformer_config.TransformerConfig(/, **data)

Bases: emmi.schemas.modules.blocks.TransformerBlockConfig

Configuration for a Transformer model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (Any)

depth: int

Number of transformer blocks in the model.

mlp_expansion_factor: int = 4

Expansion factor for the MLP hidden dimension relative to the hidden dimension. If ‘mlp_hidden_dim’ is not set, this factor is used to compute it as hidden_dim * mlp_expansion_factor.

set_mlp_hidden_dim()