Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
Emmi AI Core documentation
Logo

Tutorials

  • Prerequisites
  • Getting Started: Install & Verify
  • Training First Model (with Configs)
  • Training First Model (with Code)
  • Running First Inference

How-to Guides

  • Working with the CLI
  • How to Load a Custom Dataset
  • How to Use Private Data Source
  • How to Write Your Own Data Preprocessors
  • How to Write Your Own Data Collators
  • How to Configure All Training Options
  • How to Use Custom Model
  • How to Write Custom Loss or Metrics
  • How to Run Inference via the CLI
  • How to Run Inference via the Code

Explanation

  • Overview
    • Introduction to Emmi Framework
    • Key Concepts
    • Design Principles and Limitation
  • Disk Caching

Reference

  • Emmi Data Management
    • emmi_data_management
      • emmi_data_management.checkpoint
        • emmi_data_management.checkpoint.iohash
        • emmi_data_management.checkpoint.providers
          • emmi_data_management.checkpoint.providers.file_provider
          • emmi_data_management.checkpoint.providers.huggingface_provider
        • emmi_data_management.checkpoint.resolver
        • emmi_data_management.checkpoint.types
      • emmi_data_management.cli
      • emmi_data_management.cli_aws
      • emmi_data_management.cli_checkpoint
      • emmi_data_management.cli_huggingface
      • emmi_data_management.cli_utils
      • emmi_data_management.cli_verification
      • emmi_data_management.credentials
      • emmi_data_management.diskcache
        • emmi_data_management.diskcache.cache_benchmark
        • emmi_data_management.diskcache.lru_cache
      • emmi_data_management.interfaces
        • emmi_data_management.interfaces.azure
        • emmi_data_management.interfaces.gcp
        • emmi_data_management.interfaces.huggingface
        • emmi_data_management.interfaces.s3
        • emmi_data_management.interfaces.s3_uploader
      • emmi_data_management.logging_config
      • emmi_data_management.logging_progress
      • emmi_data_management.providers
      • emmi_data_management.verification
  • Emmi Inference Module
    • emmi_inference
      • emmi_inference.api
      • emmi_inference.cli
      • emmi_inference.cli_utils
      • emmi_inference.collator_registry
      • emmi_inference.common
      • emmi_inference.dataset_registry
      • emmi_inference.datasets
        • emmi_inference.datasets.shapenet
      • emmi_inference.device
      • emmi_inference.examples
        • emmi_inference.examples.run_abupt
        • emmi_inference.examples.run_transolver
        • emmi_inference.examples.run_upt
      • emmi_inference.logging_config
      • emmi_inference.models
        • emmi_inference.models.abupt
        • emmi_inference.models.base_model
        • emmi_inference.models.loader
        • emmi_inference.models.modules
          • emmi_inference.models.modules.attention
            • emmi_inference.models.modules.attention.anchor_attention
            • emmi_inference.models.modules.attention.dot_product_attention
            • emmi_inference.models.modules.attention.perceiver_attention
            • emmi_inference.models.modules.attention.sharedweights_crossattn_attention
            • emmi_inference.models.modules.attention.sharedweights_splitattn_attention
          • emmi_inference.models.modules.blocks
            • emmi_inference.models.modules.blocks.perceiver_block
            • emmi_inference.models.modules.blocks.transformer_block
          • emmi_inference.models.modules.continuous_sincos_embed
          • emmi_inference.models.modules.mlp
          • emmi_inference.models.modules.rope
          • emmi_inference.models.modules.rope_frequency
          • emmi_inference.models.modules.supernode_pooling_posonly
        • emmi_inference.models.pipelines
          • emmi_inference.models.pipelines.abupt
          • emmi_inference.models.pipelines.anchor_point_sampling_precollator
          • emmi_inference.models.pipelines.concat_tensor_precollator
          • emmi_inference.models.pipelines.default_tensor_precollator
          • emmi_inference.models.pipelines.field_decoder
          • emmi_inference.models.pipelines.mesh_to_grid_edges_precollator
          • emmi_inference.models.pipelines.query_to_grid_edges_precollator
          • emmi_inference.models.pipelines.shapenet_multistage
          • emmi_inference.models.pipelines.sparse_tensor_offset
          • emmi_inference.models.pipelines.surface_mask_precollator
        • emmi_inference.models.registry
        • emmi_inference.models.transolver
        • emmi_inference.models.upt
      • emmi_inference.preprocess
      • emmi_inference.runner
      • emmi_inference.s3_spooling_uploader
      • emmi_inference.writer
  • High level API (emmi)
    • emmi
      • emmi.callbacks
        • emmi.callbacks.supernode_pooling_coverage
      • emmi.datasets
        • emmi.datasets.ahmedml
        • emmi.datasets.drivaerml
      • emmi.functional
        • emmi.functional.init
        • emmi.functional.logscale
        • emmi.functional.modulation
        • emmi.functional.rope
      • emmi.models
      • emmi.modules
        • emmi.modules.activations
        • emmi.modules.attention
          • emmi.modules.attention.anchor_attention
            • emmi.modules.attention.anchor_attention.cross_anchor_attention
            • emmi.modules.attention.anchor_attention.joint_anchor_attention
            • emmi.modules.attention.anchor_attention.mixed_attention
            • emmi.modules.attention.anchor_attention.multi_branch_anchor_attention
            • emmi.modules.attention.anchor_attention.self_anchor_attention
          • emmi.modules.attention.dot_product_attention
          • emmi.modules.attention.irregular_nat_attention
          • emmi.modules.attention.perceiver_attention
          • emmi.modules.attention.transolver_attention
          • emmi.modules.attention.transolver_plusplus_attention
        • emmi.modules.blocks
          • emmi.modules.blocks.irregular_nat_block
          • emmi.modules.blocks.perceiver_block
          • emmi.modules.blocks.perceiver_transformer_blockpair
          • emmi.modules.blocks.transformer_block
        • emmi.modules.decoder
          • emmi.modules.decoder.deep_perceiver_decoder
        • emmi.modules.encoder
          • emmi.modules.encoder.mlp_encoder
          • emmi.modules.encoder.supernode_pooling
          • emmi.modules.encoder.supernode_pooling_posonly
        • emmi.modules.layers
          • emmi.modules.layers.continuous_sincos_embed
          • emmi.modules.layers.drop_path
          • emmi.modules.layers.layer_scale
          • emmi.modules.layers.linear_projection
          • emmi.modules.layers.rope_frequency
          • emmi.modules.layers.scalar_conditioner
          • emmi.modules.layers.transformer_batchnorm
        • emmi.modules.mlp
          • emmi.modules.mlp.mlp
          • emmi.modules.mlp.upactdown_mlp
        • emmi.modules.preprocessors
          • emmi.modules.preprocessors.normalizers
            • emmi.modules.preprocessors.normalizers.mean_std_normalization
            • emmi.modules.preprocessors.normalizers.position_normalizer
            • emmi.modules.preprocessors.normalizers.shift_and_scale_normalizer
            • emmi.modules.preprocessors.normalizers.types
          • emmi.modules.preprocessors.numpy_to_torch_tensor
      • emmi.optim
        • emmi.optim.lion
      • emmi.pipeline
        • emmi.pipeline.batch_processors
          • emmi.pipeline.batch_processors.moment_normalization
          • emmi.pipeline.batch_processors.position_normalization
          • emmi.pipeline.batch_processors.rename_keys
        • emmi.pipeline.collators
          • emmi.pipeline.collators.concat_sparse_tensor
          • emmi.pipeline.collators.default
          • emmi.pipeline.collators.field_decoder
        • emmi.pipeline.sample_processors
          • emmi.pipeline.sample_processors.drop_outliers
          • emmi.pipeline.sample_processors.duplicate_keys
          • emmi.pipeline.sample_processors.moment_normalization
          • emmi.pipeline.sample_processors.point_sampling
          • emmi.pipeline.sample_processors.position_normalization
          • emmi.pipeline.sample_processors.rename_keys
          • emmi.pipeline.sample_processors.replace_key
          • emmi.pipeline.sample_processors.samplewise_normalization
          • emmi.pipeline.sample_processors.supernode_sampling
      • emmi.preprocessing
        • emmi.preprocessing.calculate_statistics_all
        • emmi.preprocessing.calculate_statistics_train
        • emmi.preprocessing.process_drivaer_and_ahmed_volumetric
        • emmi.preprocessing.shapenet_car
          • emmi.preprocessing.shapenet_car.preprocess_shapenet_car
      • emmi.schemas
        • emmi.schemas.data
          • emmi.schemas.data.data_specs
        • emmi.schemas.models
      • emmi.setup
      • emmi.types
      • emmi.utils
        • emmi.utils.aistore_utils
        • emmi.utils.point_cloud_utils
        • emmi.utils.running_moments
  • Low level API (ksuit)
    • ksuit
      • ksuit.callbacks
        • ksuit.callbacks.base
          • ksuit.callbacks.base.callback_base
          • ksuit.callbacks.base.periodic_callback
          • ksuit.callbacks.base.periodic_iterator_callback
        • ksuit.callbacks.checkpoint_callbacks
          • ksuit.callbacks.checkpoint_callbacks.best_checkpoint
          • ksuit.callbacks.checkpoint_callbacks.checkpoint
          • ksuit.callbacks.checkpoint_callbacks.ema
        • ksuit.callbacks.default_callbacks
          • ksuit.callbacks.default_callbacks.dataset_stats
          • ksuit.callbacks.default_callbacks.eta
          • ksuit.callbacks.default_callbacks.lr
          • ksuit.callbacks.default_callbacks.online_loss
          • ksuit.callbacks.default_callbacks.param_count
          • ksuit.callbacks.default_callbacks.peak_memory
          • ksuit.callbacks.default_callbacks.progress
          • ksuit.callbacks.default_callbacks.train_time
        • ksuit.callbacks.early_stoppers
          • ksuit.callbacks.early_stoppers.early_stopper_base
          • ksuit.callbacks.early_stoppers.fixed_early_stopper
          • ksuit.callbacks.early_stoppers.metric_early_stopper
        • ksuit.callbacks.offline_callbacks
          • ksuit.callbacks.offline_callbacks.offline_loss
        • ksuit.callbacks.online_callbacks
          • ksuit.callbacks.online_callbacks.best_metric
          • ksuit.callbacks.online_callbacks.update_output
      • ksuit.cli
        • ksuit.cli.main_calculate_stats
        • ksuit.cli.main_train
      • ksuit.configs
        • ksuit.configs.hyperparameters
        • ksuit.configs.static_config
        • ksuit.configs.wandb_config
      • ksuit.data
        • ksuit.data.base
          • ksuit.data.base.dataset
          • ksuit.data.base.subset
          • ksuit.data.base.wrapper
        • ksuit.data.pipeline
          • ksuit.data.pipeline.batch_processor
          • ksuit.data.pipeline.collator
          • ksuit.data.pipeline.multistage
          • ksuit.data.pipeline.sample_processor
        • ksuit.data.preprocessors
          • ksuit.data.preprocessors.compose
          • ksuit.data.preprocessors.preprocessor
          • ksuit.data.preprocessors.types
          • ksuit.data.preprocessors.utils
        • ksuit.data.samplers
          • ksuit.data.samplers.interleaved_sampler
          • ksuit.data.samplers.internals
        • ksuit.data.wrappers
          • ksuit.data.wrappers.dataset_wrappers
            • ksuit.data.wrappers.dataset_wrappers.repeat_wrapper
            • ksuit.data.wrappers.dataset_wrappers.shuffle_wrapper
            • ksuit.data.wrappers.dataset_wrappers.subset_wrapper
          • ksuit.data.wrappers.property_subset_wrapper
      • ksuit.distributed
        • ksuit.distributed.config
          • ksuit.distributed.config.distributed_config
        • ksuit.distributed.functional
          • ksuit.distributed.functional.all_gather_grad_autograd
        • ksuit.distributed.gather
        • ksuit.distributed.run
          • ksuit.distributed.run.managed
          • ksuit.distributed.run.unmanaged
          • ksuit.distributed.run.utils
      • ksuit.extractors
        • ksuit.extractors.forward_hook
      • ksuit.factory
        • ksuit.factory.base
          • ksuit.factory.base.factory_base
          • ksuit.factory.base.reflection_utils
        • ksuit.factory.dataset_factory
        • ksuit.factory.optim_factory
        • ksuit.factory.schedule_factory
      • ksuit.initializers
        • ksuit.initializers.base
          • ksuit.initializers.base.checkpoint_initializer
          • ksuit.initializers.base.initializer_base
        • ksuit.initializers.previous_run_initializer
        • ksuit.initializers.resume_initializer
      • ksuit.models
        • ksuit.models.composite
        • ksuit.models.model_base
        • ksuit.models.single
      • ksuit.optimizer
        • ksuit.optimizer.optimizer_wrapper
        • ksuit.optimizer.param_group_modifiers
          • ksuit.optimizer.param_group_modifiers.base
            • ksuit.optimizer.param_group_modifiers.base.param_group_modifier_base
          • ksuit.optimizer.param_group_modifiers.lr_scale_by_name_modifier
          • ksuit.optimizer.param_group_modifiers.weight_decay_by_name_modifier
      • ksuit.providers
        • ksuit.providers.dataset_config_provider
        • ksuit.providers.metric_property_provider
        • ksuit.providers.path_provider
      • ksuit.runners
        • ksuit.runners.hydra_runner
        • ksuit.runners.stats_runner
      • ksuit.schedules
        • ksuit.schedules.base
          • ksuit.schedules.base.decreasing_progress_schedule
          • ksuit.schedules.base.increasing_progress_schedule
          • ksuit.schedules.base.progress_schedule
          • ksuit.schedules.base.schedule_base
          • ksuit.schedules.base.sequential_percent_schedule
          • ksuit.schedules.base.sequential_step_schedule
        • ksuit.schedules.constant_schedule
        • ksuit.schedules.cosine
          • ksuit.schedules.cosine.decreasing
          • ksuit.schedules.cosine.increasing
        • ksuit.schedules.custom_schedule
        • ksuit.schedules.functional
        • ksuit.schedules.linear
          • ksuit.schedules.linear.decreasing
          • ksuit.schedules.linear.increasing
        • ksuit.schedules.linear_warmup_cosine_decay
        • ksuit.schedules.polynomial
          • ksuit.schedules.polynomial.decreasing
          • ksuit.schedules.polynomial.increasing
        • ksuit.schedules.step
          • ksuit.schedules.step.decreasing
          • ksuit.schedules.step.fixed
          • ksuit.schedules.step.interval
      • ksuit.scripts
        • ksuit.scripts.validate_pytorch_compatibilities
      • ksuit.setup
      • ksuit.trackers
        • ksuit.trackers.base_tracker
        • ksuit.trackers.noop_tracker
        • ksuit.trackers.wandb_tracker
      • ksuit.trainers
        • ksuit.trainers.base
        • ksuit.trainers.types
      • ksuit.utils
        • ksuit.utils.bidict
        • ksuit.utils.code
        • ksuit.utils.common
          • ksuit.utils.common.naming
          • ksuit.utils.common.path
          • ksuit.utils.common.stopwatch
          • ksuit.utils.common.typing
          • ksuit.utils.common.validation
        • ksuit.utils.data
          • ksuit.utils.data.data_container
        • ksuit.utils.logging
          • ksuit.utils.logging.core
          • ksuit.utils.logging.formatting
          • ksuit.utils.logging.progress
        • ksuit.utils.model
          • ksuit.utils.model.helpers
          • ksuit.utils.model.model_utils
        • ksuit.utils.platform
          • ksuit.utils.platform.system
          • ksuit.utils.platform.worker
        • ksuit.utils.torch
          • ksuit.utils.torch.amp
          • ksuit.utils.torch.device
        • ksuit.utils.training
          • ksuit.utils.training.counter
          • ksuit.utils.training.running_stats
          • ksuit.utils.training.schedule_wrapper
          • ksuit.utils.training.seed
          • ksuit.utils.training.training_iteration
      • ksuit.writers
        • ksuit.writers.checkpoint_writer
        • ksuit.writers.log_writer
Back to top
View this page

emmi.modules.attention¶

Submodules¶

  • emmi.modules.attention.anchor_attention
  • emmi.modules.attention.dot_product_attention
  • emmi.modules.attention.irregular_nat_attention
  • emmi.modules.attention.perceiver_attention
  • emmi.modules.attention.transolver_attention
  • emmi.modules.attention.transolver_plusplus_attention
Next
emmi.modules.attention.anchor_attention
Previous
emmi.modules.activations
Copyright © 2025, Emmi AI Developers
Made with Furo
On this page
  • emmi.modules.attention
    • Submodules