List of Normalizer Layers

All normalizer layers scale and shift tensors, but do not have learnable parameters.

GaussianNormalizer

class GaussianNormalizer(x, encoding=True, epsilon=1e-08, dim=None)[source]

Normalize a tensor to have mean of zero and standard deviation of one.

Note

Due to machine percision, mean and standard deviation may have errors on the order of \(10^{-8}\). Using different data types may affect percision of results.

Parameters:

  • x (Tensor) – Tensor of any shape

  • encoding (bool) – If True, scale and shift a tensor to have mean of zero and standard deviation of one. If False, scale and shift from a mean of zero and std of one to the original mean and std of x. Default: True

  • epsilon (float) – Small positive value added to standard deviation for numerical stability

  • dim (Union[int, tuple, None]) – Dimensions to be reduced in \(\text{mean}(x), \text{std}(x)\). If None, reduce all dimensions for scalar min and max. Default: None

Raises:

RuntimeError – If torch.mean(x) or torch.std(x) contains infinite (inf) or not-a-number (nan) over a dimension to be reduced

Shape:

  • Input: Any shape

  • Output: Any shape (same shape as input)

Example:

>>> # use one instance and change mode
>>> torch.manual_seed(0)  # for reproducibility
>>> x = torch.rand(100, 100)
>>> normalizer = sml.GaussianNormalizer(x)
>>> y = normalizer(x)
>>> normalizer.decode()  # equivalent to normalizer.encode(False)
>>> x_reconstruction = normalizer(y)
>>> print(x.mean(), y.mean(), x_reconstruction.mean())
>>> print(x.std(), y.std(), x_reconstruction.std())
tensor(0.5003) tensor(-6.5804e-09) tensor(0.5003)
tensor(0.2879) tensor(1.) tensor(0.2879)

>>> # use two instances with different modes
>>> torch.manual_seed(0)  # for reproducibility
>>> x = torch.rand(100, 100)
>>> encoder = sml.GaussianNormalizer(x)
>>> decoder = sml.GaussianNormalizer(x, encoding=False)
>>> y = encoder(x)
>>> x_reconstruction = decoder(y)
>>> print(x.mean(), y.mean(), x_reconstruction.mean())
>>> print(x.std(), y.std(), x_reconstruction.std())
tensor(0.5003) tensor(-6.5804e-09) tensor(0.5003)
tensor(0.2879) tensor(1.) tensor(0.2879)
forward(x)[source]

Scale x to have a new mean and standard deviation.

If self.encoding is True, return \(\frac{x - \text{mean}}{\text{std} + \epsilon}\). If self.encoding is False, return \((x \times (\text{std} + \epsilon)) + \text{mean}\)

Parameters:

x (Tensor) – Tensor of any shape

Return type:

Tensor

Returns:

Tensor of same shape as x

encode(mode=True)[source]

Set the Normalizer to scale a tensor a mean of zero and standard deviation of one

Parameters:

mode (bool) – If True, set self.encoding to True. Default: True

decode(mode=True)[source]

Set the normalizer to restore a tensor to its original mean and standard deviation

Parameters:

mode (bool) – If True, set self.encoding to False. Default: True

Attributes

GaussianNormalizer.encoding
GaussianNormalizer.mean: torch.Tensor

Original mean of the initialization tensor

GaussianNormalizer.std: torch.Tensor

Original standard deviation of the initialization tensor

RangeNormalizer

class RangeNormalizer(x, encoding=True, low=0.0, high=1.0, dim=None)[source]

Normalize a tensor to fall within the range \([\text{low}, \text{high}]\)

Note

Due to machine precision, normalized values may be outside of range by errors on the order of \(10^{-8}\).

Parameters:

  • x (Tensor) – Tensor of any shape

  • encoding (bool) – If True, scale and shift a tensor to be within \([\text{low}, \text{high}]\). If False, scale and shift from \([\text{low}, \text{high}]\) to the original range of x. Default: True

  • low (Union[int, float]) – Lower bound of the normalized range

  • high (Union[int, float]) – Upper bound of the normalized range

  • dim (Union[int, tuple, None]) – Dimensions to be reduced in \(\min(x), \max(x)\). If None, reduce all dimensions for scalar min and max. Default: None

Raises:

  • ValueError – If low greater or equal to high

  • RuntimeError – If \(\min(x)\) equals \(\max(x)\) over any dimension to be reduced. This is to prevent a ZeroDivisionError from occuring in the computation of scale

Shape:

  • Input: Any shape

  • Output: Any shape (same shape as input)

Example:

>>> # use one instance and change mode
>>> torch.manual_seed(0)  # for reproducibility
>>> x = torch.normal(0, 1, (100, 100))
>>> normalizer = sml.RangeNormalizer(x)
>>> y = normalizer(x)
>>> normalizer.decode()  # equivalent to normalizer.encode(False)
>>> x_reconstruction = normalizer(y)
>>> print(x.min(), y.min(), x_reconstruction.min())
>>> print(x.max(), y.max(), x_reconstruction.max())
tensor(-4.3433) tensor(0.) tensor(-4.3433)
tensor(4.1015) tensor(1.) tensor(4.1015)

>>> # use two instances with different modes
>>> torch.manual_seed(0)  # for reproducibility
>>> x = torch.normal(0, 1, (100, 100))
>>> encoder = sml.RangeNormalizer(x)
>>> decoder = sml.RangeNormalizer(x, encoding=False)
>>> y = encoder(x)
>>> x_reconstruction = decoder(y)
>>> print(x.min(), y.min(), x_reconstruction.min())
>>> print(x.max(), y.max(), x_reconstruction.max())
tensor(-4.3433) tensor(0.) tensor(-4.3433)
tensor(4.1015) tensor(1.) tensor(4.1015)
forward(x)[source]

Scale and shift x to fall within a new range.

If self.encoding is True, return \((x \times \text{scale}) + \text{shift}\). If self.encoding is False, return \(\frac{x - \text{shift}}{\text{scale}}\)

Parameters:

x (Tensor) – Tensor of any shape

Return type:

Tensor

Returns:

Tensor of same shape as x

encode(mode=True)[source]

Set the normalizer to scale and shift a tensor to fall within range \([ ext{low}, ext{high}]\)

Parameters:

mode (bool) – If True, set self.encoding to True. Default: True

decode(mode=True)[source]

Set the normalizer to restore a tensor to its original range

Parameters:

mode (bool) – If True, set self.encoding to False. Default: True

Attributes

RangeNormalizer.encoding
RangeNormalizer.scale: torch.Tensor

Multiplicative factor to rescale range of x to interval width

RangeNormalizer.shift: torch.Tensor

Additive factor to make interval start at self.low