Modality for Reinforcement Learning

Home / Research / Dissertation / Definitions / Modality for Reinforcement Learning

In this post, I define modality within the context of a reinforcement learning agent.  As a neurophysiological concept, sensory modalities are fairly intuitive.  Hearing and vision are a typical example of two different sensory modalities.  Motor modalities are a little less intuitive, but not difficult to understand.  A motor modality may simply be any particular pattern of activated motor neurons during a movement.

But, when talking about this in terms of a Reinforcement Learning agent, we have to talk about it in the appropriate formalism.  This formalism for a reinforcement learning agent is the Markov Decision Process.  In an MDP, the analogues for sensors and motors are within the definition of states and actions.

rl

In a MDP, states and actions are typically defined as vectors and the components of the vectors are called variables or features.  I will use the terms “variables” and “features” interchangeably.

Consider, for purposes of discussion, an agent that can move in 3D space using variable thrusters oriented orthogonally from each other.  Consider a vector for a three-dimensional space that defines the state (position of the agent in the world).  The individual position values for each dimension are the state features (i.e. x, y, and z).  Similarly an action may be defined as a vector.  Our 3D agent could have the ability to thrust in any or all of the three dimensions; so its action vector may have three components indicating a thrust.  So, each thruster can be described with an action variable.  These are the analogues for sensors and motors.

Definitions

With respect to actuators, a motor modality, in the RL context, is any given subset of action features.

With respect to perception and observations, a sensory modality is any given subset of state features.

Thoughts

Is walking on two legs a different modality than hopping?  If so, the same action features may be in play, but perhaps this represents constraints on the range of those features.

Next

How is this meaningful to my research?

References

[1] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal Deep Learning,” Proc. 28th Int. Conf. Mach. Learn., pp. 689–696, 2011.

[2] D. Nozaki and S. H. Scott, “Multi-compartment model can explain partial transfer of learning within the same limb between unimanual and bimanual reaching.,” Exp. Brain Res., vol. 194, no. 3, pp. 451–63, Apr. 2009.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: