Policy Search and Network-Dependent Dynamics

At times, we wish to model a component of the dynamics with a neural network. A common example is the policy search case, when the closed-loop dynamics include a neural network controller. In such cases, we consider the dynamics to take the form of $\frac{dx}{dt} = f(x, u, p, t)$, where $u$ is the control input/the contribution to the dynamics from the neural network. We provide the add_policy_search function to transform a NeuralLyapunovStructure into a NeuralLyapunovControlStructure to include training the neural network to represent not just the Lyapunov function, but also the relevant part of the dynamics.

Similar to get_numerical_lyapunov_function, we provide the get_policy convenience function to construct $u(x)$ that can be combined with the open-loop dynamics $f(x, u, p, t)$ to create closed loop dynamics $f_{cl}(x, p, t) = f(x, u(x), p, t)$.

NeuralLyapunov.add_policy_search — Function

add_policy_search(lyapunov_structure, new_dims; control_structure)

Add dependence on the neural network to the dynamics in a NeuralLyapunovStructure.

Arguments

lyapunov_structure::NeuralLyapunovStructure: provides structure for $V, V̇$; should assume dynamics take a form of f(x, p, t).
new_dims::Integer: number of outputs of the neural network to pass into the dynamics through control_structure.

Keyword Arguments

control_structure: function that transforms the final new_dims outputs of the neural network before passing them as u into the dynamics f(x, u, p, t); defaults to (phi, x, x0) -> phi(x), passing in the neural network outputs unchanged.

The returned NeuralLyapunovStructure expects dynamics of the form f(x, u, p, t), where u captures the dependence of dynamics on the neural network (e.g., through a control input). When evaluating the dynamics, it uses u = control_structure(phi_end(x)) where phi_end is a function that returns the final new_dims outputs of the neural network. The other lyapunov_structure.network_dim outputs are used for calculating $V$ and $V̇$, as specified originally by lyapunov_structure.

add_policy_search(NonnegativeStructure(3), 1)
# output
NeuralLyapunovControlStructure
    Network dimension: 4
    V(x) = φ_V(x)²
    V̇(x) = 2∇φ_V(x)*φ_V(x)*ẋ
    u(x) = φ_c(x)

source

NeuralLyapunov.get_policy — Function

get_policy(phi, θ; fixed_point, control_structure, idx)
get_policy(phi, θ, network_dim, control_dim; fixed_point, control_structure)
get_policy(phi, θ, structure::AbstractNeuralLyapunovStructure{true}; fixed_point)

Generate a Julia function representing the control policy/unmodeled portion of the dynamics as a function of the state.

The returned function can operate on a state vector or columnwise on a matrix of state vectors.

Positional Arguments

phi: the neural network, represented as an AbstractVector of functions phi(x, θ); typically this is the phi field of the output of NeuralPDE.PhysicsInformedNN.
θ: the parameters of the neural network. For each index i of phi, θ[:φi] should be the parameters of the network phi[i]. i.e., the full neural network output should be [ phi(x, θ[:φi]) for i in eachindex(phi) ].
network_dim: total number of neural network outputs; inferred from structure if provided. See idx below for more information.
control_dim: number of neural network outputs used in the control policy; inferred from structure if provided. See idx below for more information.
structure::AbstractNeuralLyapunovStructure{true}: provides the control structure and dimensions for the neural network outputs used in the control policy.

Keyword Arguments

fixed_point: the fixed point of the system.
control_structure: function that transforms the outputs of phi[idx] before passing them as u into the dynamics f(x, u, p, t); inferred from structure if provided, otherwise defaults to (phi, x, x0) -> phi(x), passing in the neural network outputs unchanged.
idx: the neural network outputs to pass into control_structure. When structure or network_dim and control_dim are provided, this is automatically set to (network_dim - control_dim + 1):network_dim. Otherwise, this must be specified by the user.

source

NeuralLyapunov.NeuralLyapunovControlStructure — Type

NeuralLyapunovControlStructure(V, V̇, control_structure, network_dim, control_dim)

Specifies the structure of the neural Lyapunov function and its derivative.

Allows the user to define the Lyapunov in terms of the neural network, potentially structurally enforcing some Lyapunov conditions.

Fields

V(phi, state, fixed_point): outputs the value of the Lyapunov function at state.
V̇(phi, J_phi, state, dstate_dt, fixed_point): outputs the time derivative of the Lyapunov function at state.
control_structure(phi_c, state, fixed_point): transforms the final control_dim outputs of the neural net before passing them as u into the dynamics f(x, u, p, t).
network_dim: the dimension of the output of the neural network.
control_dim: the number of neural network outputs used in the control policy.

phi and J_phi above are both functions of state alone.

source