Elsevier

Information Sciences

Volume 178, Issue 23, 1 December 2008, Pages 4453-4468
Information Sciences

Inference of differential equation models by genetic programming

https://doi.org/10.1016/j.ins.2008.07.029Get rights and content

Abstract

This paper describes an evolutionary method for identifying a causal model from the observed time-series data. We use a system of ordinary differential equations (ODEs) as the causal model. This approach is known to be useful for practical applications, e.g., bioinformatics, chemical reaction models, control theory, etc. To explore the search space more effectively in the course of evolution, the right-hand sides of ODEs are inferred by genetic programming (GP) and the least mean square (LMS) method is used along with the ordinary GP. We apply our method to several target tasks and empirically show how successfully GP infers the systems of ODEs. We also describe an extension of the approach to the inference of differential equation systems with transcendental functions.

Introduction

Ordinary differential equations (ODEs) are one of the easiest media for modeling complex systems, where basic differential relationships are known between the system components. Solving a set of differential equations to produce their equivalent functions (for obtaining useful time-series data) is relatively easy. On the other hand, the inverse problem, i.e., the inference of the system of ODE from the observed time-series data, is not necessarily easy, although very important for many fields. This is because the appropriate form of the ODE (i.e., the order and terms) is not known in advance.

In this paper, we deal with an arbitrary form in the right-hand side of the system of ODEs to allow for flexibility of the model. More precisely, we consider the following general form:dXidt=fi(X1,X2,,Xn)(i=1,2,,n),where Xi is the state variable and n is the number of the observable components.

For the sake of identifying the system, we use genetic programming (GP) to evolve the ODEs from the observed time series. Although GP is effective in finding a suitable structure of the purported solution, it is sometimes difficult to optimize the parameters, such as constants or coefficients of the polynomials. This is because the ordinary GP searches for them simply by combining randomly generated constants. To avoid this difficulty, we introduce the least mean square (LMS) method.

There have been several studies for identifying differential equation models by means of EAs (evolutionary algorithms). For instance, GP was used to find a function in a symbolic form, which satisfies the differential equation and initial conditions [9]. Cao et al. used hybrid evolutionary modeling algorithms [4]. The main idea was to embed GA in GP, where GP was employed to discover and optimize the structure of a model, while GA was used to optimize its parameters, i.e., coefficients. GP was applied [3] to approximate several ODEs from the domain of ecological modeling, e.g., Lotka–Volterra and logistic equations. They showed that the GP-based approach introduced numerous advantages over the most available modeling methods. Grammatical evolution was also used for the sake of solving ordinary and partial differential equations [20]. In our previous researches [16], [17], we proposed another integrated scheme, in which the least mean square (LMS) method is used along with GP. In that scheme, some individuals were created by the LMS method at some intervals of generations and they replaced the worst individuals in the population.

In this paper, we extend our previous approach, in order to achieve the inference of the ODEs more effectively. More precisely, we empirically show the following points:

  • The success in the acquisition of ODEs, which are close to the observed time series.

  • The inference of the exact equation form, i.e., the exact causal relationship.

  • The effectiveness of the LMS method.

  • The superiority of our approach over the previous methods.

The rest of this paper is organized as follows. In Section 2, we describe the details of our method, i.e., how GP and LMS methods are integrated to work in the course of evolution. Three examples are used to examine the effectiveness of our method and experimental results are shown in Section 3. The extension of this approach is described in Section 4 for the inference problem of a differential equation system including transcendental functions. Results are discussed in Section 5 and we draw conclusions in Section 6.

Section snippets

Integration of GP and LMS

We use GP to identify a causal model in the form of the system of ODEs. Though GP is capable of finding a desirable structure effectively, it cannot always be effective in finding the proper coefficients because GP uses the combination of randomly selected ones. We have chosen the least mean square method (LMS) to tackle this deficiency of the ordinary GP. For this purpose, coefficients are not included in the terminal set of a GP individual tree. The coefficients of each term of a GP tree are

Experimental results

We have prepared three different tasks to test the effectiveness of our method. Experimental parameters are summarized in Table 1. Function and terminal sets F and T are as follows:F={+,-,},T={X1,,Xn,1}.

Inference of a differential equation system including transcendental functions

In a real-world application, many systems for dynamic modeling include periodic functions, such as sin or cos, and follow a so-called power law, i.e., require exp or log functions. In this section, we try to solve the inference problem of such a differential equation system including transcendental functions. For this purpose, we choose the following arithmetic function, variables, constants as well as transcendental functions:F={+,-,,,sin,cos,tan,ln,log,exp},T={X1,X2,,Xn,t,const},where F is

Robustness against noise

Although the previous section showed the effectiveness of our approach in producing the exact form which was very close to the target data, there is another factor to be considered, i.e., the robustness. To test the robustness of our method to the real noisy world, we conducted the E-cell experiment (Exp. 2) with noise-added data sets. Five percent and ten percent random noises were added to the target time series. The generated time series are plotted in Fig. 14 with the target data. The MSE

Conclusion

We have proposed an inference method of the system of ODEs from the observed time series by using GP along with the LMS method. We showed, by several experiments, how successfully our method can infer the causal model. More precisely, we succeeded in generating the system of ODEs which is very close to the observed time series. The effectiveness of the LMS method and the superiority of our approach over the previous method were confirmed by comparative experiments. We also described the

References (23)

  • D.W. Marquardt

    An algorithm for least-squares estimation of nonlinear parameters

    Journal of the Society for Industrial and Applied Mathematics

    (1963)
  • Cited by (60)

    • Controller design by symbolic regression

      2021, Mechanical Systems and Signal Processing
    • LatinPSO: An algorithm for simultaneously inferring structure and parameters of ordinary differential equations models

      2019, BioSystems
      Citation Excerpt :

      The research in Ren et al. (2016) uses a multi-objective Genetic Algorithm (GA) to find robust adaptation gene regulatory networks described as ODEs, and the aim is to find robust adaptation gene regulatory networks, but the learnt ODE model is not closely matching the time course data. The research in Iba (2008) evolves differential equation models by genetic programming, but this method had limitations in selecting the basis function and scalability. The research in Chen et al. (2011) presents a hybrid evolutionary method for identifying a system of ordinary differential equations (ODEs) to predict the small-time scale traffic measurements data, and it is not to search alternative structures with the same (or similar) system functionalities and behavior.

    • Numerical weather prediction revisions using the locally trained differential polynomial network

      2016, Expert Systems with Applications
      Citation Excerpt :

      The derivative series model is in principle quite different from direct composing computational techniques, which use a collection of operators and terminals of a predefined set to form symbolic tree-like structural expressions. The genetic programming can solve explicit DE forms (Cornforth & Lipson, 2013) or a pre-defined general DE system (Iba, 2008) after this manner. The D-PNN can combine the PNN functionality with some mathematical principles of DE substitutions.

    • Constructing general partial differential equations using polynomial and neural networks

      2016, Neural Networks
      Citation Excerpt :

      Partial differential equations define a variety of models for function approximation problems of dynamic systems (Rakkiyappan & Dharani, 2015) that are difficult to describe directly by unique explicit expressions (Rakkiyappan, Dharani, & Zhu, 2015). To form the solutions, one can apply evolutionary strategies (Chen, Yang, Meng, Zhao, & Abraham, 2011), genetic programming (GP) (Cao, Kang, Chen, & Yu, 2000; Iba, 2008) or artificial neural networks (ANNs) (Tsoulos, Gavrilis, & Glavas, 2009; Wojciechowski, 2012), which require differential equations to be pre-defined in a general system or standard explicit form. Sum series are widespread used to solve differential equations, e.g. power (Rudd & Ferrari, 2015), Fourier (Chaquet & Carmona, 2012) and polynomial or wavelet series (Liu, Niu, Wang, & Fan, 2014).

    View all citing articles on Scopus
    View full text