tag:blogger.com,1999:blog-107568321062020427.post1042116335729943724..comments2023-09-19T00:48:06.643-07:00Comments on Intelligent Trading: The Kalman Filter For Financial Time SeriesIntelligent Tradinghttp://www.blogger.com/profile/17765336450326139518noreply@blogger.comBlogger51125tag:blogger.com,1999:blog-107568321062020427.post-77603170171805857492016-01-27T17:22:30.547-08:002016-01-27T17:22:30.547-08:00@Jan
Thanks for sharing!@Jan<br /><br />Thanks for sharing!Intelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-376253851943429562016-01-17T07:45:09.397-08:002016-01-17T07:45:09.397-08:00Regarding the discussion of dlmFilter and dlmSmoot...Regarding the discussion of dlmFilter and dlmSmooth in the R DLM package, or the comparable two functions in the pykalman package, I don't don't why causality in the EE sense should recommend itself, particularly in a batch or moving window process. A Rauch-Tung-Striebel smoother (as in dlmSmooth, and sometimes sloppily referred to as a "Kalman smoother") is a Bayesian updater. The idea is that after one has made a forward estimate, as in dlmFilter, and the actual data is observed, it's entirely sensible to go backwards and update the state estimates in light of what was actually observed. Accordingly the filtering-smoothing process has a structure like that of the EM algorithm ("expectation-maximization") where there is a run forward through the dataset, and then backwards doing the state update. <br /><br />BTW, the pykalman package for Python is fine, is even available for Python 3, and handles missing observations like DLM does, but I found it's estimate of the log-likelihood to be broken. Also, it uses a nonlinear optimizer whereas DLM uses the SVD. Accordingly, the DLM is much more numerically stable than pykalman.<br /><br />I have started a code to implement the logic of DLM in Python, but I had to shelve it. I was hoping to simply translate and then clean up the DLM open source code into Python, but found it difficult to free of decidedly R-ish things. So, this would be a nice thing to do, but I need to design it afresh from the equations, and have not found the time yet.Jan Galkowskihttps://www.blogger.com/profile/07636706072515906253noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-79325630003537715942013-06-20T01:03:23.340-07:002013-06-20T01:03:23.340-07:00Hi Dave,
One of these days I could try to put som...Hi Dave,<br /><br />One of these days I could try to put something together. What I've found regarding the versions, is that using a more sophisticated variant (e.g. particle) does not necessarily help in short term forecasting of trades. <br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-3321142958135429102013-06-11T20:23:35.220-07:002013-06-11T20:23:35.220-07:00IT would you be able to do a post (without divulgi...IT would you be able to do a post (without divulging information) on how one of these predictors could be used?Davehttps://www.blogger.com/profile/06233866020771278274noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-18884475301011070982013-06-06T20:00:00.859-07:002013-06-06T20:00:00.859-07:00Thanks IT. Did you find that the linear version of...Thanks IT. Did you find that the linear version of the filter to be sufficient. Or did you have to move onto variants I.e unfiltered, particle, gaussian process, etc...Davehttps://www.blogger.com/profile/06233866020771278274noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-73732609585683404292013-06-03T18:03:28.695-07:002013-06-03T18:03:28.695-07:00Hi Dave,
Sorry for the delay. I appreciate your f...Hi Dave,<br /><br />Sorry for the delay. I appreciate your feedback. Without divulging any proprietary knowledge, I can only concur that that is one of the challenges I worked on for a long time.<br />There are dozens of econometric models that already exist for this; some have worked for years on some price series.<br /><br />I will add to your thoughts in that something you might also consider besides just function is to explore what types of predictors might also be useful in price forecasting. <br /><br />Regards,<br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-30256444205654890752013-05-30T20:39:59.442-07:002013-05-30T20:39:59.442-07:00yes I saw the sin function version implemented at ...yes I saw the sin function version implemented at the blog you mentioned. If one wanted to apply this but to a simple price series what kind of price function could they use? I guess the hardest part here is how do you determine what type of function to use when predicting something like price?Davehttps://www.blogger.com/profile/06233866020771278274noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-70714302928733424922013-05-30T01:37:59.678-07:002013-05-30T01:37:59.678-07:00Hi Dave,
I have experimented with both, but there...Hi Dave,<br /><br />I have experimented with both, but there aren't many (easy to implement) public packages available to do so. There was a writeup and some code available on TR8DR blog site, some time ago.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-64298508942695988892013-05-24T17:42:24.988-07:002013-05-24T17:42:24.988-07:00Have you experimented at all using either the Part...Have you experimented at all using either the Particle or Unscented version? What I am wondering is how one could can construct a simple price function so to be used in conjunction with either of the non linear versions of the Kalman filter.Davehttps://www.blogger.com/profile/06233866020771278274noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-46176183192159827372013-05-09T23:05:51.759-07:002013-05-09T23:05:51.759-07:00mahdi,
Glad to hear it. Thanks for letting me kno...mahdi,<br /><br />Glad to hear it. Thanks for letting me know and please let me know if there is any link to the article.<br /><br />Best,<br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-66037002214201460372013-05-07T13:44:37.847-07:002013-05-07T13:44:37.847-07:00Dear IT,
thanks for your help. the article is alm...Dear IT,<br /><br />thanks for your help. the article is almost done. you are the best<br />Anonymoushttps://www.blogger.com/profile/01064236445207563691noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-64305152472356353882013-03-10T17:44:33.495-07:002013-03-10T17:44:33.495-07:00Hi mahdi,
I don't have a one size fits all fu...Hi mahdi,<br /><br />I don't have a one size fits all function for you. But certainly, you could start out using a very simple RMSE metric. If you have control over fitness, you could try to target some metrics like CAGR, terminal wealth, and sharpe ratio to start.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-39207235762050791322013-03-09T07:07:18.175-08:002013-03-09T07:07:18.175-08:00Hey there,
I am working on an article and i am mi...Hey there,<br /><br />I am working on an article and i am mixing Improved particle swarm optimization with KF to get the best result on stock forecasting. i am trying to find out the fitness function but have not been successful yet. have any idea?Anonymoushttps://www.blogger.com/profile/01064236445207563691noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-61004904780234293232011-12-29T22:30:05.015-08:002011-12-29T22:30:05.015-08:00A lot of it is part art, part science. However, y...A lot of it is part art, part science. However, you can borrow from machine learning-- various validation and regularization methods to each series. Most of these methods try to generalize and tradeoff between fitted bias and variance vs model complexity.<br /><br />At the end of the day, you want to best generalize your model to minimize errors on out of sample data. That discussion however, would take a much larger post that I might consider writing some time in the future.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-45129670557055255292011-12-28T12:44:16.061-08:002011-12-28T12:44:16.061-08:00Anon from above (should register; excellent websit...Anon from above (should register; excellent website). <br /><br />Thanks for your answer. Playing around with KF in Matlab/ R so understand it better. Still not sure about setting the R/Q ratio systematically. Some time series that I am applying KF to are "noisier" than others. They are RW's with some transients on top. <br /><br />For some financial series I'd want the KF to update quiet fast, while slow for others. Can I invoke any arguments from information theory/ volatility analysis/ autocorrelation to set an optimal R/Q ratio systematically for any given arbitrary RW time series rather than choosing a ratio that "looks" reasonable? <br /><br />Thanks.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-57730590561615361292011-12-13T00:17:57.973-08:002011-12-13T00:17:57.973-08:00anon,
Sorry I've been out a few days, but the ...anon,<br />Sorry I've been out a few days, but the update will automatically happen in the KF equation updater. You can do many things to adjust and toy around with some variables like the gain factor, K, itself. In the R program DLM, I believe they show examples of how do deal with discrete time jumps and quickly updating based upon the jump size.<br /><br />All you should have to really do (assuming Gaussian type RW) is set the initial R and Q Values to reasonable sizes and ratios, the rest will work itself out in the update (feedback) process. If is updating too slow or fast, you can try to run sweeps of R, Q sizes, and ratios of meas/process noise to your time series.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-20194000754207969332011-12-09T07:32:47.423-08:002011-12-09T07:32:47.423-08:00hi,
Coming back to the question of R & Q. For...hi,<br /><br />Coming back to the question of R & Q. For time series processes should I update R (measurement noise) based on realized volatility calculation to capture the varying volatility dynamics of financial time series? How should I update Q if I do this for this R?<br /><br />Cheers!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-9712604685474222052011-12-03T14:40:57.527-08:002011-12-03T14:40:57.527-08:00thanks K.thanks K.Intelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-69693156727818444332011-12-01T23:32:11.802-08:002011-12-01T23:32:11.802-08:00Just thought I would make a comment on the 'dl...Just thought I would make a comment on the 'dlm' package.<br /><br />I believe the function dlmFilter is what xfm was looking for; dlmFilter is causal, while dlmSmooth is not.Knoreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-7024740240690761812011-11-30T19:09:05.471-08:002011-11-30T19:09:05.471-08:00anon,
One way I approach it is to start by visuali...anon,<br />One way I approach it is to start by visualizing the result running sweeps of R and Q values and observing if the response is what I am looking for in the estimator.<br />I've found in the case of estimating time series as in the example I posted is to use a very small value of process noise Q, relative to the measurement noise, R, as we are looking for a fairly smooth estimate of the process.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-4318602815855246142011-11-23T23:46:58.276-08:002011-11-23T23:46:58.276-08:00Hi, IT
One question about the kalman fiter: How d...Hi, IT<br /><br />One question about the kalman fiter: How do you estimate the measurement noise R and the process noise Q? Or do you assign a constant to them? Thanks.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-67159097346092742072011-11-05T00:21:13.228-07:002011-11-05T00:21:13.228-07:00Hi Peter,
It's been a long while since I wrot...Hi Peter,<br /><br />It's been a long while since I wrote this post and the accompanying code. But generally, we simply enter some actual price series or a rw model, or variant such as arma/arima, we also intialize the algorithm with an estimate of the mean and covariance of the process.<br />Some of these initial estimates are arrived at via trial and error or having knowledge of the system behavior beforehand. <br /><br />Blue line is xhat (prior) forecast overlaid with actual series (present) in Red. The Red series in this example is a simple gbm model of a random walk.<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-20145169811960018992011-10-31T20:49:18.275-07:002011-10-31T20:49:18.275-07:00IT,
Regarding Kalman filter, how do you model the...IT,<br /><br />Regarding Kalman filter, how do you model the returns (or do you model the price). Which process do you use? From my understanding, we can't simply plug in the returns without modelling them as an ARMA/ARIMA process, at the least. Does the graph on top show xhat (the posterior) or xhatminus (a priori) forecast. <br /><br />Thanks for writing this blog!<br /><br />PeterPeterhttps://www.blogger.com/profile/18285057131325157840noreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-38705411322421536032011-09-23T20:03:57.968-07:002011-09-23T20:03:57.968-07:00IT,
What I actually meant is what you described i...IT,<br /><br />What I actually meant is what you described in the second paragraph of your response, ie. updating the Beta parameters of a linear model dynamically by utilising the Kalman filter. If that's already a common occurrence in portfolio modelling, I take comfort in the fact that I seem to be on the right track.<br /><br />Now all that remains is finding appropriate factors. Thanks for your response.<br /><br />Cheers,<br />JeffAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-107568321062020427.post-44730991635629832542011-09-23T17:29:59.515-07:002011-09-23T17:29:59.515-07:00scottc,
Thank you for the kind words. Regarding m...scottc,<br /><br />Thank you for the kind words. Regarding multivariate modeling, I haven't done too much in this area with respect to the kalman filter. However, there is an R package called KFAS you might want to look into.<br /><br />http://cran.r-project.org/web/packages/KFAS/index.html<br /><br />ITIntelligent Tradinghttps://www.blogger.com/profile/17765336450326139518noreply@blogger.com