finding exoplanets

For Google Summer of Code 2022 I worked with ML4Sci on a project with a deliberately open brief: take publicly available observational data and build a model that predicts whether a given star has a planet around it. Any data source, any method, no limit on the models applied.

I chose lightcurves — the brightness of a star measured over time. When a planet crosses in front of its star the curve dips, and the shape of that dip is the signal. Lightcurves appealed to me because they are close to what the telescope actually recorded; less has already been decided for you.

the data

Two sources, combined. The Kepler Object of Interest cumulative table lists every star Kepler examined and what was eventually concluded about each one, but not the observations themselves. For those I used Lightkurve, which pulls the curves down by catalogue number.

Raw lightcurves are not something you can hand to a model. They are of unequal lengths, they are full of NaNs, they carry long-term trends from dying stars and instrument drift, and they have outliers. So I built two datasets out of the same curves and compared them:

• Global view, following the preprocessing described in Shallue and Vanderburg — fold on the orbital period, bin, normalise. A lot is decided for you, but it is decided well.
• Raw view — the lightest processing that produces something a model will accept at all: drop the outliers, resample onto equal time-steps, and stop there.

the models

I used tsai to benchmark a range of time-series classifiers across both datasets, then fully trained the ones that looked promising: ResNet, InceptionTime, XceptionTime, MiniRocket and a Time Series Transformer. Training everything properly would have cost more compute than I had, so the benchmark was there to choose.

The benchmark pointed at ResNet. The benchmark was wrong. The Time Series Transformer, which the quick pass had not covered, turned out to be the best model on test — 87% validation and 84% test accuracy after a thousand epochs, with the dropout rate tuned.

what I would tell myself

Processing mattered more than the architecture. The gap between the global view and the raw view was wider than the gap between any two models I tried. The preprocessing paper was the most useful thing I read all summer, and if you are starting this project, start there rather than with the model zoo.

I was limited by memory, not by ideas. Above roughly 500 samples I hit GPU out-of-memory errors, and reducing the batch size did not rescue it. So the entire study runs on about 500 examples. For deep learning that is nothing, and it is the reason 84% is the number rather than something better. Kepler observed approximately 150,000 stars — there is a great deal of headroom in simply being able to load them all.

If I picked this up again I would start with the full Kepler set, extend to K2 and TESS, and move from binary classification to multiclass — a planet or not is a much blunter question than the data can answer.

Thanks to my mentors Sergei Gleyzer and Jason Terry, to my fellow contributor Mahdi Boulila, and to the ML4Sci community.

My project code is at Torus403/GSoC22. The wider project, including the kinematic and tabular approaches other contributors took, lives in ML4SCI/EXXA.