So about ten days ago I got interested in this project, which describes itself as the ‘hardest data science tournament on the planet’. This would be numer.ai — and I’ve decided to write a series about my experiences with getting to know this platform, what is it about, what are the people like etc. So, lets get down to business with my experience discovering numer.ai and why I think it is interesting. This is a very high level look at the Numerai platform while also a high level look at data science in general in case you are in the dark about that.
What is Numerai?
I found out about Numerai when I came across Lex Fridman’s interview with Richard Craib, founder of Numerai. I quickly came to understand that Numerai was a platform for aggregating as many opinions about stock market movements as possible. But how? And why? Lets start with my understanding of the why first, and why I think this is interesting.
Numerai is trying to do something very interesting. It is basically a Hedge Fund that is utilizing a large and open group of data scientists to make investment decisions for them in a tournament style of play. The larger the better because the Hedge Fund is making decisions based on ‘The Wisdom Of the Crowd’ — the more opinions given, the more opinions they can average out, which typically leads to a more ‘correct’ answer or in this case prediction. But Numerai want two aspects to be true about these predictions. Firstly, they should be informed opinions, based on data. This is why people with data science skills and interests are in need — another reason will be discussed in the ‘How Numerai’ part below. Secondly, to make sure opinions have a suitable motivation behind them, participants can ‘stake’ something of value behind their predictions in a tournament, meaning they have ‘skin in the game’ — again I will discuss this more in the ‘How Numerai’ part. If they are not confident they will not stake — this is important when trusting the wisdom of a crowd. While wisdom is hard to define, we can use ‘desire to not lose money’ as a proxy for it.
This is a neat experiment in getting many opinions (also called models) of predictions of stock performance under the condition that those giving their opinion actually care about the outcome, and then creating what Numerai call the ‘Meta Model’ — a compilation of everyone staking something of value in a tournament. How this tournament is scored is something I will discuss in another post (when I understand it better). So the power of this approach is Numerai harness over 2000 opinions, weekly, on how stocks will perform. Not only that, because these opinions come from individuals who don’t technically work together, those models/opinions have no reason to be correlated with each other. The models are about as diverse as can be expected. The same is most likely not true with Hedge Funds with rows of quants who all work closely with each other. Having said that, the Numerai community is open, friendly and collaborative so there is an element of group think going on as well, no doubt.
To participate in a Numerai tournament, you take data given by the Numerai people and try to build a model that matches a bunch of features in the data to targets in the data. To Machine Learning experts what features and targets are is obvious. To the novice, a little explanation is needed so we can understand how Numerai works.
A little Data Science aside — skip if you know what features and targets are.
Features are individual elements of data (a datum) which have a numerical value given for it. An example might be ‘temperature in Waikiki today’. I just checked and today it was 26 C (79 F). That’s below average given the range over the entire year (it is winter in the Northern Hemisphere as I write this). So lets say I am trying to predict how much ice cream will be sold down Waikiki. This feature (temperature) is likely to have some influence on a target we want to know, namely, volume of ice cream sold. Another feature might be hotel occupancy down Waikiki. Maybe that number is only 300 rooms out of 5000 (it’s also 2021 and we are still in the middle of the Covid-19 pandemic). Volume of ice cream (target — we are trying to predict) is perhaps 100 scoops — not a lot for a tourist destination. One important thing that is done with data like this is it is normalized. It can be hard to relate numerical values for temperature, hotel occupancy and scoops of ice cream. None of these share the same units for example. And it turns out a lot of machine learning algorithms hate having to deal with things in different units and certainly with features that have very different numerical scales (degrees Celsius and number of tourists could be several orders of magnitude different). So the numbers are normalized, usually between 0 and 1. For the temperature in Waikiki we might put 15 °C as 0.0 and 35 °C as 1.0 and scale everything else in between. For hotel occupancy, maybe we make this number the percentage of rooms in Waikiki occupied — also a number between 0.0 and 1.0 (100%). As for scoops of ice cream, maybe some domain knowledge tells us Waikiki can not sell more than 10000 scoops a day, so scoops sold would be divided by 10000, giving a number between 0.0 and 1.0. So in this brief example we would have two features, temperature (26/(35–15) = 0.55) and occupancy (300/5000 = 0.06) and a target of (100/10000 = 0.01). And then we do this over many days, collecting another row of feature values that give a target value. See table below:
This table could get quite large, with many many rows. Our job is to apply a machine learning algorithm to this table and build a model that can take the features and predict the target. So maybe we have a row with tomorrows temperature and hotel occupancy (these things can be found out or guessed) and we make a prediction on the target — scoops of ice cream to be sold.
Still with me? OK, good. So Numerai do the same thing. But their features are things that Numerai think will effect stock performance and the target is a measure of whether a stock position is favorable or not favorable. I will discuss the contents of Numerai’s data set in another post — but for now, this is the how of the tournament. Numerai publish data and a community of data scientists build models to make predictions.
But before I end this post, I need to explain two more things about the how Numerai works.
Firstly, the details of the features that Numerai use are largely unknown to the data scientists. That is, they are essentially given as Feature A, Feature B etc, , there are currently 310 of these features (they are grouped into groups of features but not too much can be discerned from that). This may seem odd, but it is a very smart thing to do. Without knowing what Numerai’s data is derived from, data scientists can not second guess what the features mean. We are completely blind to their syntax, their meaning, so our models do not become biased by what our human brains think is important. This is key in my opinion. The models that are built are only based on numerical values between zero and one.
Secondly, the Numerai data is broken up into ‘Eras’ (time periods) but the aim is not to predict time series data. Going back to our lovely Waikiki example, we don’t want to predict how many scoops will be sold based on yesterday, the day before and last week, as if one day directly influences the next. We are focused here on only using the features to predict the target in that row. I described this concept in a post on the forums over at numer.ai but rather than link to it, I will copy-pasta it below:
Say I want to predict what kind of vehicle is coming down my road next. It is far away so I can only determine some coarse grain properties. I can see its color, I can see how fast it is going, I can see if its exhaust is clear or sooty. Yellow, slow, sooty features suggest the next vehicle will be a school bus (in North America anyway). If those features were red, fast and clear exhaust that would suggest a Ferrari. Now there is no way this data will tell me what the vehicle after the Ferrari is, so no point in trying to model yellow->red and slow->fast and soot->clean.
Oh and one last thing — Numerai does not reward participants who stake on the tournament with a kickback from the Hedge Fund, or with a real fiat currency at all (they used to). They have their own cryptocurrency, which can be exchanged for fiat currency. You do not participate in the Hedge Fund directly at all. This is a common misconception. Mmmmm, maybe my next article will consist of a list of common misconceptions about Numerai.
To end, for me this is an exciting platform. I doubt I will make money, but I have no doubt I will learn a lot. It combines the ‘wisdom of the crowd’ theory with ‘skin in the game’ motivation to build a model of the market to essentially guide a hedge fund. Participants get clean feature/target data so they do not have to spend a lot of time cleaning it up themselves. I think this is a smart idea. Payouts in cryptocurrency sour me a little — but honestly not too much. It will be exciting to see where this goes.