WHY A DATA MARKETPLACE
Among the many different snippets of visual memory, one kept recurring to me often over the past thirty years. As I walked from my Real Analysis class to my dormitory on La Loma street in Berkeley, lined with blossoming plum trees, I murmured, "life keeps getting more complex.". I was, of course, too young to have said this with full intention. The complexity I was contemplating could not have anything to do with my laptop, my e-mail account, my mobile phone, my checking account login, or my social media posts. I had none of those things! (Yes, I feel lucky to have had such an ideal environment to focus, read, and learn.)
"We live in a complex world. Complexity is increasing."
Over the years, as I cumulatively remembered this moment - what I mean is, I remember the epiphany as well as one of the prior times I remembered and thought about it– I’ve always smiled, waving a finger at that young boy: "you haven’t seen nothing yet!"
Overcoming this complexity seemed possible, and in most of my academic, software engineering, finance, and consulting activities, I have chased data to make sense of the complexity.
Some strange and before-their-time chases were:
- Early 80’s, solar insolation and illumination data collected from sensors in San Francisco
- Late 80’s, real-time radiation and load data to optimize energy use in a high rise
- Early 90’s, prepayment speeds of mortgages as interest rates were coming down
- Late 90’s, travel demand and equipment life data to trade used passenger aircraft
- Early 2000’s, default rates of companies that were recently rated A or higher by the S&P
- Mid 2000’s, volume of mortgage securitizations on the Wall Street and their ratings
- Early 2010’s, relationship between balance sheet health and stock appreciation
- Mid 2010’s, change that wind and solar brought to electricity prices in Texas and California
"We humans are hampered by cognitive biases."
My preferred line of attack to these problems were models. Sometimes very complex models. We scraped for data because what we needed was not collected through social media or curated by anybody in an observable fashion. We finally ended up deciding that a curated data marketplace is essential. To best understand how we decided that unbiased, comprehensive and curated data is just as important as good models, we need to understand biases that we have as humans. The best intentioned and most educated of us has these biases.
The human mind, left to its normal course of evolution, did not develop fast enough to handle the modern complex world. Concepts, events and markets were separated with little interaction and could be analyzed separately. Social and political trends, specifically globalization enabled by free market thinking, container shipping, e-commerce, and the internet, made things that were separated all come together and become related. Digitization of everything combined with globalization related everything to everything else. Secular analysis was no longer guaranteed to give correct results.
"We need rich, diverse, and unbiased data to feed our models."
"We can overcome complexity with quick insights derived from such data."
A few billion people who were never heard from before the year 2000 now participated and competed in the new unified marketplace. Incumbent participants felt their problems were now intractable. At no other time in human history has our collective experience changed so profoundly, so fast.
But humans are good at survival. To deal with the complexity and the information overload, the human mind takes shortcuts. Shortcuts are best explained in terms of cognitive biases (I will devote another post to biases.). However, these biases cloud our critical thinking and make clean and objective data even more essential.
Let’s summarize how we got to building this data marketplace.
- We live in a complex world. Complexity is increasing.
- We humans are hampered by cognitive biases.
- We need rich, diverse, and unbiased data to feed our models.
- We can overcome complexity with quick insights derived from such data.
Hence, we opened this data marketplace, altadata.io, for data users and data providers. Our site is new. Visit us often (or better yet, subscribe) to see new products and new marketplace features, especially on quick insights.
We are looking for data partners. The data that we prefer are data that impact many people.