I always starts with students by explaining how our intuition breaks in high-dimensions (spikiness, volumes,...) and how that carries when fitting/training models or searching optimization space.
It's a very important fundamental for modern data-science, to give one intuition about stochastic gradient descent, high-dimensional models, ... And this book starts with just that. I'm hooked. Thanks for sharing.
Data science is always a very overloaded term ever since it took off way back in the 2010s. One of, if not the most, durable definition of this that likely can also be the most valuable because it’ll probably land your jobs(even today) is being able to make decisions from looking at data that have an team wide(good IC like job security) scope at the least and company wide scope(very rich).
Building that intuition is incredibly difficult. It can be learned if one likes to solve and think about problems that way. Like for example you can get quite far with knowing how to use linear regression(for example coefficients of linear regression can be determined using a deterministic algorithm using linear algebra yet knowing the assumptions of linear expected value and constant or variance is more useful as is the knowledge of what probability model to use to define the random variable(hmm are these Bernoulli events or poison)).
How to do sampling(like using reservoir sampling when you have an infinite sample count e.g in a long running crowd sourced survey to not over or under sample buckets for calibration).
Or just rule of thumbs like how # of samples needed for moving decimal point on significance varies roughly as inverse of sqrt of N and probably much more in case of interacting factors.
I have heard it confidently stated, repeatedly, by extremely technical coder types, that "once we have enough data, we don't need Statistics"
It's a pretty funny sentiment.
It's also, unfortunately, reflective of how even talented individuals pass through advanced accreditation programs without locking blades with - or at least entering a general understanding of - fundamentals. Old timey British boarding schools would have called it "Logic" back in the day.
I am reminded once more of the differences in MSFS and Xplane: one uses a statistical table based model based on existing aircraft, and the other uses laminar flow analysis, fluid dynamics, and physics. One of them tells you something about unknown aircraft, and the other most definitely does not.
I’ve had the good fortune of taking two courses at Columbia in the Social Sciences department(Andrew Gelmam and Ben Goodrich teach there). I think they probably are right up there if not the best at trying to teach students how to work practically with Statistics(specifically Bayesian statistics). Though they have always lamented that most schools do a poor job of teaching it such that kids can apply it.
In my experience Data Science looks very little like it used to a few years ago, and the priority skill these days is good strong understanding of the basics and very good sense of judgement. To me, statistics is the absolute number one priority for any data scientist. You need to fully and deeply understand just basic concepts in statistics in order to translate what you see into action and do what you’re really there to do which is to prevent screwing up and acting on the wrong information or what’s more likely the wrong interpretation of the information.
For me the most valuable skill I have is a lot of experience applying and learning about Bayesian statistics: what it is (the beginning parts of Jaynes Probability Theory were not useful practically but deeply significant in helping me understand what it means and where it comes from), seeing lots of probabilistic models in the wild, playing around with them in both personal and professional worlds. Some people play video games, I love building hierarchical models. The nice thing is that in addition to it being very expressive It’s also just so much easier, such an intuitive way to avoid footguns because it just requires you to conceptualize one small bit at a time. When you’re done you get the inference for free with lots of charming stops along the pareto frontier between rigor and compute. Variational inference, expectation maximization, EM, Laplace. You can understand all of them with just a few concepts. Plus marginalization is just so unbelievably elegant to me. What is so surprising and beautiful to me is that Bayesian inference and marginalization are so useful and practical today. That being said there are plenty of unintuitive surprises, which is also a plug to not just understand the math but the theory and fundamentals to know how to interpret what you’re doing and seeing.
Also again this is still a great guide with lots of super important stuff (SVD/PCA linear algebra and linear regression (so much reward from just understanding linear regression from multiple perspectives)), no doubt. But if you really truly understand the basics you don’t need to worry about graph Laplacians (though highly highly recommend it’s also beautiful). Because more and more you can outsource the question of which method is ideal to a deep research agent that will read and understand arxiv for you. But you still have to audit it which means just really understanding the fundamentals is so crucial nowadays.
That and valuing speed and practicality. Strongest discriminator between someone junior and someone senior is recognizing when to reach for something simple and when you need to bring out bigger guns.
I think the biggest phase transition happened roughly fall of last year with AI adoption. My entire job is now AI orchestration, and then a LOT of my time spent writing the specs/prompts, reviewing and validating the output imperfectly and generally being very paranoid. But it's to a point where no one I know really has a choice but to work like this because doing things by hand is just way slower, even though I find slop to be like nails on a chalkboard. And the "sloppiness" has been getting progressively less sloppy. Reading Google Docs generated by Opus 4.8 is torture, whereas 5.6 can actually write concisely and clearly. So I expect a lot of the paranoia and pain to gradually decrease over time.
That being said, domain knowledge is still very important but you just don't need nearly as much tribal knowledge (because RAG or a tuned LLM can wrangle the chaos of your company documentation) or broad technical knowledge (because if you understand stats and PCA congratulations, when the coding agent recommends probabilistic PCA you now get to learn what it is, why its good for your use case (maybe), and get the gist fairly quickly; you no longer need to know about it in the first place. That was the return on investment for breadth of technical knowledge: knowing what to even look into).
I won't say "the field has moved towards a bigger focus on fundamentals" though -- this is all just one internet person's opinion. No one is telling me this is happening I'm just realizing that these are the muscles I am flexing more nowadays and my deep breadth of technical knowledge muscles are not flexing as much any more. Like: you are now inundated with recommendations of various methods from a coding agent: you need to understand them quickly; fundamentals gives you that. At least I think so, but of course, maybe I'm wrong about this due to misunderstanding the actual skills involved in what I'm doing now.
It is in the same vein but more applied to engineering and sciences. If you don't know him, I definitely recommend checking out his youtube channel, his lectures on Physics-inspired neural networks are top notch.
I always starts with students by explaining how our intuition breaks in high-dimensions (spikiness, volumes,...) and how that carries when fitting/training models or searching optimization space.
It's a very important fundamental for modern data-science, to give one intuition about stochastic gradient descent, high-dimensional models, ... And this book starts with just that. I'm hooked. Thanks for sharing.
See this older hacker news thread as well: https://news.ycombinator.com/item?id=45116849 A Random Walk in 10 Dimensions (2021)
Almost orthogonal vectors is a critical concept to understand for machine learning.
Data science is always a very overloaded term ever since it took off way back in the 2010s. One of, if not the most, durable definition of this that likely can also be the most valuable because it’ll probably land your jobs(even today) is being able to make decisions from looking at data that have an team wide(good IC like job security) scope at the least and company wide scope(very rich).
Building that intuition is incredibly difficult. It can be learned if one likes to solve and think about problems that way. Like for example you can get quite far with knowing how to use linear regression(for example coefficients of linear regression can be determined using a deterministic algorithm using linear algebra yet knowing the assumptions of linear expected value and constant or variance is more useful as is the knowledge of what probability model to use to define the random variable(hmm are these Bernoulli events or poison)).
How to do sampling(like using reservoir sampling when you have an infinite sample count e.g in a long running crowd sourced survey to not over or under sample buckets for calibration).
Or just rule of thumbs like how # of samples needed for moving decimal point on significance varies roughly as inverse of sqrt of N and probably much more in case of interacting factors.
I would like a book on that :)
The discipline you are looking for is called Statistics
I have heard it confidently stated, repeatedly, by extremely technical coder types, that "once we have enough data, we don't need Statistics"
It's a pretty funny sentiment.
It's also, unfortunately, reflective of how even talented individuals pass through advanced accreditation programs without locking blades with - or at least entering a general understanding of - fundamentals. Old timey British boarding schools would have called it "Logic" back in the day.
I am reminded once more of the differences in MSFS and Xplane: one uses a statistical table based model based on existing aircraft, and the other uses laminar flow analysis, fluid dynamics, and physics. One of them tells you something about unknown aircraft, and the other most definitely does not.
I’ve had the good fortune of taking two courses at Columbia in the Social Sciences department(Andrew Gelmam and Ben Goodrich teach there). I think they probably are right up there if not the best at trying to teach students how to work practically with Statistics(specifically Bayesian statistics). Though they have always lamented that most schools do a poor job of teaching it such that kids can apply it.
[dead]
In my experience Data Science looks very little like it used to a few years ago, and the priority skill these days is good strong understanding of the basics and very good sense of judgement. To me, statistics is the absolute number one priority for any data scientist. You need to fully and deeply understand just basic concepts in statistics in order to translate what you see into action and do what you’re really there to do which is to prevent screwing up and acting on the wrong information or what’s more likely the wrong interpretation of the information.
For me the most valuable skill I have is a lot of experience applying and learning about Bayesian statistics: what it is (the beginning parts of Jaynes Probability Theory were not useful practically but deeply significant in helping me understand what it means and where it comes from), seeing lots of probabilistic models in the wild, playing around with them in both personal and professional worlds. Some people play video games, I love building hierarchical models. The nice thing is that in addition to it being very expressive It’s also just so much easier, such an intuitive way to avoid footguns because it just requires you to conceptualize one small bit at a time. When you’re done you get the inference for free with lots of charming stops along the pareto frontier between rigor and compute. Variational inference, expectation maximization, EM, Laplace. You can understand all of them with just a few concepts. Plus marginalization is just so unbelievably elegant to me. What is so surprising and beautiful to me is that Bayesian inference and marginalization are so useful and practical today. That being said there are plenty of unintuitive surprises, which is also a plug to not just understand the math but the theory and fundamentals to know how to interpret what you’re doing and seeing.
Also again this is still a great guide with lots of super important stuff (SVD/PCA linear algebra and linear regression (so much reward from just understanding linear regression from multiple perspectives)), no doubt. But if you really truly understand the basics you don’t need to worry about graph Laplacians (though highly highly recommend it’s also beautiful). Because more and more you can outsource the question of which method is ideal to a deep research agent that will read and understand arxiv for you. But you still have to audit it which means just really understanding the fundamentals is so crucial nowadays.
That and valuing speed and practicality. Strongest discriminator between someone junior and someone senior is recognizing when to reach for something simple and when you need to bring out bigger guns.
In your opinion how did it look just a few years ago, and why do you think the field has moved towards a bigger focus on fundamentals?
I think the biggest phase transition happened roughly fall of last year with AI adoption. My entire job is now AI orchestration, and then a LOT of my time spent writing the specs/prompts, reviewing and validating the output imperfectly and generally being very paranoid. But it's to a point where no one I know really has a choice but to work like this because doing things by hand is just way slower, even though I find slop to be like nails on a chalkboard. And the "sloppiness" has been getting progressively less sloppy. Reading Google Docs generated by Opus 4.8 is torture, whereas 5.6 can actually write concisely and clearly. So I expect a lot of the paranoia and pain to gradually decrease over time.
That being said, domain knowledge is still very important but you just don't need nearly as much tribal knowledge (because RAG or a tuned LLM can wrangle the chaos of your company documentation) or broad technical knowledge (because if you understand stats and PCA congratulations, when the coding agent recommends probabilistic PCA you now get to learn what it is, why its good for your use case (maybe), and get the gist fairly quickly; you no longer need to know about it in the first place. That was the return on investment for breadth of technical knowledge: knowing what to even look into).
I won't say "the field has moved towards a bigger focus on fundamentals" though -- this is all just one internet person's opinion. No one is telling me this is happening I'm just realizing that these are the muscles I am flexing more nowadays and my deep breadth of technical knowledge muscles are not flexing as much any more. Like: you are now inundated with recommendations of various methods from a coding agent: you need to understand them quickly; fundamentals gives you that. At least I think so, but of course, maybe I'm wrong about this due to misunderstanding the actual skills involved in what I'm doing now.
[flagged]
[dead]
Related: Steve Brunton is also releasing a new book soon https://x.com/eigensteve/status/2055341831702057189
It is in the same vein but more applied to engineering and sciences. If you don't know him, I definitely recommend checking out his youtube channel, his lectures on Physics-inspired neural networks are top notch.
Related book by Blum, Hopcroft, Kannan:
Foundations of Data Science (2020):
https://home.ttic.edu/~avrim/book.pdf
It's a good book on what it is about but Foundations of Data Science it is not.
It is very narrowly focused on the authors' research interest around theoretical results on spectral approximations.
I find their work very interesting but it certainly does not teach you the foundations you need to know to do data science
That is very cool.
Anyone knows how can I compile the latex sources into an epub?
pandoc is probably the easiest.