Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.
> I thought the supreme court said the tariffs are illegal without congressional approval
The ruling wasn't as generalised as that. What it said was that the president doesn't have the authority to impose tariffs under the provisions of the International Emergency Economic Powers Act, which Trump had cited as the basis for some tariffs. That made those tariffs effectively illegal, but the ruling said nothing about tariffs which had other legislative provisions as their legal basis. The tariffs that you see today cite some other rule as their basis rather than the IEEPA.
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
I think this is the only sensible way to work with agents, if you care about code quality and reliability but still want the benefits of AI. There seem to be three camps that people more or less fall into: (a) AI is terrible/bad/evil and should never be used, (b) you should one-shot everything and be happy if it seems to "work" when you try it, (c) the middle ground, where the AI writes code which you carefully review.
I definitely prefer (c). But I get why (b) can feel necessary. If your competition is using (b) there can be pressure to do the same just to keep up.
I think c is the only way it can sustainably work. The idea of b, that software is running and nobody there knows how it works, doesn’t seem like a good foundation for a business to run on.
My personal challenge is that if c, generate then review, isn’t pretty close to a one-shot then I am almost certainly negative for time versus creating from scratch (accounting for over-documenting, prompting, and enforced pauses for generation).
The sunk cost fallacy bites and then bites again and again.
I think it must depend at least partially on the task, too. At the extreme, there are things where you won't care beyond "it seems to work" because you only needed it to run once and it got useful results.
Yes, but all the "don't care" stuff needs to be inside a "don't care" module. For example, I don't actually care about GUI code. I just need "pressing this button emits this event or callback etc". If the GUI module interface is well defined then it doesn't matter how it's implemented as long as it works.
The problems start when you don't have clean separations. If your GUI code is also implementing ad hoc business logic like policies and workflows etc then you won't know what you care about and what you don't.
So you need to build little ring fenced enclaves where you can say "do whatever you want to implement this interface with this behaviour". If it gets fucked up you could just throw away the whole module and start again.
Exactly. I think this is it. Clearly separate what must be perfect and well understood (security, business logic), from bits that you are happy to throw some code at and see if they work. And don’t let the latter get its hands on the former. I think going back to OOP principles and SRP/separation of concerns can make this work. Just some conscious planning needed rather than handing over the reins entirely.
You've selectively quoted the article. The full quote (emphasis added):
"When Moore’s Law slowed in the mid-2000s (specifically, single-threaded performance stagnated), we suddenly had to think about parallelization, architecture, memory locality, etc."
Your link is talking about transistor count. The article is talking about single-threaded performance. Today's CPUs are faster in large part because they have more and more cores.
I disagree with this. Unlike training models (which requires huge compute), harness development is available to anyone with an editor and ideas. That means that solo devs and small startups can still make meaningful progress.
Also, having only a "standard implementation" makes no sense for a harness. A standard implementation would need to try to be as good as possible at all things. But you'd often want a specialised harness designed for exactly your use case.
While it depends entirely on personal opinion, I believe that it helps significantly. It helps build up the problem/solution from the ground up and he goes through each step out loud to help a student understand. I get that some people hate it and think it is so childish for such simple steps. For other people, it helps a lot. Learning on KA works best if started aat a young age and done consistently (you get used to the style and you learn everything from the ground up).
In my case, it's not so much that I find it childish, it's that I find it incredibly distracting which means I can't focus as easily on the content. Instead of "oh, now I understand how that equation works" I catch myself thinking "why is he repeating the same word over and over - that's so annoying!". Then my brain starts obsessing over "when is he going to do it again?", even though I don't want it to. It's a bit like trying to deliberately ignore something. It becomes all you can think about.
Oh ok, got you. While it entirely differs on how different people receive ideas, for me, I just try to keep the end goal in mind (what are we ultimately moving towards) and just think of those repeated words as just small steps rather than just annoying repeating word.
OTOH creating neural connections takes time. It's a real distance. Small for us, but great for the cells. That's why explaining difficult concepts too fast, while cool for people getting things instantly, is not good for general population.
I very much agree and it saddens me often when I see folks dial it up to 1.5x or 2x "because it is boring otherwise". They might think they just get as much, only in half the time, but I'd hypothesize that they retain less because the mind needs some time to digest.
Similar with the trend to auto-cut pauses that speakers make, just to cram in more text into a youtube video. Destroys not just the pacing but any chance for the mind to take its time.
There’s a great lecture on giving lectures by an MIT prof. He says you can give no more than 3 ideas per hour, and you must repeat yourself at least 3 times for each idea. And while, part of his thesis is because people might miss something due to their attention, and needs of the group vs individual I think his experience is still highly relevant. I know that I don’t learn as much unless there’s room for almost boredom. I find there is a feeling similar to boredom that I have to experience to really internalize something. It’s almost like I can feel the neurons rewiring (though I suspect it might just be an emotion that encourages the energy expenditure necessary for learning). If I skim, or do doubletime, then that feeling usually never comes, and there’s a good chance my future recall will be poor…
When I’m understimulated by a teacher, my mind wanders. I’ll start thinking about something else entirely. When my mind comes back, I’ll have missed a couple minutes of content. I can overcome this with willpower, but doing so is exhausting.
Playing videos at 1.5-2x fixes the problem entirely. My attention is held without any effort.
If a concept is new or difficult enough, I just slow the video back down or pause it whenever I need to think things through.
I don't think you can make blanket statements. Consider:
1. Teacher 1 speaks at 100 words per minute.
2. Teacher 2 speaks at 50 words per minute, for the exact same content as Teacher 1.
You're not really losing anything by playing Teacher 1 at normal speed vs Teacher 2 at 2x speed. For every learner, there will be an optimal teaching speed for a given topic. This may mean listening to some audio at normal speed, some at higher speeds, and some even at slower speeds. I always customise my playback speed to the specific content I'm trying to learn.
Those two teachers will rarely say the same words. Chances are, teacher 2 chooses their words more carefully, and gives you then space to absorb that higher information density. In the end, both may communicate the same amount of information.
It's not about the rate of speaking but rather about what is actually being spoken. I am currently going through the Neetcode DSA course and it's just obnoxious. His video on Union Find, a 15 LoC algorithm, is 20 minutes.
I'd argue (as the article does), that you don't 'really' learn from watching stuff, fast or slow, but from actually working with and doing. This is completely obvious in the physical domain. You could take a ten year course on the basketball free throw and watch a billion hours of video on it, but then when you get on the court and actually do it - you're going to shoot pretty much like somebody who's never shot before.
And I think the same is true in academic domains. If you're trying to think about something consciously, you're going to do a poor job of it. And the only way to get the unconscious competence is through a bunch of 'doing'. Chess is a great example of this because it's purely mental yet again you're never going to gain any degree of significant competence without playing a lot. Interestingly this even applies after mastery. A master who has played e.g. 1. e4 their entire life and then swaps to 1. d4 isn't going to be playing at the same level as he plays 1. e4 even if watches thousands of hours of videos on it beforehand. That connection you make by doing and interacting with things is just critical.
It is a spectrum. If you give your mind some space, it will start churning through the key ideas, fill in gaps, turn things upside down, ask questions. That starts during the explanations and their pauses. And ideally you have some empty time right after to think about it, maybe do an exercise, maybe try to explain it to somebody. Work it, like a good dough.
2x listening suffocates this process. Like too little sugar or too low temperature or too little rest for your yeast dough. It might "feel" good but you are missing out so much...
I remember reading a study a while back about learning from videos at 1.5x to 2x speed. The key observation that actually made the learning more efficacious was to watch the video at 2x speed two times back to back. Apparently, this method was also better than watching the video one time at normal speed. I suppose it's because one get twice the exposure for the same amount of time.
I haven't looked for the study, but I can if you are truly curious.
I think there was a study omparing watching an educational video in 1x speed and watching it two times back to back in 2x speed equating the time spent on the material, which found the second group faired better.
Though it's not the question I asked. Would the second group fair somewhere close to the first if it only had listened once?
(I don't buy the argument that this would not be fair since they would use the extea time to watch more stuff ... because that's rarely what people do with their extra time. Especially not watching the same thing again.)
Ironically, spaced repetition is the key to successful learning.
Obviously one can overdo it. And there is variance in what people require. But not giving things time, instead giving in to the dopamine-craving, that's killing your learning abilities.
Repeating a sentence twice in a row during an explanation, is not spaced repetition at all. Spaced repetition means testing recall at increasing intervals, usually starting 1 day apart.
"Paris is the capital of France. Paris is the capital of France" -> not spaced repetition.
"Paris is the capital of France. [one day later]. What is the capital of France? [three days later] What is the capital of France?" -> spaced repetition.
And in fact, it's this plus practice where the real value lies in learning and memorising. The only purpose of an explanation from a teacher is to help you understand the concept. Generally once you understand it once, you don't need to understand it again - unless you forget, and that's where the practice / spaced repetition comes in. That's why I prefer the teaching part to be as efficient as possible, so I can focus my limited time on the parts that will make me internalise it.
I originally learned linear algebra from Gilbert Strang's OCW videos, and I was only able to stick with them by 2x'ing the playback rate; at this point, hearing Strang at his normal speaking rate sounds to me like hearing him after a Robitussin bender.
I wish YouTube let you increase the speed of videos past 2x. I play almost everything at 2x now, and I have no problem comprehending videos at that speed. I’d like to try 2.5x or 3x but the ui doesn’t have that option.
There's a bunch of different chrome extensions that do this, and you can do it yourself from the Chrome console with `$0.playbackRate = 3` or whatever.
It might be annoying if you already know the content. Frame it for the least common denominator (pun intended) - the person who has the time and needs the repetition to pick up a concept.
I've not tried it myself, but I consistently hear good things about Math Academy. It uses a knowledge graph so that it knows whether you've satisfied prerequisites for a topic, and it adapts itself to your level. I've never taken the plunge because it's very expensive at $49 a month, but it seems to be the one that serious maths learners use.
I did the trial of Brilliant the other day. You can get 7 days free, then another 7 days added if you cancel the first trial before the end. After that you can continue to get 2 free lessons per day with ads. I think it's about $20 a month for unlimited lessons. The maths problems are more visual (which makes them quite intuitive) and it has a nicer, more polished UI than Khan Academy. It's easier to stay in the mood for doing more lessons. The main downsides compared to Khan are that the explanations are quite minimal (short descriptions, no videos), and it doesn't reach as advanced a level as Khan. I'm looking at it as more a way to do a refresher on the maths I already learned at school rather than a serious learning tool. If it were $5 a month I'd probably consider subbing for a couple of months and trying to blitz through all the material before switching back to Khan to fill in the gaps.
Not sure that I'm adding anything to this, or that what I'm saying is even relevant, but Sal does (or did) have a deviated septum, and as someone who suffers from the same / similar root cause of breathing issues, I'm pretty sure speaking is (or was) taxing for him, probably contributing at least mildly to such a style.
I find it difficult to understand people who are wildly skeptical about LLMs leading to AGI (assuming we can even agree on what that means). Consider:
- They can already reason better than many humans and are still improving all the time
- Harnesses are improving all the time
- We're already exploring things like long term memory, long term goals, and other things that humans have which LLMs traditionally lack
- An AI agent can read and reason about every piece of AI research ever published, including looking for insights that humans may have missed. A team of humans could never do this even if they dedicated their whole lives to it.
- They can design and execute experiments on a mass scale to determine what does and doesn't work
- Large AI labs have more than sufficient resources and motivation to throw at the problem, and are in fact doing this.
So you believe LLMs (despite their inherent deficiencies vs EBMs [0], etc.) can lead to what you'd consider AGI, but you also admit that there is no agreement what AGI actually would be and you further don't provide your own definition? But you are surprised that some (like e.g. Yann LeCun (I am very convinced by his published works beyond his authority in the field, but am willing to admit his could be seen as a biased position)) are skeptical?
If you provide what you'd consider AGI, we may not agree on that definition, but I and other skeptics could at least discuss with you whether A.) that seems reasonably achievable given LLMs inherent limitations and B.) whether any of what you'd listed is actually likely to get us there.
As it stands, neither is possible without knowing what you believe AGI to be, but for what it's worth, coming from someone who both does see LLMs as valuable tools but whose definition for AGI also contains, among other things, reliable self-assessment of factual uncertainty [1] and basic counting and grade school maths [0][2] without tools or eternally scaling training data, I have yet to read any evidence that LLMs can achieve my, rather strict, metric for AGI.
These models are amazing tools, their ability to leverage massive amounts of high quality training data to further sciences truly awe inspiring, but that does not mean intelligence, at least in my definition that requires some internals these models have never been proven to possess. It's nuts that solving Erdos problems can be done by a model which struggles to count or solve a sudoku without external tools, but that's where the technology has been for years now and no paper I have read has shown that LLMs can overcome that to any scalable degree. You can push further with training data, but the limitations remain, albeit less noticeable. Any externalities, be it tools (self-scripted or called by the model), external memory solutions of all shapes and sizes, etc. I personally also feel cannot be required for or lead towards AGI as intelligence may be better leveraged by such externalities, but should never require them, so much of your suggestion I feel shouldn't be considered even if one believes LLMs can yield intelligence. I will admit that I am very extreme here though, this is not a position held by everyone for good reason. At the end I will always point towards the "extraordinary claims require extraordinary proof" of it all and that LLMs, in the face of any doubt, should be viewed akin to how Stockfish can play better than any grandmaster, but that does not mean intelligence, at least in my world.
If your definition for intelligence does not require basic arithmetics or an understanding of ones own knowledge gaps, then maybe LLMs can achieve that, but I'd push back on that truly rising to the AGI moniker. Maybe a more comprehensive or even my definition of AGI is possible whilst keeping the autoregressive nature after all, but there is no evidence supporting that by itself and quite a few things that haven't even begun to be overcome before something of that magnitude could be honestly considered.
It's akin to "let's colonise Mars by 2020 or 2030 or 2040 for sure, then terraform it" proposals. If that were possible, wouldn't we see a lot of these methods applied on earth and in a moon base long before (as in, we'd have had a permanent moon base in the early 2000s)? Same with LLMs, if they can truly yield AGI, we'd see some of the major deficiencies dealt with long before. The fact that we neither are terraforming earth, nor have any permanent off world colonies, nor have solved some of the listed, inherent limitations with LLMs by their design, that's what informs my skepticism that both are reasonably achievable in the timelines some industry "experts" (read hype merchants) propose on the regular. You tend to see some progress, a path toward solving actionable problems long before full implementation, at least in the real world...
> one thing is pretty clear: the act of familiarizing yourself with a language no longer matters
I completely disagree with this statement as a premise to the article.
Sure, if you have zero care about whether the agent's work is reliable or not, then there's no need to learn a language. But if you're a developer who cares about their work and isn't just outputting 100% vibe code, then at a minimum you should be familiar enough with the language that you can review the agent's output ,follow along with the code, and make an informed decision about whether to commit.
Interestingly, I've found that my approach to learning languages has fundamentally changed. Before, I'd study a book on the one or two languages which were important to me at that time. The goal was to become proficient. Now I choose to read about a variety of languages simply because they demonstrate some interesting paradigm that is new to me (e.g. Haskell -> functional, Elixir -> concurrency). I read a programming book like I'd read an engaging narrative non-fiction book: cover to cover relatively quickly, then I'm done. This gives me that "familiarity" which is useful for agentic coding, without bothering to learn every obscure bit of syntax or library call. My priority has become breadth (get an overview of many languages and paradigms) rather than depth (learn a language or two really well).
The author and his cited examples are folks with decades of experience shipping broadly used OSS so for them it’s probably a lot less of an issue to switch languages, even if the agent is papering over a lot of the details for them in the initial releases.
