The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it
Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.
Does it check out? India and Pakistan both have nukes and keep semi-regularly fighting each other.
MAD "on paper" prevents either side from going far enough to provoke the other into using nukes, but even then it's fundamentally flawed because it works on the assumption that both sides are both rational and believes the other side to be rational, as well as that both sides understands the others red lines well enough.
Already Reagan realised that isn't necessarily true - after Able Archer '83, he realised that the Soviet leadership seemed to genuinely believe that the US might be prepared to carry out a first strike, and that Able Archer got dangerously close to convince them one might be imminent. It's one of the things he noted as a reason to get in the room with them and negotiate.
If you believe the other side is irrational (whether or not that is because you are irrational), and think they're about to strike, MAD turns from a deterrence into a reason to try to preempt to ensure you're the "least destroyed" by hitting harder, sooner.
Minor skirmishes are not war. China and India have border skirmishes with literal wooden sticks, neither side will use live ammo because they both have nukes.
Conveniently shifting to a different conflict, and ignoring that India and Pakistan have carried out missile strikes on each other, killed civilians, occupied territory.
Like I said, minor skirmishes. No war between India and Pakistan since they both got nukes in the 70s. Not bad for two states that have been hostile their entire existence.
That is a good optimization, but you still need to figure out the starting state at a given location, and for quite a lot of languages that is not solvable in the local case.
E.g. if you're looking at a page of Ruby, you can not in the general case know if it's inside or outside a quoted string, as the quote character can be any arbitrary character (and I really mean any. "% x " excluding the double-quotes is a valid Ruby quoted string where the quote character is space but you can e.g. pick a unicode codepoint you're not going to use anywhere else in the file)
So unless you parse from the start of the file, you're left with fuzzy matching, and that can be made good enough the vast majority of time.
Most readers are women, and the largest genre by far is romance novels, so book stores catering to women is a matter of chasing sales.
Otherwise fully agree with your points. The non-fiction market is easier to saturate as there's only so many unique angles on something, and people will look for something that feels authoritative rather than something fresh, while in fiction you're likely to read many similar premises because it's what you like.
Fiction writers also have to hustle a lot to be successful. The average full time writer in the UK would've made more at McDonalds in the UK, and I doubt it's much better elsewhere.
It's both an extremely competitive field, and it's incredibly hard to convince people to read your book even for those near the top of their field in terms of quality.
But non-fiction writers have an additional problem, that I think the article glosses over:
If I like, say, romance novels, I'm not looking for new books to be dramatically different to what I read before. A lot of genre readers - the by far largest genres are romance and thrillers - expect books that stick closely to expected conventions. Romance readers will form a mob with pitchforks if there isn't a happily ever after, for example.
Non-fiction, on the other hand, must set itself apart. Why should I read your How to Win Friends and Influence People look-alike when I can read Dale Carnegie's original?
It needs to be very different or clearly better and I have to find it in between the many hundreds of thousands (not an exaggeration; might even under-estimate the number at this point) of new non-fiction titles every year, where fiction titles are not:
You don't need 50 books on meditation, or 50 books on repairing your washing machine, but you might have 50 books in the same narrow sub-genre of fiction because it's what you like to read, and so the opportunity cost of picking up an unknown name feels very different.
A whole lot of the non-fiction market is also being cannibalised by blogs, tiktok, youtube videos and the like - and LLMs -, whereas a video is more likely to tease the fiction book rather than replace the need for it.
Also when I think of better end of non-fiction. How many books on certain topic I need? Say ornithology or on birds... Maybe a few at best. Unless I am collector or interested in that specific niche. Which already massively cuts down audience. Take history. Huge overall. But if you only go for specific events, periods or areas(town for example) it might cut down audience significantly.
So I think it is fair to say for fiction that some niches could be lot smaller say than generic fantasy, romance and such. And also I can read lot more fantasy books if I get around to it... Some of them are bloody enormous series too.
It's a sea change because a lot of this code looks plausible and seems to work to someone inexperienced at assuring quality, and it will take some time before the people emboldened by that learn.
I just yesterday had an agent produce a prototype that appeared to work and that looked plausibly well engineered, except I knew from experience that what it had done should be doable with a tenth of the codebase.
Someone without experience might've just accepted that prototype. It took one prompt to get it turned into a reasonable MVP, but you'd need to know what to ask, and you'd need to know you needed to ask.
Same. Claude "loves" writing tools to make checks deterministic so much that I actually have to pull it back sometimes and point out that some things need to keep using LLMs. But even then, I have it "farm out" a lot of things to scripts using dirt-cheap models for things I don't need a Sonnet or higher level model for as well. The proportion that runs on the SOTA models keeps dropping.
There are instances full of actual Nazis on Mastodon - that we can choose to refuse to interact with those instances (yes, I've blocked a nazi instance on my instance) does not prevent them from talking amongst themselves or to people who want to talk to them.
Censorship is not refusing to listen to people we find abusive. People are not entitled to force me to listen.
A federated platform is far more resistant to censorship than X is.
How is it not censorship if you are making the decision for not only yourself, but others, too? It sounds like saying bleeping swear words on TV isn't censorship because you just don't want to hear bad language.
People are choosing to be on instances which block certain things. That's their choice. I want to be able to have a conversation without filling it with racist screaming. Choosing to be in places that don't allow that is my choice.
If I refuse Nazis access to my house and other people chose to visit my house knowing I won't let Nazis in, I'm not censoring Nazis as I have no influence on their freedom of speech - they remain entirely free to speak elsewhere, to people who are willing to listen. I have no power to stop them from speaking, just to offer those who don't want to listen somewhere where they won't have to.
Nobody is forced to join a Mastodon instance that censors Nazis - there are many that don't, or you can run your own.
I never used either Twitter or Mastodon and don't quite understand what people use these platforms for. But I hear folks saying they are there because other people they know are there.
How would it work on Mastodon if you want to communicate with two people that happen to have each others instances blocked? Is it like having to use both Twitter and BlueSky or does it solve the problem more elegantly?
If that matters to you, you pick an instance that rarely blocks, and then you can talk to both of them seamlessly.
Occasionally some people are affected because they've picked a major-ish but somewhat block-happy instance, in which case they can either move their account (the experience for a move on Mastodon isn't great but it does work) or create a second one if it really matters to them.
Usually people will find out pretty early on if they're on an instance whose policies they agree with or not, so even if the account migration is jankier than it should be, it's rarely a major problem (in that instance - if instances shut down it's more problematic)
That's what we thought when we applied for .name, especially coupled with mail forwarding at doe.name so jane and john could get jane@doe.name and john@doe.name respectively.
In practice the demand was lacklustre.
We ran a webmail provider prior to that, with 60k domains or so, and the demand was higher for that, basically. To get a decent return out of it, you'd probably have to package it up with services on top, and our investors made us sell off the webmail service when we got .name... It turned out to be the more profitable of the two services. Doh.
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
reply