That o3 does well on frontier math held-out set is impressive, no doubt

I think there is plenty of room for doubt still. elliotglazer on reddit writes:

Epoch’s lead mathematician here. Yes, OAI funded this and has the dataset, which allowed them to evaluate o3 in-house. We haven’t yet independently verified their 25% claim. To do so, we’re currently developing a hold-out dataset and will be able to test their model without them having any prior exposure to these problems.

My personal opinion is that OAI’s score is legit (i.e., they didn’t train on the dataset), and that they have no incentive to lie about internal benchmarking performances. However, we can’t vouch for them until our independent evaluation is complete.

(emphasis mine). So there is good reason to doubt that the “held-out dataset” even exists.

Unfortunately “states of quantum systems form a vector space, and states are often usefully described as linear combinations of other states” doesn’t make for good science fiction compared to “whoa dude, like, the multiverse, man.”

How do you figure? It’s absolutely possible in principle that a quantum computer can efficiently perform computations which would be extremely expensive to perform on a classical computer.

i read the title and was like damn we’re dunking on game engines now?

Wait I know nothing about chemistry but I’m curious now, what are the footguns?

I read one of the papers. About the specific question you have: given a string of bits s, they’re making the choice to associate the empirical distribution to s, as if s was generated by an iid Bernoulli process. So if s has 10 zero bits and 30 one bits, its associated empirical distribution is Ber(3/4). This is the distribution which they’re calculating the entropy of. I have no idea on what basis they are making this choice.

The rest of the paper didn’t make sense to me - they are somehow assigning a number N of “information states” which can change over time as the memory cells fail. I honestly have no idea what it’s supposed to mean and kinda suspect the whole thing is rubbish.

Edit: after reading the author’s quotes from the associated hype article I’m 100% sure it’s rubbish. It’s also really funny that they didn’t manage to catch the COVID-19 research hype train so they’ve pivoted to the simulation hypothesis.

Mr. Costantino said the design was not at fault and that the towering mast, which stood 237 feet tall, had not created “any kind of problem.”

“The ship was an unsinkable ship,” he said. “I say it, I repeat it.”

- Designer of sunken ship