Posts
-
Eight Years Later
CommentsSorta Insightful turns eight years old today!
I am writing this while traveling and sick with COVID, which is the clearest sign that I am in this for the long haul. I don’t remember where it’s from, but long ago I remember reading a novelist’s guide for how to write a novel. It had a bunch of typical advice, but ended with “sometimes the way to write a novel is to just go write a f***ing novel.” That’s where I’m at with this blog.
I have a bit of an ADHD relationship with writing, where I will go a long time without writing anything, then stay up until 5 AM revising a post. Getting started is hard but sustaining effort is easy.
Highlights
I’m done with writing MIT Mystery Hunt! Freedom!
The post about it was pretty rewarding to write, although “post” may be the wrong word. Credit to CJ for noting that the post was long enough to pass NaNoWriMo.
Did it need to be that long? Eh, probably not, but I kept having ideas I wanted to put in. I also knew I was not in a mood to revise it to be shorter. The revisions I did were focused on grammar, word choice, and overall flow, rather than deciding if an idea should be cut. (Feel free to draw a comparison to the Hunt itself if you want.)
To be honest, I do not expect to write a post of that length again. I started working on that post right after Mystery Hunt 2023 finished, and hardcore focused on getting it out ASAP. “As soon as possible” turned out to be 3 months, ending right when planning for ABCDE:FG ramped up. The end result was that I was exerting “write Mystery Hunt” levels of effort (10-20 hrs/week) for 1.5 years straight. That was just too much, and I burned out on doing anything with my free time besides vegetating.
My burnout lined up almost exactly with getting a copy of The Legend of Zelda: Tears of the Kingdom. Let’s say my escapism into TotK was especially therapeutic. I’m done with the game now, and am still adjusting to not having a “default” action taking up mindshare. “I should write a puzzle”, “I should write about writing puzzles”, or “I should go find more Koroks” have been like, the only three thoughts I’ve had outside of work since last year. Gotta have more thoughts than those!
Statistics
Posts
I wrote 6 posts this year, down from 7 last year. In my defense, this is a bit misleading, especially if I pull up the time stats.
Time Spent
I spent 189 hours, 45 minutes writing for my blog this year, around 1.9x as long as last year.
Around 170 hours of that was on the Mystery Hunt retrospective post.
View Counts
These are view counts from August 18, 2022 to today.
259 2022-08-18-seven-years.markdown 372 2022-10-01-generative-modeling.markdown 333 2023-01-20-mh-2023-prelude.markdown 1929 2023-04-21-mh-2023.markdown 114 2023-05-09-bootes-2023.markdown 153 2023-07-19-ml-hurry.markdownPosts in Limbo
Post about Dominion Online:
Odds of writing this year: 5%
Odds of writing eventually: 25%My priorities are moving away from Dominion. This may be the first year I skip the yearly championship, I’m not too interested in grinding my skill level back up. Still, I can’t let go of writing this eventually.
Post about Dustforce:
Odds of writing this year: 20%
Odds of writing eventually: 60%I still think Dustforce is one of the best platformers of all time. I’ve been playing a lot more Celeste recently thanks to Strawberry Jam, but Dustforce still does some things that no one else has replicated in a satisfying way.
Post about puzzlehunting 201:
Odds of writing this year: 50%
Odds of writing eventually: 90%This is a post I’ve been considering for a while. There are introduction to puzzlehunt posts, a so-called “puzzlehunt 101”, but there is no “puzzlehunting 201” for people familiar with the basics and interested in solving faster. I was going to write this last year, embedding puzzle content in it for Mystery Hunt, but didn’t have a solid puzzle idea and got too busy. Now I’m less busy.
Post about AI timelines:
Odds of writing this year: 90%
Odds of writing eventually: 99%I’m not planning a big update to the last timelines post. I just think it’s time to review what I wrote last time, given that it’s been 3 years. (And selfishly, because I think I called a lot of things correctly and want to brag about that.)
-
Machine Learning Got Itself in a Big Damn Hurry
CommentsThree months ago, I was at an “Intersection of AI and My Little Pony Fandom” panel. It was a panel about the ways the MLP fandom has used AI to generate creative work, starting from finetuned GPT-2 in 2020, through voice synthesis via 15.ai, and ending with, of course, Stable Diffusion. More specifically, the finetuned Pony Diffusion checkpoint, whose finetuning cost is estimated as tens of thousands of dollars. The talk ended with a proof-of-concept of a Discord bot that roleplayed a pony, via GPT-3.5, whose avatar was in-painted to diffe1ent expressions based on emotions inferred from chat history.
As I asked questions about compute resources and the presenter’s position on generative AI ethics, I had a moment of realization. I was at a pony convention. Why are we talking about whether an RTX 3090 is big enough to finetune a LLaMa checkpoint? How are we talking about Vicuna, here of all places, while people are dressed up in cosplay next door?
When people talk about technology improving more quickly, it usually evokes thoughts of the singularity. Technology indistinguishable from magic, making it easier to create more magic. But, culture and communication are technologies too. The much easier and less-speculative way for a field to move faster is by having more people working in that field. If research is an API call away, then congratulations, we’ve democratized ML, as long as you’re willing to pay for access. Combine that with the adoption of low-friction social media (aka Twitter), and you’ve got something going.
If the engine of invention is powered by people sharing random ideas until good ones emerge, then I can’t help but wonder if the best inventions are ones that make sharing ideas easier.
(Previous post about How to Invent Everything)
This field is just getting so big. Things change so fast! The MLP AI enthusiasts mentioned some pretrained LLMs that I had not even heard of. It’s not my field, but, like, I do this for a living. I was easily top 1% generative AI knowledge among bronies in 2020. Now I’m like, top 5%? A 5x increase sounds right. I can only attribute the growth to one truth: there are signs of life, and people are hungry.
I remember being a young whippersnapper, in the deep learning wave of 2015. Then I was the new guard. The old guard would complain that “You can’t just take an old idea, do it with a neural net, call it deep learning, and claim that part of ML for deep learning”, as researchers continued to take old ideas and reshape the field around a different MLP: the multilayer perceptron. Now people do the same thing with LLMs.
The flood is growing, and my time to drink from it is the same. Deciding what to drink is getting harder.
I’m noticing a trend of people posting LLM summaries of papers, talks, etc. They’re always attributed to the LLM, and they’re never fact-checked. I have a long history of learning and appreciating that the map is not the territory, and this trend is a bit like if people who loved maps got access to a map-making tool and created a billion maps. It is not even that they understand they’re working with approximations of reality. That would be better. What’s happening is that they don’t care they’re on an approximation of reality. If you like searching for truth, it is a very personal kind of hell.
The bitter lesson I’m taking is that I will have to get used to this. I will have to use an LLM where I can explain the loss function, but can’t explain the emergent phenomena. I will have to learn all the random tricks people have found for LLM prompting. There is going to be too much content to accept any non-augmented workflow. Such is the future of knowledge work.
“But why is this prompt so effective? Do we have a way to inspect what preferences the RLHF reward has extrapolated, given the preference labels we have?”
“The stuff is what the stuff is, brother! Accept the mystery.”
-
A Boötes Shaped Addendum to Writing Mystery Hunt 2023
CommentsI promise this will not be as long as the previous post, but I will assume you’ve read it.
First, when discussing calibration in Mystery Hunt, I wrote the following:
If we had ever revisited a Ministry puzzle around the middle of the year, when feeders were getting written in earnest, it would have been very obvious that our difficulty calibration was off. But why would you solve a puzzle from an old Mystery Hunt, when there are a bunch of puzzles to testsolve for the upcoming Mystery Hunt? The clock is ticking, after all. Maybe the answer is that yes, you should actually go solve 1-2 old puzzles in the same conditions as your testsolves to calibrate your ratings. If you ever do this, let me know, because I have never, ever seen it done or heard of it being done.
I’ve since had two people from Galactic tell me that they did this for the Students round in Mystery Hunt 2021. For their big testsolve of that round, 5 Students puzzles were not ready yet, so they took 5 random Fish puzzles from Mystery Hunt 2015 and put them in as placeholders. After the testsolve, they checked solve times and found their Students puzzles were only a little harder than the Fish puzzles.
So, someone has done this before! That being said, I’m not sure what Galactic would have done if it turned out their Students puzzles were a lot harder instead of a little harder. By this point they would have already had 90% of the round written, and I don’t know how much room they would have had to steer difficulty.
Now, on to the main reason I’m writing this post: teammate just announced Admiral Boötes’ Cosmic Discovery Expedition: Further Galaxies.
As the FAQ mentions, the puzzles in that puzzlehunt were originally going to appear in Mystery Hunt 2023. However, as a response to the Hunt going long, huntcomm decided to scope down the Boötes round during Hunt.
This was pretty controversial, since people wanted to see their work in Mystery Hunt, and cutting the puzzles meant that wouldn’t happen. When this decision was announced, team leadership asked everyone to keep it a secret until they had met with the affected authors and decided what would happen to the removed puzzles. Most authors were interested in running them as a puzzlehunt later in the year, so the secrecy stuck until we were ready to announce everything.
Knowing about Boötes made the “teammate should have run the AI rounds as a separate hunt” chatter interesting to think about. Congrats, you’ve (sort of) got your wish. I’ve been told that cutting all AI rounds during Hunt was on the table in huntcomm discussions, but was quickly deemed too drastic. Hopefully you can appreciate how hard it would have been to decide to cut 4 rounds in the middle of Hunt, remake all of endgame and runaround within a few hours, and explain why the already-manufactured coin had characters you’d never seen during solving.
There is the argument that more people would have solved the AI round puzzles if they were run separately from Hunt, rather than nuked by free answers. I think the flip side is that many fewer people would know the AI round puzzles existed. There are plenty of people who only do MIT Mystery Hunt and nothing else. As a reference point, I did four Mystery Hunts before I did my first P&A magazine.
We’ve approached this puzzlehunt from the standpoint of minimizing extra work needed to run it, while still preparing the experience we want teams to see. I’m especially curious what the uptake will be. We talked a bit about writing new rounds that would ramp up in difficulty to ABCDE, but quickly found there wasn’t enough interest to make that happen, so everything is going to be hard Mystery Hunt level. Is there appetite for puzzles this hard at an event outside Mystery Hunt? The closest analogue I know of is CRUMS Puzzlehunt, a hunt with 6 puzzles + a meta with every puzzle targeting Galactic Puzzle Hunt or Teammate Hunt difficulty.
Will big teams reconvene for this, or split up? Will we get new solvers that want to jump in the deep end? When will the first team finish? These are all hard questions to answer, because it’s really unclear what the solve power of teams will be and whether people will be available during a random part of June. One advantage of Mystery Hunt always happening on MLK weekend is that anyone who wants to attend has ample warning time to clear their calendar.
It’s all very unprecedented. I’m excited to see how people feel about it!