I just returned home from DEFCON 34 after achieving 3rd in the CTF Finals with SuperDiceCodeLovers.
When I landed in Vegas, I had no badge but also didn't intend to spend all of my time on the CTF. Well, the opposite happened, and I ended up spending the first 2 days in Vegas essentially locked up in an Airbnb alongside 20+ other people trying to finish and improve our tooling as much as possible before competition start. This also meant learning how tooling I've never seen before worked and improving it in a matter of a few hours, which, needless to say, was not simple.
I was super fortunate that the competition start on the first day was delayed by an hour, because I wasn't able to complete a tool that I was working on called 'Polycule' until about 10 minutes after the original start time. Funnily enough, it turned out to be one of the biggest factors to our success over the weekend, in that it was a credential pool for our LLMs that handled request forwarding, rate-limit and usage observation, cool-down handling, and context-switching, which supported in placing 3rd in the finals on ~6 Codex 20x and 3 Claude Max subs (<$1500) — which, compared to a couple of the other teams, is chump change.
The second day of the comp was certainly our best; though we were absolutely bleeding points in the Attack portion of the A/D, living around 7th place in terms of points, we were dominating on KoTH and staying afloat in defense. The night of the second day, we spent an hour game planning how to pull through, and it was essentially dominate 2 of the 3 services or die. Our approach for this, given the way the competition had moved thus far, was to see how much we could tune/improve our usage of LLMs in the competition, since they had shown themselves to be the real competitor this year when tick speeds were so fast it left no time for human interpretation of challenges and instead just slop management. We noticed that if we wanted a new slopper to hop onto a challenge and/or had multiple sloppers working on the same challenge, that at certain points they were doing redundant work just to come up to the same conclusions. So we planned to utilize an existing communication layer that our captain @dudcom developed a few months prior called 'CCP' that essentially acts like a shared note-taking application that your agents can subscribe into and talk about any discoveries and methods they've used. Essentially a tool that the worker agents were encouraged to reach out to, to cut understanding cost and improve reasoning.
At least as far as the KoTH went, there were 2 modes of gameplay: discovery and hill-climbing. Many times you would see a team win the first tick and then start marginally improving their solution by ~.1 points; other times you would see huge jumps where the baseline across all teams was, say, 1000 points, and suddenly it jumps up (or down) by a few thousand. The former being hill-climbing and the latter, discovery. By first having the agents (let's assume 50 of them) set off un-coordinated to achieve some semblance of a scoring solution, it is likely that they developed their solution using different techniques, some being more efficient than others. Even if we get only 3-4 different methods out of them, we (the agents) can cache all of this to the communication layer and start efficiently hill-climbing the best methods to achieve the highest score and win the ticks.
Sadly, as day 3 started, we quickly got fucked in A/D and were made to feel like we were grasping air when chasing the backs of Blue Water in the KoTH with their unlimited LLM budget. I still thoroughly enjoyed the weekend, and competing here, despite the huge AI-reliance, made me feel the exhilaration of CTFs again. Not so much in the competition but in the preparation, tool dev, long nights, and abstract technical conversations.
My idea of what could potentially lead to a more "true" CTF next August would be to bring back LiveCTF, but under slightly different terms. Instead of 1v1 format, I think we could utilize all 8 CTF floor badges in an 8v8 LiveCTF. Given, say, 3 challenges in 1 hour across 16 people, where the potential outcomes are either 2-1 or 3-0, I think it could lead to pretty interesting new competition tactics as well as keeping alive the older event format. As far as KoTH goes, it really is difficult to see how we can possibly design it so that AI isn't the deciding factor. One method proposed by fuzzcom was to reduce tick speed to only 3 in the entire day, as well as incorporating a metric for grading the quality of the solution as opposed to solely the efficiency. It's an interesting proposal, but I personally don't see a way in which we can look at a solution and say it is objectively better than another one, especially in a way that LLMs cannot optimize for, unless perhaps the scoring metric was left undisclosed, in which case I'm sure many teams wouldn't be happy not knowing why their solution scored less than the next teams'.
Anyways, that was my DEFCON — lots of good meals and fun events such as a 6-hour Monopoly game introducing the notion of equity, which resulted in more than a few heated debates and lots of under-the-table scheming.
Next week I'll be heading to Korea University to study more lower-level system design as an exchange student and practice my Korean. Wish me luck ✌️