Blog ›Rambling about LLMs

LifeTechnology

Translated from Korean by Claude · Read the original

Cars, cell phones, and now LLMs. I'd call them inventions that completely changed the world. But LLMs look like they'll change the world more than anything else.

Since an LLM is a machine that, given some text, picks the most fitting next word, I thought it would have obvious built-in limits. But apparently if you feed it enough training data, it can solve most algorithm problems this way too. There's probably an algorithm for computing the NNth digit of π, but if you just memorize all of it, you don't need that algorithm for smallish NN, sort of.

Personally, even up until 2025 I don't think I'd ever even considered building a product by relying on AI, but now the amount of code I write by hand has dropped sharply in just a year. There seems to be a lot of worry about whether this amazing thing might take our jobs. So I decided to jot down, in no particular order, whatever comes to mind about LLMs.

Leverage

In the stock market, leverage, simply put, means something that tracks a particular stock at N times. For example, if KOSPI went up 4% today, a 2x leveraged KOSPI product went up 8% today.

I can't think of a better analogy for LLMs. I think LLMs put leverage on an individual's abilities. You probably know well that even with the same LLM model, the quality of the result differs depending on who's using it.

Say you want to design a website. Typically you'd end up with something like this.

A website designed by AI

It's flashy, but it looks way too ‘AI-ish’. Doesn't it feel like less care went into it? This is the kind of design people call ‘AI slop’.

If a product that wants to attract users were designed like this and launched, that'd be a bit of a shame. The biggest problem is that people won't see this kind of design as ‘made with care’, and if someone launched it as is, they probably weren't even aware of that fact. Having never done design, they'd never have received feedback on bad design, and naturally they'd have a much weaker sense of ‘good design’ than someone who has. This is one of the reasons the results differ even with the same LLM.

This is honestly something I think about a lot of AI slop: just roughly piecing together a UI framework might have produced something better looking. Leverage works in the negative direction too, after all…

The experience and context a person picks up through life or work are a huge asset that LLMs don't come with built in. From my own experience working with LLMs, I felt that people can handle ‘context’ more effectively than LLMs, and came to think that the ‘context’ each person has about their work is the decisive factor that makes results differ even with the same model.

‘Context’ here can be many things: knowledge directly related to the project you're building (like a spec), knowledge from the field the project belongs to, seemingly useless trivia, things you've experienced and felt in life, and so on. All of it counts.

Personal experience

Tag ratings

I'd like to share the story of how solved.ac recently optimized tag ratings.

solved.ac is, simply put, a problem bank, and each problem has a ‘difficulty’ and ‘tags’ indicating its type. These values are contributed review-style by people who solved the problem, so difficulties and tags can change over time.

A ‘tag rating’ is a metric showing how good a user is at a certain type of problem. Here's how it's calculated: within a tag, take the 50 hardest problems the user has solved and add them up. Previously, to compute this, every time a user solved a problem we fetched all the problems they'd solved along with their tag info and diligently computed the rating. Since the user with the most solves has solved around 17,000 problems, this was the main culprit slowing the site down.

While reimplementing the site this time, I asked an LLM to optimize tag ratings. I explained the situation above and added the instruction: “Difficulty is an integer from 0 to 30, and there are about 200 tags. Please optimize this by thinking of it like an algorithm problem.” It told me to build a cache resembling a 3D array that stores, per user and per tag, how many problems were solved at each difficulty.

The optimization the AI suggested

Surprisingly, it was exactly the same structure as the cache table we were already running, and it didn't help much. Users × tags × difficulties is 200 000×200×30=1 200 000 000200\,000\times 200\times 30 = 1\,200\,000\,000 (1.2 billion) by simple arithmetic, and if you make that an SQL table, the queries themselves become very expensive. That was precisely why tag computation was slow in the first place.

In reality we only showed ratings for tags with 500 or more problems, so the number of tags was about 15\dfrac{1}{5} of that, but it still wasn't a small number. Even stored sparsely there were 47 million records at the time of the version upgrade, and the list of tags with 500+ problems itself could change, so there was more to consider.

Since I know solved.ac better than anyone, I was able to quickly find a compromise taking solved.ac's various circumstances into account.

  • I could work out that since ① a single person solves at most 17,000 problems anyway, giving each bucket 2 bytes lets us store up to 65,535, so we can cache how many problems of each difficulty were solved within 64 bytes per tag.

  • I also knew ② that ratings for tags like ‘arithmetic’ don't mean much, and that rating ‘BFS’ and ‘DFS’ separately when there's already a ‘graph theory’ rating is somewhat unnecessary, i.e., that some of the tags currently getting tag ratings don't really mean much.

  • So if I hand-picked just a few tags to compute tag ratings for, and if there were 32 or fewer, it'd fit in 2 KiB (= 64 B × 32) per person, and the math showed that ③ at 200,000 users, 391 MiB fits comfortably in solved.ac's Redis capacity,

  • and I got as far as concluding that if we also bitmasked the problem × tag table and handled it in Redis, we could count by user × tag × difficulty blazingly fast without any JOINs in the DB.

As a result of this optimization, solved.ac now computes tag ratings on problem solve in ~70 µs instead of ~600 ms, an improvement of about 8,000x, and it also became possible to update users' tag ratings in real time when a problem's difficulty or tags change, which we couldn't do before.

I don't think the AI couldn't have found the bolded parts. Given time, it seems perfectly capable of analyzing them. But I think figuring out those facts and combining them into a conclusion without background knowledge would have taken a lot more time (and token usage) than with it. AI can do everything, but time and tokens are finite, and I think personal experience still shines in this way.

Of course, once I proposed this solution, it was the LLM that did all the implementation. It implemented it very quickly and correctly, which really surprised me. It was already last year that an LLM all-solved the ICPC World Finals problems and beat the top human team, so I think we can now trust it to implement things well without many mistakes when the plan is already properly laid out.

AI proficiency

With LLMs creating such big synergy, using them well obviously matters. I think it's worth knowing what LLMs are good and bad at, and how to make them do a better job. I think learning by actually using LLMs is the most effective way. Ask it to build something you couldn't build yourself, hook up MCP and have it resolve feedback that came in by email on its own, find out once whether this buddy made some huge mistake and wiped your DB (…), and so on. You need to get to know what kind of tool this is.

One thing I've found is that LLMs aren't good with visual stuff. It's especially true for design, and it stands out more when you give it an image and ask it to reproduce it exactly. It looked like it was diligently describing the image in text and then implementing from that text alone. Which makes sense. I felt that LLMs making plausible-looking UIs isn't really because they can visually look at a UI and judge it aesthetically, but because they were well drilled by rote learning. So I don't use LLMs when I need a design that breaks out of the mold.

On the other hand, they seem to have a deep understanding of the domain knowledge of many widely used technologies, so getting into a new domain has become much easier for me than before. For example, with an LLM's help I was able to bring Terraform into my work right away, and in the process I realized just how badly I'd been managing infrastructure all this time.

That said, watching how the world is going, I get the feeling that in some circles AI proficiency alone is overemphasized. To say it again, AI is indeed a tool that can do everything, but the speed and cost differ by who's using it, and time and tokens are limited. Naturally, people who can produce greater results in the same time at a lower cost are preferred. Hiring people based solely on ‘how well they use AI’ is, I think, as absurd a decision as picking a 3x leverage investor over a 2x leverage investor without even looking at their returns.

Coding tests

As for why companies have shut their hiring doors so tightly, I think it's because right now even companies can't easily tell who a ‘good candidate’ is.

These days coding tests aren't done much anymore. From a company's perspective, if you interviewed every candidate, your engineers couldn't get any work done, so coding tests existed to filter out only the candidates with a guaranteed minimum skill level. It was about cutting costs. But now the cost of recording the computer screen, filming the screen with a camera so the hands are visible, and so on, has gotten too high. And the question of whether it's meaningful to ask about this kind of knowledge when LLMs solve every problem anyway has become another reason coding tests are fading out.

That doesn't mean you can bring every candidate in for an interview. You'd still have to screen somehow… but now that LLMs exist, it's become hard to tell what kind of people perform well. Now individual ability would have to be evaluated multiplied by AI proficiency, which means tests should probably let you use AI. Things are wavering right now, but I think it'll eventually converge in that direction. I also wondered whether renting out a whole contest venue and having people take coding tests on-site might work, but that doesn't seem very realistic.

From the candidate's side, won't it remain true that building experience related to your field is the most important thing? I think I'll keep advising people to build up lots of personal/team project experience if they want a job. Use LLMs a lot, crash some servers, publish your work and get feedback; I think you'll gradually learn how to do the same things faster and better.

Somewhat unexpectedly, I think studying algorithmic problem solving as a hobby (not as coding test prep) still has meaning in the same vein, from the standpoint of personal growth. I think developing problem-solving skills matters more than learning the data structures or algorithms themselves. Though if it's purely for getting a job, I'm not sure it's a cost-effective way to train.

Other random thoughts

  • There's a claim that LLM research and usage accelerate the climate crisis. It seems true for now, but personally I have a somewhat optimistic hope that over time LLMs will contribute enough to scientific research to offset all of it.

    • Image generation AI is a different story, and I don't think it'll be of much help to humanity.

  • I used to think there'd be some programming problems only humans could solve, but seeing how well they solve them these days, I wonder if design is just something they're bad at because they haven't had enough rote drilling yet. Either way, it's true they're not good at it yet.

  • Vulnerability analysis and hacking by LLMs are becoming more and more frequent, and I suspect it's just that humans have been writing shoddy code all along. Human slop is probably no less formidable than AI slop…

    • Scary times, but won't they pass soon?

    • I want to hack my own site, but Claude won't let me, so I'm debating whether to buy a computer that can run a local LLM.

On a separate note, LLMs are originally AIs that write text, yet funnily enough I rarely use them when I write. This post was also written entirely by hand. That said, if you're reading this in English, the translation was done by an LLM, so, well, that's a bit of a shame.

Comments

No comments yet.