From what I read about the freelancers’ perspective, a lot of the newer gigs they get are to clean up work already done by AI instead of the opportunity to start new like they did in the past.
I'm just wondering (without really being an AI fanboy, but using it daily), isn't the whole challenge just more prompting, such as instructing the agents to analyze the codebase, propose refactoring approaches, etc? Maybe it's a bit more sophisticated prompting than a non-technical person would do but prompting nonetheless.
I'm asking because I have read these comments here and there about fixing the mess AI made. I know what mess it usually creates and it doesn't get better with the new models. But I also know that whatever is created by AI kind of works more often than not, and once something breaks, it's just one prompt away to fixing it.
In my experience, the problem is that once a codebase passes like ~50,000 lines, it's too big for AI to make good architectural choices (this is controversial, but I have yet to see evidence otherwise).
So what I do is mostly use the AI to understand things like architecture, encapsulation, deployment pipelines, types, migration structure, etc.
It's a lot of "Audit the callers of X and give me a report" as well as reading code. I'm looking for stuff that doesn't pass my smell test, especially relating type types/interfaces. I look at the AI recommendations, but they're only useful once I've scoped them down to more granular questions, such as assessing the interfaces between specific layers of a pipeline.
That process is about understanding the shape of the solution that exists and then trying to formulate what shape it should have.
Then it's a lot of identifying a very boring, vanilla architectural target and moving towards it in phases, using the strangler fig pattern pretty aggressively. I review every PR, but mostly at the level of architecture rather than every specific line.
The codebase size will be limited to the LLM context window size. Once the codebase grows larger than the context window, some of the codebase will have to be removed from the context and thus not be known to that LLM instance.
There's also a well documented phenomena [1] where LLMs struggle to recall information in the middle of the context.
Yeah over the past year I've become convinced that humans' limited working memory is actually a superpower, because it allows us to avoid this type of problem.
Isn't it less that we avoid the problem of limited working memory & more that we are constantly confronted by it and so we have had to devise processes for quickly organizing data by relevance to immediate & long-term goals, forgetting information that is no longer relevant, and creating new working memory chunks?
This feels like more of the same, but the comments in the code aren't as entertaining.
Previously, I'd be contracted to clean up messes from cut-rate dev shops. Now, there's plenty of opportunity for me to clean up messes from AI-driven development.
I pay an illustrator for every illustration on The Consensus, and paid a graphic designer for the logo after doing a first version myself. AI artwork, let alone AI writing, is just not great for your image.
You can find an illustrator anywhere in the world for the budget you'd like to spend (The Consensus is bootstrapped, a shoe-string budget). Would love to stop seeing tech companies using AI illustrations.
Unfortunately, it's not only big tech companies... but it's especially infuriating when it comes from a billion or trillion dollar company. As if they don't have the money to hire a professional illustrator.
For me there is almost nothing more grating than obvious AI illustrations. It makes it near impossible for me to take the content seriously. I'm wondering: do these people just have no taste? Or do they think nobody cares? Or they just don't care themselves?
I wonder if it's like Sam Altman not using capitalization, and other people in tech following suite just to signal in-group status. Deliberately producing bad output just to show you're part of a certain subculture.
From my own experience as a full time developer who is doing some side quests beside my main job - I’ll speak for myself, I got lucky to help a company get AI ready and help them define what ‘success’ even means working with it. And thinking about other companies who are just using it without a plan or strategy just gives me the feeling that this consultant / implementation work will rather be more than less within the next couple of years.
It sounds like they're mainly saying that there are certain categories of work in which people use AI to handle things themselves (directly via ChatGPT) and other categories for which they still post a freelance listing (especially if it requires skills across multiple domains).
Writing, illustration, and tech support seem to be the categories people handle themselves most often via AI, according to the first graphic... But then the second graphic suggests that the number of freelance postings for a top 'do-it-yourself-with-AI' category like writing is only down about 0.5% year over year. That almost suggests there isn't really an effect. Even the largest-changing category in that graphic, web development, is down only about 2%.
I think I conceptually can agree with their implication that AI allows freelancers to take on more breadth on their own, but the data/graphics they're presenting are confusing me. I'm not sure I get it.
Wow, so cool, I'm sure the world really wants to know OAI's thoughts on freelance work.
I'm just real tired of these companies inserting themselves everywhere. I look forward to when investors realise that their money was just yeeted into the furnace and the budget for this sort of thing evaporates.
Would be nice with a concise executive summary at the top. My interpretation: AI may be commoditising individual tasks while increasing demand for people who can combine those tasks into complete deliverables. That’s my whole read with some nuances on the different task shifts.
Very strange and I think I know the reason. At best, Upwork gave them some API access or just sent them the raw data to execute their "partnership" research. Then they launched an agent with their custom skills and it built that report. It's a report, it says something, but it's shallow. Probably they even instructed the agent to make it write reassuring narrative.
Maybe I missed it or maybe they are dodging the bigger question: is freelance work and the amount of money paid for it going up or down and by how much?
From what I read about the freelancers’ perspective, a lot of the newer gigs they get are to clean up work already done by AI instead of the opportunity to start new like they did in the past.
I've done a fair amount of this work.
Pretty challenging once the codebase gets large, but also a lot of fun to take a big mess and bring it under control.
I'm just wondering (without really being an AI fanboy, but using it daily), isn't the whole challenge just more prompting, such as instructing the agents to analyze the codebase, propose refactoring approaches, etc? Maybe it's a bit more sophisticated prompting than a non-technical person would do but prompting nonetheless.
I'm asking because I have read these comments here and there about fixing the mess AI made. I know what mess it usually creates and it doesn't get better with the new models. But I also know that whatever is created by AI kind of works more often than not, and once something breaks, it's just one prompt away to fixing it.
In my experience, the problem is that once a codebase passes like ~50,000 lines, it's too big for AI to make good architectural choices (this is controversial, but I have yet to see evidence otherwise).
So what I do is mostly use the AI to understand things like architecture, encapsulation, deployment pipelines, types, migration structure, etc.
It's a lot of "Audit the callers of X and give me a report" as well as reading code. I'm looking for stuff that doesn't pass my smell test, especially relating type types/interfaces. I look at the AI recommendations, but they're only useful once I've scoped them down to more granular questions, such as assessing the interfaces between specific layers of a pipeline.
That process is about understanding the shape of the solution that exists and then trying to formulate what shape it should have.
Then it's a lot of identifying a very boring, vanilla architectural target and moving towards it in phases, using the strangler fig pattern pretty aggressively. I review every PR, but mostly at the level of architecture rather than every specific line.
The codebase size will be limited to the LLM context window size. Once the codebase grows larger than the context window, some of the codebase will have to be removed from the context and thus not be known to that LLM instance.
There's also a well documented phenomena [1] where LLMs struggle to recall information in the middle of the context.
[1] https://charlesanthonybrowne.medium.com/how-llms-end-up-igno...
Good post.
Yeah over the past year I've become convinced that humans' limited working memory is actually a superpower, because it allows us to avoid this type of problem.
Isn't it less that we avoid the problem of limited working memory & more that we are constantly confronted by it and so we have had to devise processes for quickly organizing data by relevance to immediate & long-term goals, forgetting information that is no longer relevant, and creating new working memory chunks?
This feels like more of the same, but the comments in the code aren't as entertaining.
Previously, I'd be contracted to clean up messes from cut-rate dev shops. Now, there's plenty of opportunity for me to clean up messes from AI-driven development.
[flagged]
I pay an illustrator for every illustration on The Consensus, and paid a graphic designer for the logo after doing a first version myself. AI artwork, let alone AI writing, is just not great for your image.
You can find an illustrator anywhere in the world for the budget you'd like to spend (The Consensus is bootstrapped, a shoe-string budget). Would love to stop seeing tech companies using AI illustrations.
Unfortunately, it's not only big tech companies... but it's especially infuriating when it comes from a billion or trillion dollar company. As if they don't have the money to hire a professional illustrator.
For me there is almost nothing more grating than obvious AI illustrations. It makes it near impossible for me to take the content seriously. I'm wondering: do these people just have no taste? Or do they think nobody cares? Or they just don't care themselves?
I wonder if it's like Sam Altman not using capitalization, and other people in tech following suite just to signal in-group status. Deliberately producing bad output just to show you're part of a certain subculture.
I have a history of not using capitalization going back to the early 90's.
Old UNIX habit.
Had no idea it was a new "in-crowd" signal.
From my own experience as a full time developer who is doing some side quests beside my main job - I’ll speak for myself, I got lucky to help a company get AI ready and help them define what ‘success’ even means working with it. And thinking about other companies who are just using it without a plan or strategy just gives me the feeling that this consultant / implementation work will rather be more than less within the next couple of years.
It sounds like they're mainly saying that there are certain categories of work in which people use AI to handle things themselves (directly via ChatGPT) and other categories for which they still post a freelance listing (especially if it requires skills across multiple domains).
Writing, illustration, and tech support seem to be the categories people handle themselves most often via AI, according to the first graphic... But then the second graphic suggests that the number of freelance postings for a top 'do-it-yourself-with-AI' category like writing is only down about 0.5% year over year. That almost suggests there isn't really an effect. Even the largest-changing category in that graphic, web development, is down only about 2%.
I think I conceptually can agree with their implication that AI allows freelancers to take on more breadth on their own, but the data/graphics they're presenting are confusing me. I'm not sure I get it.
Content writing is down 0.57 percentage points to a share of now 1.27%, which is a ≈31% drop.
Oh I see, yes you’re right, my mistake. I completely missed that.
Wow, so cool, I'm sure the world really wants to know OAI's thoughts on freelance work.
I'm just real tired of these companies inserting themselves everywhere. I look forward to when investors realise that their money was just yeeted into the furnace and the budget for this sort of thing evaporates.
Would be nice with a concise executive summary at the top. My interpretation: AI may be commoditising individual tasks while increasing demand for people who can combine those tasks into complete deliverables. That’s my whole read with some nuances on the different task shifts.
Very strange and I think I know the reason. At best, Upwork gave them some API access or just sent them the raw data to execute their "partnership" research. Then they launched an agent with their custom skills and it built that report. It's a report, it says something, but it's shallow. Probably they even instructed the agent to make it write reassuring narrative.
Maybe I missed it or maybe they are dodging the bigger question: is freelance work and the amount of money paid for it going up or down and by how much?
so 'build me ebay clone for $500" is still not possible. Would be interesting to see how it changed on toptal.
[flagged]