Interacting with LLMs or AI APIs sounds like other software patterns, it doesn't matter that its AI or an LLM really, you are calling a function and providing inputs and expecting output. You get better output when your inputs are tuned to the scenario. Some of your inputs in this paradigm could be considered as optional parameters because you still get output without them.
If you need to remember parts of the inputs in between user sessions then you need to save state of those somewhere to a disk. Databases are a common choice especially in web development but you could also just put things in a file. Another option if this isn't a web development context is to use something like sqlite since it will help organize the data a little better than say CSVs or similar.
Proper usage of LLMs so you don't just flood them with useless context will just be custom tailored prompts that only include the pertinent context, with prompts saying how it's related to what you're looking for. I don't think there's a cheap way around it maybe on the plus side you can tune them using ai code.
I think tools are really over used and over-rated and have had horrible experience with them, nothing beats just custom tailoring stuff and setting up a system around it.
What I do is use elixir pheonix, have a genserver keep track of the user state and I just include the related state in the request and just helper functions to generate the related prompts per type of state/context and append them wherever makes the most sense.
I think LLMs make most sense to be viewed of as singular atomic interactions where you have the whole input (prompt/context/data) and get a concrete output. Everything else just seems like people being lazy trying to avoid thinking about the best way of structuring it. Where you put the context/data and how you include it will vary per prompt or the specific atomic interaction, there is no standard rule each interaction is unique. You have to experiment and see what provides the best output for each kind of request. I'd read Anthropics prompting docs if you haven't it's very good.
https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
My way of thinking is just viewing every isolated LLM request as a unique function that is
the prompt + llm = a unique function
Context is just what data you pass into the function (prompt+llm+settings(temp, etc))(data) to get whatever specific output you want. The prompt includes prewriting user/system messages, system prompt, structured output stuff or whatever. Any single request might lead to 1 or 30 of these that feed back into each other.
But yeah based on that it depends on just custom tailoring them for anything, it's pretty conceptual and intellectual but I find it fun but I don't think there's any easy way around it. Having the ability to have all your requests be stateful and modify what goes into the prompt based on the current user state (like genservers/elixir makes very easy) is a nice technical thing that helps though.
I embed tons of separate pieces of information, save the vectors in a db. Embed the user's question, then have a stored procedure in the db to calculate the top 10 (or 20 or 50 depending on the model) similar pieces of information.
I have an editor where I can ask a question and it brings up the most related pieces of info, and if I change any of those pieces it will update the embedding in the db
That's a good approach. But what I'm looking for is a bit different, more like Segment, but for LLMs. Something that when a user lands on your website, clicks around, and interacts with your app, you get a full behavioral context out of the box, including click path, location, language, currency, etc. You can then inject that context directly into your prompt so the LLM understands what the user is doing and responds without guessing or asking.
I think MCP is the right place to declare the context management API; the C in MCP is Context. As far as building goes, you could build a (universal) context store. I guess the value would be to bring the context closer to the model?
Using MCP, this could be a method that would get the context to take decisions.
For example, here's an example of how I use it currently:
```
const context = await getContext();
const response = await generateText({
system: `Here's the user context: ${context}`,
messages: [...]
});
console.log(context);
// "First-time visitor using Google Chrome on a MacBook, browsing from San Francisco.
// Landed on the pricing page from a Google ad, clicked to compare plans,
// then visited the enterprise section before initiating a support chat."
```
It's like a session recorder for LLMs that captures rich user behavior and traits (like device, browser, location, and journey) and turns them into LLM context. Your agent or app instantly becomes more helpful, relevant, and aware without wiring up your own tracking and enrichment pipeline.
I don't know of anything off-the-shelf, but you could query analytics tools at runtime (e.g. Mixpanel, PostHog) to gather the raw data, and use a generic summarizer to turn that into behavioral context that's usable downstream.
Yeah, exactly. My whole point is to avoid doing all that. It adds up fast. What I really want is something that handles the heavy lifting end-to-end: tracking, interpreting, and outputting a prompt-ready summary like:
"The user landed on the pricing page from a Google ad, clicked to compare plans, then visited the enterprise section before initiating a support chat."
That reads like the kind of session context you’d use for things like breadcrumbs and the like. Just keep a summary going in the user session, re-pack it or summarize it as soon as it gets above a threshold.
Would love to have something like a hotjar/analytics script that could automatically collect context and then I could query it to produce context for a prompt
Great idea, you should build it. Then do a Show HN with it
It might help with context generation. But honestly, most of the work is still in tracking, processing, enriching (different services, like IP location, etc), and all the plumbing around it.
Interacting with LLMs or AI APIs sounds like other software patterns, it doesn't matter that its AI or an LLM really, you are calling a function and providing inputs and expecting output. You get better output when your inputs are tuned to the scenario. Some of your inputs in this paradigm could be considered as optional parameters because you still get output without them.
If you need to remember parts of the inputs in between user sessions then you need to save state of those somewhere to a disk. Databases are a common choice especially in web development but you could also just put things in a file. Another option if this isn't a web development context is to use something like sqlite since it will help organize the data a little better than say CSVs or similar.
Proper usage of LLMs so you don't just flood them with useless context will just be custom tailored prompts that only include the pertinent context, with prompts saying how it's related to what you're looking for. I don't think there's a cheap way around it maybe on the plus side you can tune them using ai code. I think tools are really over used and over-rated and have had horrible experience with them, nothing beats just custom tailoring stuff and setting up a system around it.
What I do is use elixir pheonix, have a genserver keep track of the user state and I just include the related state in the request and just helper functions to generate the related prompts per type of state/context and append them wherever makes the most sense.
I think LLMs make most sense to be viewed of as singular atomic interactions where you have the whole input (prompt/context/data) and get a concrete output. Everything else just seems like people being lazy trying to avoid thinking about the best way of structuring it. Where you put the context/data and how you include it will vary per prompt or the specific atomic interaction, there is no standard rule each interaction is unique. You have to experiment and see what provides the best output for each kind of request. I'd read Anthropics prompting docs if you haven't it's very good. https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
My way of thinking is just viewing every isolated LLM request as a unique function that is the prompt + llm = a unique function Context is just what data you pass into the function (prompt+llm+settings(temp, etc))(data) to get whatever specific output you want. The prompt includes prewriting user/system messages, system prompt, structured output stuff or whatever. Any single request might lead to 1 or 30 of these that feed back into each other. But yeah based on that it depends on just custom tailoring them for anything, it's pretty conceptual and intellectual but I find it fun but I don't think there's any easy way around it. Having the ability to have all your requests be stateful and modify what goes into the prompt based on the current user state (like genservers/elixir makes very easy) is a nice technical thing that helps though.
I embed tons of separate pieces of information, save the vectors in a db. Embed the user's question, then have a stored procedure in the db to calculate the top 10 (or 20 or 50 depending on the model) similar pieces of information.
I have an editor where I can ask a question and it brings up the most related pieces of info, and if I change any of those pieces it will update the embedding in the db
That's a good approach. But what I'm looking for is a bit different, more like Segment, but for LLMs. Something that when a user lands on your website, clicks around, and interacts with your app, you get a full behavioral context out of the box, including click path, location, language, currency, etc. You can then inject that context directly into your prompt so the LLM understands what the user is doing and responds without guessing or asking.
What is the application specific scenario that is requiring this context? Everyone has different scenarios and this might not make sense
I think MCP is the right place to declare the context management API; the C in MCP is Context. As far as building goes, you could build a (universal) context store. I guess the value would be to bring the context closer to the model?
The value is building the context itself.
Using MCP, this could be a method that would get the context to take decisions.
For example, here's an example of how I use it currently:
```
const context = await getContext();
const response = await generateText({ system: `Here's the user context: ${context}`, messages: [...] });
console.log(context);
// "First-time visitor using Google Chrome on a MacBook, browsing from San Francisco.
// Landed on the pricing page from a Google ad, clicked to compare plans,
// then visited the enterprise section before initiating a support chat."
```
It's like a session recorder for LLMs that captures rich user behavior and traits (like device, browser, location, and journey) and turns them into LLM context. Your agent or app instantly becomes more helpful, relevant, and aware without wiring up your own tracking and enrichment pipeline.
A context inference service sounds valuable but I wonder what your moat would be.
Yep, that's something I'd have to figure out.
I don't know of anything off-the-shelf, but you could query analytics tools at runtime (e.g. Mixpanel, PostHog) to gather the raw data, and use a generic summarizer to turn that into behavioral context that's usable downstream.
Yeah, exactly. My whole point is to avoid doing all that. It adds up fast. What I really want is something that handles the heavy lifting end-to-end: tracking, interpreting, and outputting a prompt-ready summary like:
"The user landed on the pricing page from a Google ad, clicked to compare plans, then visited the enterprise section before initiating a support chat."
That reads like the kind of session context you’d use for things like breadcrumbs and the like. Just keep a summary going in the user session, re-pack it or summarize it as soon as it gets above a threshold.
You might find this useful: https://context7.com/
Super interesting! However, it focuses on external sources rather than the user journey.
I haven’t solved this, but sounds super useful!
Would love to have something like a hotjar/analytics script that could automatically collect context and then I could query it to produce context for a prompt
Great idea, you should build it. Then do a Show HN with it
Exactly! Something like a tag you install and then query prompt-ready contexts.
MCP maybe? You could provide tools for the LLM to discover that data at runtime.
It might help with context generation. But honestly, most of the work is still in tracking, processing, enriching (different services, like IP location, etc), and all the plumbing around it.