I do feel frustrated with the current state of evaluations for long-lived sessions with many tool calls -- by default OpenAI's built-in eval system seems to rate chat completions that end with a tool call as "bad" because the tool call response is only in the next completion.
But our stack is in Go and it has been tough to see a lot of observability tools focus on Python rather than an agnostic endpoint proxy like Helicone has.
Looks great! debugging agents is a huge pain for me, and this actually looks useful.
Love the time travel and trajectory clustering ideas. Bookmarked to try it soon
Congrats on the launch! On a tangential note, is this work open source or do you guys have some technical report that you could share? I am specially interested in your results on the clustering methods for surfacing behavioural patterns. Thanks!
We're new to the open source scene so we don't have anything published yet but plan to in the future. A basic overview of the way we do clustering is we condense stateful information -> create a state embedding -> create tags -> cluster based on distance of tags + embeddings.
Feel free to reach out if you want some guidance. At a minimum your SDK should be open source since it potentially touches sensitive data and you’ll want to build trust. Also, it probably technically already is unless you’ve only released Python binary wheels.
I am not an expert but still I am building enough agents. But I don't understand how this tool can be integrated with an exisiting system. Is it like an APM for agents if I understand it correctly ?
the way it is integrated (its explained more in the docs) is by installing the python/typescript sdk and writing "lai.init()" at the top of your code. Then we capture all LLM calls and tools with integrated providers (similar to LLM ops platforms). If you want to manually add more information you can add decorators, lai.create_step/create_event "logs", etc.
We then take all this information you give us and try to transform it i.e group together similar nodes, run an agent to evaluate a session, or to find root cause of a session failure in the backend.
Langfuse and Helicone work well for traditional LLM operations, but AI agents are different. We discovered that AI agents require fundamentally different tooling, here are some examples.
First, while LLMs simply respond to prompts, agents often get stuck in behavioral loops where they repeat the same actions; to address this, we built a graph visualization that automatically detects when an agent reaches the same state multiple times and groups these occurrences together, making loops immediately visible.
Second, our evaluations are much more tailored for AI Agents. LLM ops evaluations usually occur at a per prompt level (i.e hallucination, qa-correctness) which makes sense for those use cases, but agent evaluations are usually per session or run. What this means is that usually a single prompt in isolation didn’t cause an issue but some downstream memory issue or previous action caused this current tool to fail. So, we spent a lot of time creating a way for you to create a rubric. Then, to evaluate the rubric (so that there isn’t context overload) we created an agentic pipeline which has tools like viewing rubric examples, ability to zoom “in and out” of a session (to prevent context overload), referencing previous examples, etc.
Third, time traveling and clustering of similar responses. LLM debugging is straightforward because prompts are stateless and are independent from one another, but agents maintain complex state through tools, context, and memory management; we solved this by creating “time travel” functionality that captures the complete agent state at any point, allowing developers to modify variables like context or tool availability and replay from that exact moment and then simulate that 20-30 times and group together similar responses (with our clustering alg).
Fourth, agents exhibit far more non-deterministic behavior than LLMs because a single tool call can completely change their trajectory; to handle this complexity, we developed workflow trajectory clustering that groups similar execution paths together, helping developers identify patterns and edge cases that would be impossible to spot in traditional LLM systems.
Colloquially, AI agents are just while loops with LLM calls and tool calls. More specifically, what distinguishes an agent from LLM pipelines is that its next step is determined dynamically (based on the output of the previous one) so the execution path isn’t fixed. The boundary between complex LLM chaining and agents is pretty fuzzy, but we support both.
Gotcha, you're using the "LLM calling tools in a loop" definition. I think that's a decent one, but I worry that many people out there are carrying around completely different ideas as to what the term means.
- "LLM running tools in a loop" - often used by Anthropic, generally the most popular among software engineers who build things
- "An AI system that performs tasks on your behalf" - used by OpenAI, I dislike how vague this one is
- "an entity that perceives its environment through sensors and acts upon that environment through actuators to achieve specific goals" - the classic academic one, Russell and Norvig. I sometimes call this the "thermostat definition".
- "kinda like a travel agent I guess?" - quite common among less technical people I've talked to
The way I draw the line is to focus on the "agency" aspect.
In workflows/pipelines the "agency" belongs to the coder/creator of the workflow. It usually resembles something like a "list of steps" or "ittt". Examples include traditional "research" flows like 1. create search terms for query; 2. search; 3. fetch_urls; 4. summarise; 5. answer
In agents the "agency" belongs, at one point or another, to the LLM. It gets to decide what to do at some steps, based on context, tools available, and actions taken. It usually resembles a loop, without predefined steps (or with vague steps like "if this looks like a bad answer, retry" - where bad answer can be another LLM invocation w/ a specific prompt). Example: Fix this ticket in this codebase -> ok, first I need to read_files -> read_files tool call ... and so on.
In the research workflow example, what if the first set of search queries don’t return good results. If the LLM tool loop decides to refine the queries, would this be “agency”?
I'd say so, yeah. If the LLM "decides" what steps to take, that's an agent. If the flow is "hardcoded" then it's a workflow/pipeline. It often gets confused because early frameworks called these workflows/pipelines "agents".
LLM as a judge for agent usually has context overload and even if you have a really good prompt for your evaluation, LLMs hallucinate because there is just too much information to ingest. So we created an agentic pipeline to basically do evaluations on rubrics which have better results and dont miss intricacies due to the overloaded context.
I'm reading: the difference is that this is an agent as a judge rather than an LLM as a judge, paired with more structured judging parameters. Is that right? Is the agent just a loop over each criterium, or is it also reflecting somehow on its judging or similar?
looks cool—what do you mean clustering similar responses. Usually llm outputs are a bit different would those be the clustered together or is it exact text similarity
thank you, this is cool/interesting. i work in this space and I was thinking yesterday that it would be an interesting contemporary witness to record competition & then see how things shake out.
my vision is that the market is not really prepared for that right now, the best way is this guys is solving a really niche problem with their plataform and then expanding trough more areas
What kinds of rubrics can one specify? Is there a tutorial or a page with some examples of this kind of rubrics defined in Lucidic?
I do feel frustrated with the current state of evaluations for long-lived sessions with many tool calls -- by default OpenAI's built-in eval system seems to rate chat completions that end with a tool call as "bad" because the tool call response is only in the next completion.
But our stack is in Go and it has been tough to see a lot of observability tools focus on Python rather than an agnostic endpoint proxy like Helicone has.
we're working on that right now, we'd love to hear your opinions(if you're interested you can send us an email at team@lucidic.ai).
Looks great! debugging agents is a huge pain for me, and this actually looks useful. Love the time travel and trajectory clustering ideas. Bookmarked to try it soon
Awesome--let us know what you think!
Congrats on the launch! On a tangential note, is this work open source or do you guys have some technical report that you could share? I am specially interested in your results on the clustering methods for surfacing behavioural patterns. Thanks!
We're new to the open source scene so we don't have anything published yet but plan to in the future. A basic overview of the way we do clustering is we condense stateful information -> create a state embedding -> create tags -> cluster based on distance of tags + embeddings.
Feel free to reach out if you want some guidance. At a minimum your SDK should be open source since it potentially touches sensitive data and you’ll want to build trust. Also, it probably technically already is unless you’ve only released Python binary wheels.
I am not an expert but still I am building enough agents. But I don't understand how this tool can be integrated with an exisiting system. Is it like an APM for agents if I understand it correctly ?
the way it is integrated (its explained more in the docs) is by installing the python/typescript sdk and writing "lai.init()" at the top of your code. Then we capture all LLM calls and tools with integrated providers (similar to LLM ops platforms). If you want to manually add more information you can add decorators, lai.create_step/create_event "logs", etc.
We then take all this information you give us and try to transform it i.e group together similar nodes, run an agent to evaluate a session, or to find root cause of a session failure in the backend.
I'm looking into a tool like this for my startup. Why should I use this over Langfuse or Helicone?
Langfuse and Helicone work well for traditional LLM operations, but AI agents are different. We discovered that AI agents require fundamentally different tooling, here are some examples.
First, while LLMs simply respond to prompts, agents often get stuck in behavioral loops where they repeat the same actions; to address this, we built a graph visualization that automatically detects when an agent reaches the same state multiple times and groups these occurrences together, making loops immediately visible.
Second, our evaluations are much more tailored for AI Agents. LLM ops evaluations usually occur at a per prompt level (i.e hallucination, qa-correctness) which makes sense for those use cases, but agent evaluations are usually per session or run. What this means is that usually a single prompt in isolation didn’t cause an issue but some downstream memory issue or previous action caused this current tool to fail. So, we spent a lot of time creating a way for you to create a rubric. Then, to evaluate the rubric (so that there isn’t context overload) we created an agentic pipeline which has tools like viewing rubric examples, ability to zoom “in and out” of a session (to prevent context overload), referencing previous examples, etc.
Third, time traveling and clustering of similar responses. LLM debugging is straightforward because prompts are stateless and are independent from one another, but agents maintain complex state through tools, context, and memory management; we solved this by creating “time travel” functionality that captures the complete agent state at any point, allowing developers to modify variables like context or tool availability and replay from that exact moment and then simulate that 20-30 times and group together similar responses (with our clustering alg).
Fourth, agents exhibit far more non-deterministic behavior than LLMs because a single tool call can completely change their trajectory; to handle this complexity, we developed workflow trajectory clustering that groups similar execution paths together, helping developers identify patterns and edge cases that would be impossible to spot in traditional LLM systems.
This makes sense. We'll look into this some more, will be making a decision next couple days :)
Good luck!
How does Lucidic define the term "AI agent"?
Colloquially, AI agents are just while loops with LLM calls and tool calls. More specifically, what distinguishes an agent from LLM pipelines is that its next step is determined dynamically (based on the output of the previous one) so the execution path isn’t fixed. The boundary between complex LLM chaining and agents is pretty fuzzy, but we support both.
Haha also our whole backend is in Django :)
Gotcha, you're using the "LLM calling tools in a loop" definition. I think that's a decent one, but I worry that many people out there are carrying around completely different ideas as to what the term means.
Do you have a writeup on the different interpretations of "AI Agent"?
I need to put one together. The big ones are:
- "LLM running tools in a loop" - often used by Anthropic, generally the most popular among software engineers who build things
- "An AI system that performs tasks on your behalf" - used by OpenAI, I dislike how vague this one is
- "an entity that perceives its environment through sensors and acts upon that environment through actuators to achieve specific goals" - the classic academic one, Russell and Norvig. I sometimes call this the "thermostat definition".
- "kinda like a travel agent I guess?" - quite common among less technical people I've talked to
I gathered over a hundred on Twitter last year, summarized by Gemini here: https://gist.github.com/simonw/beaa5f90133b30724c5cc1c4008d0...
I also have a tag about this on my blog: https://simonwillison.net/tags/agent-definitions/
The way I draw the line is to focus on the "agency" aspect.
In workflows/pipelines the "agency" belongs to the coder/creator of the workflow. It usually resembles something like a "list of steps" or "ittt". Examples include traditional "research" flows like 1. create search terms for query; 2. search; 3. fetch_urls; 4. summarise; 5. answer
In agents the "agency" belongs, at one point or another, to the LLM. It gets to decide what to do at some steps, based on context, tools available, and actions taken. It usually resembles a loop, without predefined steps (or with vague steps like "if this looks like a bad answer, retry" - where bad answer can be another LLM invocation w/ a specific prompt). Example: Fix this ticket in this codebase -> ok, first I need to read_files -> read_files tool call ... and so on.
In the research workflow example, what if the first set of search queries don’t return good results. If the LLM tool loop decides to refine the queries, would this be “agency”?
I'd say so, yeah. If the LLM "decides" what steps to take, that's an agent. If the flow is "hardcoded" then it's a workflow/pipeline. It often gets confused because early frameworks called these workflows/pipelines "agents".
I see, that's a good way to think about it
Is the front end built using AI? It's unusable on Pixel 8a. You may lose users, please fix the responsive design.
Given that its a tool for development, it seems wise for them to focus on priorities other than mobile phone usablity
Nice, i think that yall are on the correct path betting on evals, but please make your ui less "generic"
You say your rubric approach is “better than llm as a judge.” Can you please elaborate on what makes you say that?
LLM as a judge for agent usually has context overload and even if you have a really good prompt for your evaluation, LLMs hallucinate because there is just too much information to ingest. So we created an agentic pipeline to basically do evaluations on rubrics which have better results and dont miss intricacies due to the overloaded context.
I'm reading: the difference is that this is an agent as a judge rather than an LLM as a judge, paired with more structured judging parameters. Is that right? Is the agent just a loop over each criterium, or is it also reflecting somehow on its judging or similar?
Congrats on the launch - would be great to read more about the clustering approach you're taking
looks cool—what do you mean clustering similar responses. Usually llm outputs are a bit different would those be the clustered together or is it exact text similarity
Love the UX. From the value POV, I am yet to see/experience how it differs from competitors. P.S. I currently use Braintrust and Opik
Excited to try this
I've been keeping a rolling list of LLMOps/AI agent observability products funded by YC. What problems does Lucidic solve that the others do not?
https://hegel-ai.com https://www.vellum.ai/ https://www.parea.ai http://baserun.ai https://www.traceloop.com https://www.trychatter.ai https://talc.ai https://langfuse.com https://humanloop.com https://uptrain.ai https://athina.ai https://relari.ai https://phospho.ai https://github.com/BerriAI/bettertest https://www.getzep.com https://hamming.ai https://github.com/DAGWorks-Inc/burr https://www.lmnr.ai https://keywordsai.co https://www.thefoundryai.com https://www.usesynth.ai https://www.vocera.ai https://coval.ai https://andonlabs.com https://lucidic.ai https://roark.ai https://dawn.so/ https://www.atla-ai.com https://www.hud.so https://www.thellmdatacompany.com/ https://casco.com https://www.confident-ai.com
You should compile these into a Gist or some static page.
Here's my full list: https://gist.github.com/areibman/b1f66a9a037005b2d4bbf5ba2e5...
thank you, this is cool/interesting. i work in this space and I was thinking yesterday that it would be an interesting contemporary witness to record competition & then see how things shake out.
yet another observability tool thats joining the already overcrowded space
my vision is that the market is not really prepared for that right now, the best way is this guys is solving a really niche problem with their plataform and then expanding trough more areas