# I gave my on-call agent Hindsight memory. Here's what changed.

I pasted a production-looking alert into my own incident agent: payments-api returning 502s, Redis reporting "Too many connections". The answer was correct and useless: check Redis, scale it, add monitoring. Anything a search engine could have told me.

Then I turned memory on. Same alert, different answer: this matches INC-001 from May 8, and the fix that worked was raising Redis `max_connections` from 50 to 200 and setting the payments pool limit to 120. MTTR was 22 minutes.

That gap is the whole project.

I built this because the pattern is familiar to anyone who has carried a pager. An alert fires, the on-call engineer half-remembers a similar incident from a few months ago, and then spends twenty minutes digging through old tickets and chat threads for the fix. The knowledge exists, but it lives in people's heads and in postmortems nobody rereads. I wanted an agent that treats every resolved incident as something to learn from, so the next engineer starts from what the last one already figured out.

## What OnCall Memory does

OnCall Memory is a small Streamlit app. You paste an alert and a log snippet, and it produces two diagnoses side by side. The left one has no memory. The right one is grounded in past incidents stored in [Hindsight](https://github.com/vectorize-io/hindsight), an open-source [agent memory](https://vectorize.io/what-is-agent-memory) system.

The stack is deliberately boring: Streamlit for the UI, Groq (`openai/gpt-oss-120b`) for generation, and Hindsight as the only persistent store. There is no database of my own.

![Memory OFF vs ON](https://raw.githubusercontent.com/aryantyagi2211/oncall-memory/main/screenshots/captureit_07-11-2026_at_16-26-29.png align="center")

## How Hindsight is used

Hindsight exposes three operations, and I use all three. The [Hindsight docs](https://hindsight.vectorize.io/) cover them in detail, but the core calls are this small:

```python
from hindsight_client import Hindsight

client = Hindsight(base_url=HINDSIGHT_URL)

# retain: store a resolved incident
client.retain(
    bank_id="oncall-v3",
    content="INC-001: payments-api Redis connection exhaustion. "
            "Fix: raised max_connections 50 -> 200, pool limit 120.",
)

# recall: find relevant history for a new alert
client.recall(bank_id="oncall-v3", query="payments-api 502 Too many connections")

# reflect: reason over that history
client.reflect(bank_id="oncall-v3", query="How should I fix this incident?")
```

- **retain** runs in two places. A seed script loads 15 synthetic postmortems with stable IDs and dates (payments-api, auth-service, Kafka consumers, Kubernetes, Postgres). And a "Mark resolved" form saves the root cause and fix of a new incident.
- **recall** runs before the memory-enabled answer. The app shows the top five deduplicated incidents with ID, date, service and fix.
- **reflect** produces the reasoning the final answer is built from.

## Telling similar incidents apart

My seed data has several Redis incidents on purpose. INC-001 is connection exhaustion, fixed by raising `max_connections`. INC-015 is a failover that left 80 workers pointing at a retired primary, fixed by cutting the DNS cache TTL from 120s to 10s. INC-013 is missing TTLs on cache keys leading to OOM.

A keyword search for "Redis" returns all of them. For the "Too many connections" alert, recall put INC-001 on top. For a failover timeout alert, it surfaced INC-015. Same service, same technology, different fixes, and the right one came back.

![Recalled memories](https://raw.githubusercontent.com/aryantyagi2211/oncall-memory/main/screenshots/captureit_07-11-2026_at_16-27-30.png align="center")

## Making it learn

After a resolved incident, I saved INC-016 (stale DNS cache on payments-api workers after a Redis failover) through the "Mark resolved" form. A few minutes later I pasted a new failover alert, and INC-016 came back as the first recalled memory.

![Learning](https://raw.githubusercontent.com/aryantyagi2211/oncall-memory/main/screenshots/captureit_07-11-2026_at_16-34-05.png align="center")

## What went wrong

- **The reasoning model returned empty answers.** `gpt-oss-120b` spends tokens thinking before it writes. With a small `max_tokens`, `message.content` came back empty and my app treated that as a failure. Raising `max_completion_tokens` to 2000 and setting `reasoning_effort` to low fixed it. My fallback model name also no longer existed on Groq, so the fallback never worked.
- **My own validator hid a correct answer.** I added a check that rejects any incident ID the model cites that wasn't actually recalled. The first version was too strict and replaced a correct answer with "no relevant history". It now rejects only IDs absent from recall.
- **The save form can pollute memory.** "Mark resolved" saves whatever is in the incident box. I saved one incident while the old alert was still in the box, and it distorted later recalls. I had to reseed into a fresh bank.
- **Recalled isn't the same as used.** On the failover alert the answer cited INC-015, while INC-016, the one I had just saved, ranked first in recall. The memory is there, but the model leans on the closest older match.
- **The data is synthetic.** The 15 seeded incidents are made up. It shows the mechanism, not results on real production history.

## Lessons

1. **Make the difference visible.** A Memory ON/OFF toggle taught me more than any log, and it makes the value obvious to someone seeing it for the first time.
2. **Store exact fixes, not summaries.** "Raised `max_connections` from 50 to 200" is useful at 2 AM. "Fixed Redis" is not.
3. **Don't let the model invent history.** Check every cited incident ID against what was recalled.
4. **Test the model layer separately.** My memory worked long before my LLM calls did.
5. **Validate what you save.** A memory system is only as good as the postmortems you put in.

Code is on [GitHub](https://github.com/aryantyagi2211/oncall-memory). Video walkthrough: <https://youtu.be/0vvGlgbJXCc>

Tagging [Code.in](http://Code.in).