AI Home Server, Memory & Messaging Platform

Context

I run this as a real home server, not a one-off Raspberry Pi demo. The current host is a 4 GB Raspberry Pi 5 running Debian 13 on ARM64, with a 2 TB NVMe SSD as its primary storage.

The server supports a broader personal infrastructure stack:

• Hermes agent sessions and messaging gateways
• Matrix and Discord delivery surfaces
• Calendar, email, file, web, and scheduled automations
• Docker services, PostgreSQL workloads, Caddy reverse proxying, and operational logs
• Persistent semantic memory shared across agent sessions

The platform began on AWS EC2. I later migrated it home to gain direct control over storage, service behavior, logs, upgrades, and operating cost while retaining the same production concerns: process supervision, recovery, security boundaries, and reliable message delivery.

Problem

A useful personal agent needs more than a model endpoint. It must stay available, reach the right tools, deliver responses through normal communication channels, and remember durable context without replaying entire conversation histories into every prompt.

The memory layer created a second infrastructure problem. The Pi 5 kernel uses 16,384-byte memory pages. Qdrant’s official ARM64 Docker image and prebuilt standalone binary were compiled with a jemalloc configuration that assumed 4 KB pages, so both failed immediately on this host with:

<jemalloc>: Unsupported system page size

At the same time, embedded Qdrant was not a durable alternative. An embedded vector store places a file lock on its storage directory, while this system can have multiple Hermes processes such as the gateway, CLI sessions, and scheduled jobs. The memory backend needed to be a separate service that all of them could query safely.

My Role

I designed, deployed, migrated, and continue to operate the stack.

Key work:

• Migrated the original container stack from AWS EC2 to a Raspberry Pi 5 home server.
• Built Qdrant 1.18.2 natively from Rust source for the Pi’s 16 KB page-size environment.
• Configured Qdrant as a restartable systemd user service with local-only network exposure and persistent NVMe storage.
• Integrated Hermes with Mem0 in OSS mode, using Qdrant for vectors, OpenAI for embeddings, and DeepSeek for fact extraction.
• Structured memory into separate layers for durable facts, active user context, session history, and reusable procedures.
• Integrated Matrix and Discord gateway workflows and routed 5,000+ messages across bridged services.
• Operated Docker, PostgreSQL, Caddy, TLS, logs, backups, credentials, and automation integrations.

System Architecture

Messaging clients
  Matrix / Discord / local
          ↓
Hermes gateway [systemd]
          ↓
Agent runtime
  ├── tools + scheduled jobs
  └── memory
      ├── compact profile
      ├── Mem0 OSS
      │   ├── DeepSeek extraction
      │   ├── OpenAI embeddings
      │   └── Qdrant on NVMe
      ├── session history
      └── procedural skills
          ↓
Linux + Docker + PostgreSQL + Caddy
          ↓
Pi 5 + Debian ARM64 + 2 TB NVMe

The messaging gateway and Qdrant run as separate supervised services. This keeps chat available even if semantic-memory calls become slow or temporarily unavailable.

Memory System, Layer by Layer

“Memory” is not one database in this system. Each layer has a different job.

1. Compact Built-In Memory

Hermes keeps a small, high-priority memory and user-profile block for facts that should be available every session, such as stable preferences, environment conventions, and operating context. It is deliberately size-limited so important details do not disappear inside a large prompt.

2. Mem0 OSS for Semantic Memory

Mem0 decides what conversational information should become a durable fact. In the active configuration, it runs in OSS mode rather than using Mem0’s hosted platform.

When a turn completes, the provider can send the user and assistant exchange to a DeepSeek extraction model. Mem0 identifies durable facts, reconciles them with existing memories, and performs additions or updates rather than blindly saving the transcript.

For recall, Hermes starts a semantic prefetch when a new turn begins. If the backend is slow, the agent continues instead of blocking indefinitely; explicit memory search remains available as a fallback. A circuit breaker pauses calls after repeated failures so an unavailable vector database does not take down normal chat.

3. OpenAI Embeddings

Durable memory text is converted into 1,536-dimensional vectors with text-embedding-3-small. The same embedding model must be used for both writes and searches because Qdrant collections require a fixed vector dimension.

This is an intentional low-RAM tradeoff. The Pi stores and searches vectors locally, while embedding and extraction inference run remotely. That avoids loading local language and embedding models into a 4 GB machine, but it also means the memory path is not fully air-gapped.

4. Qdrant for Local Vector Storage

Qdrant is the retrieval database, not the entire memory system. Mem0 handles fact extraction and memory policy; Qdrant stores vectors and returns semantically similar records.

The active collection uses:

• 1,536-dimensional vectors
• Cosine similarity
• On-disk vectors and payloads
• Payload indexes for user and agent scoping
• Two storage segments
• A low memory-mapping threshold for the constrained host
• Local-only HTTP and gRPC listeners

The collection is configured for durability on the constrained host with on-disk storage and local-only network exposure.

5. Session History and Skills

Semantic facts are not a substitute for full history or procedures. Past conversations stay in searchable session storage for questions such as “where did we leave this project?” Reusable operating procedures live as skills rather than being mixed into personal memory. Longer human-curated documentation stays in the project wiki.

This separation keeps each retrieval path focused:

Stable preference or fact       → compact memory / Mem0
Past conversation or decision   → session search
Reusable technical procedure    → skill
Long-form canonical reference   → project documentation

Why Qdrant Required a Native Rust Build

Qdrant uses tikv-jemallocator as its global allocator on ARM64. The allocator is statically linked into the release binary, so changing Docker base images, preloading libc, or adjusting runtime allocator flags cannot repair a binary compiled for the wrong page size.

The reliable fix was to compile Qdrant on the Pi itself. Cargo rebuilt jemalloc against the running 16 KB page-size environment.

Qdrant v1.18.2 source
        ↓
Cargo release build on ARM64 Pi 5
        ↓
jemalloc detects 16 KB system pages
        ↓
Native 76 MB Qdrant binary
        ↓
systemd user service on localhost

The release build was also a resource-management exercise. Qdrant’s link-time optimization can exceed the Pi’s 4 GB of physical RAM, particularly during the final Rust linking stage. I added a temporary 12 GB swap file, kept the full release optimizations enabled, completed the native build in roughly an hour, installed the resulting binary, and then removed the build-only swap.

I chose extra swap over weakening the release profile because the constrained build environment should not force a permanently slower production binary.

Operating on a 4 GB Pi

The memory service is tuned around the machine rather than pretending it has cloud-server resources:

• Qdrant binds only to loopback and stores data on NVMe.
• Vectors and payloads are kept on disk instead of forcing the collection into RAM.
• A low memory-mapping threshold lets the kernel page data from NVMe as needed.
• The service uses restart-on-failure, a restrictive file-creation mask, task limits, and an increased file-descriptor limit.
• Qdrant runs outside Docker because the published ARM64 image contains the incompatible allocator build.
• Extraction and embedding models stay remote so the Pi can prioritize gateways, storage, and service orchestration.
• Health checks, service state, logs, and collection status provide separate failure boundaries for the gateway and memory database.

The design uses the Pi for what it does well: always-on orchestration, local persistence, networking, and lightweight services. Heavy inference remains external.

Messaging and Automation

The same host connects agent reasoning to ordinary communication surfaces. Matrix and Discord gateways let workflows arrive where conversations already happen instead of requiring SSH or a dedicated dashboard.

The agent can then reach tools for calendar, email, files, web research, local services, and scheduled jobs. The provider scopes recall by user identity and tags writes with gateway-channel metadata. That keeps unrelated users separated while making it possible to inspect or filter memories by delivery surface.

Key Engineering Decisions

Compile for the Real Hardware

The allocator failure was below the application layer. Treating it as a Docker or environment-variable problem would only hide the boundary. Verifying the 16 KB kernel page size and rebuilding the Rust dependency on-device fixed the actual incompatibility.

Separate the Vector Service From Agent Processes

A standalone Qdrant daemon avoids embedded-storage file locks and gives the CLI, gateway, and scheduled workers one concurrent backend with its own restart and health behavior.

Keep Storage Local and Models Remote

Local vectors preserve direct control over persistence and make retrieval fast on the home network. Remote extraction and embeddings fit the 4 GB memory budget. The tradeoff is explicit rather than accidental.

Use Different Stores for Different Kinds of Memory

Durable facts, complete transcripts, procedures, and long-form documentation have different retention and retrieval needs. Treating all four as “chat history” would increase prompt size and reduce accuracy.

Design Failure as a Normal State

The gateway, memory provider, vector service, and external model APIs can fail independently. Supervision, bounded waits, explicit search fallback, and circuit breaking keep one degraded integration from freezing the whole agent.

Result

The system now runs the Hermes gateway and a healthy, source-built Qdrant service on the Raspberry Pi 5. It provides persistent semantic recall across sessions while preserving gateway identity boundaries, without replaying full transcripts into every prompt.

The project demonstrates more than self-hosting. It combines ARM64 systems debugging, Rust compilation, Linux service management, low-memory architecture, vector retrieval, agent-memory policy, gateway integration, and day-to-day operations in one working environment.

Engineering Takeaways

• Low-level platform details such as allocator builds and kernel page size can invalidate an otherwise correct deployment artifact.
• A vector database stores and retrieves embeddings; extraction, deduplication, and memory policy belong to a separate layer.
• Small machines benefit from hybrid architecture: local persistence and orchestration, remote model inference.
• Persistent agent memory works best when semantic facts, session history, procedures, and documentation remain separate.
• Reliable agent infrastructure requires bounded latency, restart behavior, observability, and degraded-mode operation, not just successful model calls.

Disclosure Notes

This page intentionally omits IP addresses, hostnames, domains, private network routes, credentials, tokens, webhook URLs, gateway configuration, real messages, admin surfaces, and backup paths. Hardware, public software versions, and sanitized architecture are included because they explain the engineering decisions without exposing the live system.