Rayhan Patel
Software Engineer, AI Agents

Rayhan Patel

I build AI agents and measure whether they actually work.

Agent memory and LLM evaluation at 3E this summer. First-author paper at a RecSys 2026 workshop. Finishing an M.S. in Applied Machine Learning at the University of Maryland.

Available June 2027, full-time San Francisco Bay Area, CA Open to relocation
3E · Agent memoryMulti-tenant MCP service, benchmark and judge
EvoRank · First authorRecSys 2026 workshop, +0.0057 NDCG@10
2nd place · Cactus × DeepMind0.99 F1 on-device function calling
Winner · BigThink × World LabsBest Scientific & Engineering Application
01

Experience

AI Platform team
Jun 2026 – Aug 2026
Bethesda, MD (hybrid)

3E

AI Engineer Intern

Took the agent-memory question from open research to a working MCP service in ten weeks, for a production compliance-AI platform.

  • Built the platform's only cross-system scoreboard for agent memory, comparing four existing memory systems (Hindsight, Cognee, Zep and AgentCore Memory) on the LoCoMo and LongMemEval benchmarks, with byte-identical prompts and sha-pinned question sets so runs stay comparable.
  • Calibrated the LLM judge against blinded human graders and constructed ground truth before trusting its scores. The rubric, not the judge model, was the accuracy lever.
  • Wrote the architecture proposal from the results. The systems tied on easy recall but split on temporal reasoning, so the design routes by use case behind one API contract.
  • Implemented the core of the multi-tenant memory MCP service on AWS Lambda and DynamoDB: six operations, tenant access that denies by default, versioned writes and about 125 conformance tests.
  • Built a 120+ model capability catalog with live probes for the company's internal AI gateway.
PythonAWS LambdaDynamoDBMCPClaude Agent SDKLLM-as-a-judgeLoCoMoLongMemEvalLangfuse
Two-product AI startup
Jan 2024 – May 2024 · Mar 2025 – Jul 2025
Remote

Euler AI

Founding ML/Software Engineer (full-time, remote), 2025 · ML/AI Engineering Intern, 2024

Built the search and recommendation path of a conversational shopping assistant, plus the evaluation that measured it.

  • Built the search and recommendation path of a shopping assistant as FastAPI services: a quantized DistilBERT intent router, embedding retrieval, a reranker and PII guardrails.
  • Shipped an LLM-as-a-judge (G-Eval) batch pipeline that measured hallucination across the product.
  • Designed a healthcare system that turns electronic health records into medical codes, so clinicians spend less time on paperwork.
PythonFastAPIAsyncIODistilBERTEmbeddingsRerankingG-EvalGCPVertex AICloud Run
02

Selected work

Numbers are measured, links are live.
Live system · soloJan 2026 – present

Production Chat Agent

The agent you can talk to on this site

A FastAPI and React agent that answers questions about my work. It streams over SSE, gates off-topic questions before the expensive call, reads a pasted job post, and keeps chatting when memory, database or tracing are down.

83% of claims checked verbatim95 tests, CI eval gate
  • Every sentence of an answer is stamped checked, inferred or unknown: a fast model proposes a verbatim profile quote, code confirms it and checks every number. First run: 52 claims, 83% checked verbatim, none unknown.
  • Replaced Google's SDK with raw async HTTP/2 after it hung the server, and traced a silent production hang to a cache deadlock by bisecting with diagnostic endpoints.
  • Oct 2026: found it had been silently down for months (retired model, closed billing); wrote the post-mortem, made model IDs configuration, added a health probe that makes a real call, a fallback model, signed sessions, quotas and an MCP interface.
Open source · soloOct 2026

mcp-trust-monitor

Open-source guard for MCP tool calls

A Python SDK and CLI that holds an agent's MCP tool calls when a server's tool definitions change, until an operator approves the new revision.

2 alpha releasesMIT licensed
  • Protected calls check approval, current metadata, launch identity and session freshness before dispatch. A changed definition, failed observation or quarantine blocks the call.
  • Background monitoring, a local operator console and decision-record export for audits.
YC × Cactus × DeepMind hackathon · team of 4Apr 2026

HVAC Copilot

On-device voice and vision agent for AR glasses

A field assistant for HVAC technicians running Gemma 4 E4B fully on-device. One runtime serves three clients: Chrome, iPhone Safari and Rokid AR glasses over WebRTC.

TTFT 217 ms bare vs 3.9–4.5 s loaded6 tools, on-device
  • Built the assistant runtime and a six-tool function-calling surface, with the on-device/online boundary enforced in the dispatcher, not just the prompt.
  • Wrote a streaming sanitizer for Gemma 4's three tool-call formats so raw tool syntax never reaches the user.
  • Profiled the latency end to end and traced the 4 s first token to CPU prefill of the system prompt and tool schemas.
Hackathon · built end to endFeb 2026

FunctionGemma Hybrid Router

2nd place, Cactus × Google DeepMind hackathon

Routes function calls between FunctionGemma-270M on-device and Gemini Flash-Lite in the cloud, deciding before inference because the small model reports high confidence even when it's wrong.

0.99 F1548 ms average70% on-device
  • Pre-routing difficulty scoring, a validation gate that catches hallucinated tool names, and deterministic argument extraction for the 270M model's broken integers.
  • Proved that on-device beats a perfect cloud answer once local F1 passes 0.48 under the contest's scoring, and built around that.
BigThink × World Labs hackathon · soloMar 2026

Auto-Eval3D

Winner, Best Scientific & Engineering Application

Scores whether a generated 3D world stays consistent across viewpoints: prompt to World Labs Marble world, four inside-out views, then structured multi-view reasoning in Gemini.

Track winner4 views, 1 score, full trace
  • Built the inside-out WebGL camera rig and a three-phase reasoning prompt that flags Janus artifacts, warping, scale shifts and floating geometry.
  • After the win I re-audited the traces. The demo's 1/10 score was judging a badly placed camera, not a bad world, so I rewrote the capture rig.
Take-home · solo · code on requestApr 2026

Top-K k-NN kernel for the Cerebras WSE-2

Take-home, written in CSL

A distributed Top-K nearest-neighbour kernel across a grid of processing elements on the wafer-scale engine simulator: vectorized distance compute, a bounded heap per element, then a merge.

6/6 grader cases2.68× fewer cycles
  • 2.68× fewer cycles than my first correct version, with bit-identical results on all six cases.
  • Measured a radix-sort variant at +24% cycles and rejected it.
UMD study · 3 authors · my methodMar – May 2026

LLM inference config tuning

An LLM agent that tunes vLLM serving configs

Multi-objective tuning of LLM serving: throughput, time to first token and time per output token. I designed and built the LLM-agent method, one of five compared head to head against Bayesian optimization on shared benchmark infrastructure.

9 vLLM knobs4 GPU allocations5 methods compared
  • Same quasi-random warm start as the Bayesian baselines, so the comparison is fair by construction. Then one model call per trial, seeing the full sorted benchmark history, proposes the next config.
  • Every LLM-proposed config passes type coercion, schema repair and a hallucinated-key guard before it reaches a GPU. Benchmarked on Slurm.

MedBill

Anthropic × Maryland hackathon · solo · Apr 2026

Redacts patient information on-device, then uses Claude on the redacted text to find overcharges and draft a dispute letter. A deterministic gate, not the model, decides No Surprises Act claims.

RegSentinel and SDS Intake Agent

Prototypes · solo · May 2026

A chemical-compliance agent on the Claude Agent SDK and an MCP server, with cited answers and human escalation for high-risk calls, plus a Safety Data Sheet extractor where code verifies what the model extracts. Built as prototypes of 3E's problem space before I applied. The offer followed.

PathGuard

UMD × Ironsite hackathon · team lead of 3 · Feb 2026

An on-device vision-language model describes the scene and drives zero-shot hazard detection (Grounding DINO, SAM2, Depth Anything V2) inside the worker's walking path, with audio alerts.

Retail Catalog Voice AI

Product build · solo · Apr 2026

Voice in, spoken answer out, grounded in the catalog with no invented bottles: local parsing of the shopper's constraints filters the catalog before Gemini sees it.

English2SQL

Team project · backend and RAG · 2024

Natural language to SQL over Postgres with schema-aware prompting, evaluated on execution accuracy (+9% on a 120-query set).

Now buildingMoonNod, a campus marketplace for used items and sublets, built with a team of four for my NLP course.

03

How I work

Prove the measurement before trusting the result

At 3E I calibrated the LLM judge against blinded human graders before reading a single score. The rubric, not the judge model, turned out to be the accuracy lever. EvoRank gates the search with a pre-spend headroom check and re-scores every winner on held-out queries.

The harness is part of the latency budget

My on-device agent answered in 217 ms with a bare prompt and 3.9 to 4.5 s once about 935 tokens of system prompt and tool schemas were loaded. I traced it to CPU prefill. Tool definitions are not free.

Let the model handle language and let code make the decisions

In MedBill the model reads the bill, but a deterministic gate decides whether a No Surprises Act claim stands. In the SDS agent the model extracts, and CAS check-digit math verifies. The check digit doesn't hallucinate.

First author, equal contribution
2026

EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines

GenAIECommerce 2026, a RecSys 2026 workshop (Minneapolis) · Rayhan Patel and Shabaz Patel (equal contribution)

An LLM evolves complete learning-to-rank pipelines (features, models, losses, ensembles) for multi-objective hotel search. The contribution is the audit: a pre-spend headroom gate and a transfer audit on held-out queries.

All three runs beat an Optuna-tuned LambdaMART on relevance (best +0.0057 NDCG@10, p < 0.0001, 59,902 held-out queries), and the best run beat it on every objective. A one-shot late submission would have placed 20th of 340 on the original Kaggle test set.
Co-author
2023

Building Domain-Specific LLMs Faithful to the Islamic Worldview: Mirage or Technical Possibility?

NeurIPS 2023 Muslims in ML workshop · Shabaz Patel, Hassan Kane, Rayhan Patel

Tested whether prompting, RAG over 54K hadith, or GPT-3.5 fine-tuning makes LLMs answer faithfully to scholar answers. Built the evaluation (BERTScore and embedding distance) and presented the paper.

Few-shot prompting was the most reliable across models. Fine-tuning on hadith plus IslamQA gave the best F1 (0.402).
05

Recognition

  • 2nd place, Cactus × Google DeepMind Hackathon$1,000 in Gemini API credits
    Feb 2026
  • Winner, Best Scientific & Engineering ApplicationBigThink × World Labs Hackathon
    Mar 2026
  • AWS re:Invent All Builders Welcome GrantLas Vegas, Nov 30 – Dec 4
    2026
  • AWS Certified AI PractitionerAmazon Web Services
    2026
  • Finalist, Pear VC + OpenAI HackathonSan Francisco
    2024
06

Education

  • University of Maryland, College ParkM.S., Applied Machine Learning · Natural Language Processing, Computing Systems for ML, Optimization, Algorithms and Data Structures
    Aug 2025 – May 2027
  • B.S. Abdur Rahman Crescent Institute of Science & TechnologyB.Tech., Computer Science and Engineering · Chennai, India
    Aug 2019 – May 2024
07

Toolbox

Agents and LLMs
LLM agentsAgent memoryMCPClaude Agent SDKFunction callingLLM evaluation and benchmarkingLLM-as-a-judgeRAGGuardrailsOn-device LLMs
Backend
Python (AsyncIO)FastAPISSEPydanticpytestPostgreSQLSupabaseTypeScriptReact
Cloud and infra
AWS (Lambda, DynamoDB, Bedrock)GCP (Vertex AI, Cloud Run, Cloud SQL)DockerGitHub ActionsRailwayVercelCloudflareLinuxSlurm
ML and systems
PyTorchscikit-learnLightGBM / XGBoostLearning to rankvLLMCUDA kernels (Numba)Cerebras CSL
Tools
Claude CodeLangfuseGit
Open source
Mem0, merged PR (2023), metadata support for added data sources

Hiring for agents, evals or ML systems from June 2027?

Email is fastest. I reply within a day. Or ask my agent first and email me what it couldn't answer.

rayhanbp@umd.edu LinkedIn