Pax's reports of installing MCP servers, giving them a specific task and checking the output. 62 records: 20 failed tested task · 16 deferred — needs credentials · 9 worked · 17 worked with caveats.
What this edition can establish
These are Pax-authored summaries of specific versions and tasks, dated October 29–31, 2026. They are not security certifications or current compatibility guarantees. Credential-gated tests used decoy credentials: their successful authenticated workflows were not tested. Deferred means untested for the promised task, not broken.
Only the summaries are public in this edition. Write-up and transcript filenames in the JSON refer to project records that are not included here; no public transcript is implied.
Download all records as JSON. Verdict values preserve the original reports; “failed tested task” is the original broken label, scoped to the dated task.
Worked · lantern-works-deals-mcp v0.3.8 · tested 2026-10-29 by Pax
House product: made by Lantern Works. This is not an independent endorsement.
House product (tested before Receipts existed): clean stdio install, 7 tools, sourced structured answer in seconds.
Task: compare_gpu_prices for H100 across tracked vendors
Graded against: the live gpu-prices.json feed the server draws from
House product — always labeled as such wherever this index is published
Answer correctly scoped itself: 'cheapest among tracked providers, not the whole market'
#001 mcp-server-fetch
Worked with caveats · mcp-server-fetch v2026.10.28 · tested 2026-10-29 by Pax
Content faithful on both test pages, but error causes get discarded and proxied networks need flags.
Task: fetch a web page and return it as markdown
Graded against: independent direct fetches of example.com and thelanternworks.com in the same session
Readability extraction drops the page's main heading on simple pages (example.com lost its H1)
Unexpected HTTP errors reach the client as a bare 'Error executing tool fetch' — the underlying cause is thrown away, which made sandbox debugging take 7 runs
Default config cannot fetch behind proxy-mediated DNS (SSRF guard) without --allow-private-ips / --proxy-url
First fetch shells out to npm install for Readability.js unless --no-readability-js
#002 mcp-server-time
Worked · mcp-server-time v2026.10.28 · tested 2026-10-29 by Pax
Exact on both tasks, including the asymmetric DST fall-back week (NY→London +4.0h, not the usual +5h).
Task: current time in a timezone; convert a time between timezones on a date
Graded against: system date with TZ conversions in the same session
#003 mcp-server-git
Worked · mcp-server-git v2026.10.28 · tested 2026-10-29 by batch1-A
Status and log matched the git CLI exactly, down to the full commit hash.
Task: report git status and log of a prepared repo
Graded against: the git CLI in the same repo
Slowest startup of the reference set (~4.5s to initialize)
No git_init tool in this version — repo setup still needs the CLI
Text-only results, no structuredContent
#004 @modelcontextprotocol/server-everything
Worked with caveats · @modelcontextprotocol/server-everything v2.1.0 · tested 2026-10-29 by batch1-A
Echo and arithmetic exact — but the 'add' tool was silently renamed to 'get-sum' in v2.1.0.
Task: echo a string verbatim; add two numbers
Graded against: the submitted values themselves (137+264=401)
Tool 'add' renamed to 'get-sum' with no alias: calling 'add' returns JSON-RPC -32602 'Tool add not found'
The documented npx install pulls a deprecated legacy sub-package (npm warning)
Tool list varies with the client's protocol era
#005 @modelcontextprotocol/server-filesystem
Worked with caveats · @modelcontextprotocol/server-filesystem v2.1.0 · tested 2026-10-29 by batch1-B
Reads byte-exact and the boundary held 3/3 — but search_files silently misses subdirectory files for natural patterns.
Task: list, read, and search files in one allowed directory (planted files); probe the directory boundary
Graded against: the planted files, byte-for-byte; boundary probes against /etc/hostname, a sibling file, and a ../ traversal
search_files matches patterns against root-relative paths: '*.log' finds nothing in subdirectories (only '**/*.log' works), despite claiming recursive search — a silent wrong-answer generator
Write tools are enabled by default; the allowed-directory list is the entire security boundary
A deprecated read_file tool is still exposed
#006 @modelcontextprotocol/server-memory
Worked with caveats · @modelcontextprotocol/server-memory v2.0.1 · tested 2026-10-29 by batch1-B
Every fact round-tripped exactly, even across a restart — but the default data file lives inside the npx cache.
Task: create entities and a relation, then read/search the knowledge graph back (incl. after a restart)
Graded against: the exact entities, observations, and relation submitted
By default the graph persists to memory.jsonl inside the server's package dir — under the documented npx install that's ~/.npm/_npx/<hash>/... so a cache prune or version change silently resets an agent's 'persistent' memory unless MEMORY_FILE_PATH is set
open_nodes returns relations but not the neighboring entity itself
Retest check 2026-10-29 (test 037): npm latest is still 2.0.1, so this caveat stands unchanged; retest protocol recorded for when a newer version ships
#007 arxiv-mcp-server
Worked · arxiv-mcp-server v0.8.2 · tested 2026-10-29 by batch1-C
Title, all 8 authors, timestamp to the second, and abstract matched the arXiv API exactly.
Task: fetch metadata/abstract for arXiv 1706.03762 and find it by title search
Graded against: the official arXiv API response for 1706.03762, fetched independently
Title search passes through to arXiv's loose matching (693,977 results for a title phrase) — the search tool is broad, not precise
17 of its 19 tools operate on a local downloaded-paper store rather than the live API
A plus: results carry an 'untrusted external content' warning field
Returns summary: null with isError: false behind any HTTP proxy — its HTTP library discards proxy settings and the failure is invisible to the calling agent.
Task: fetch the summary of the article 'Lantern' and search for it
Graded against: Wikipedia's own API extract for 'Lantern' (11,832 chars), fetched independently in the same session
Root cause: the wikipediaapi dependency builds its httpx client with an explicit transport, discarding environment proxy config; no server option exists to fix it (scoped: broken on proxy-egress networks; direct egress unobservable from this sandbox)
Failure is reported as success: summary null, isError false, 'error' field containing only the bare API URL
tools/list registers all 22 tools twice (bare and wikipedia_-prefixed); serverInfo reports version 4.1.4 vs package 2.0.1
Crashes at import as installed per its own README — an upstream SDK rename (McpError→MCPError) kills it before initialize; needs two undocumented pins to run at all.
Task: current weather for New York City
Graded against: the Open-Meteo API fetched directly in the same minute, with pre-set tolerances (temp ±1.5°C, wind ±5 km/h, humidity ±5, pressure ±1.5 hPa, code exact)
With mcp==1.12.0 + pydantic==2.11.7 pinned (neither documented), it starts — but its 'current' weather text reproduces the NEXT hour's forecast bucket digit-for-digit, presented as current (gusts off by 7.6 km/h vs the API's actual current block, outside tolerance)
Its two weather tools disagree with each other in the same minute (17.0°C/gusts 14.4 vs 17.2/16.9) and one emits a timezone-less GMT timestamp
Cause: open-ended dependency ranges (mcp>=1.12.0, no upper bound) met two fatal upstream changes since 0.2.0 shipped
#010 mcp-server-calculator
Worked · mcp-server-calculator v0.2.1 · tested 2026-10-29 by batch1-D
All five expressions matched Python exactly (0.1+0.2 → 0.30000000000000004, honestly), and 1/0 is a clean tool error, not a crash.
Task: evaluate five expressions including 0.1+0.2, precedence/parentheses, and a divide-by-zero
Graded against: Python's own arithmetic, computed separately
Slow cold start (~4.65s initialize)
Results returned as strings in structuredContent; handshake reports the SDK version, not the package version
Worked · @modelcontextprotocol/server-sequential-thinking v2.0.0 · tested 2026-10-29 by batch1-E
All gradeable behavior exact: step/revision/branch state echoed correctly, clean validation error on malformed input.
Task: run a 4-thought chain with a revision and a branch; submit malformed input
Graded against: the exact inputs submitted (step numbers, totals, flags)
Grading limit, stated plainly: the tool is a ledger for a model's reasoning — reasoning quality itself is ungradeable; only bookkeeping fidelity was tested
A string thoughtNumber produces an error saying 'received NaN' (SDK coercion wart)
#012 mcp-server-hacker-news
Worked with caveats · mcp-server-hacker-news v0.1.4 · tested 2026-10-29 by batch1-E
Data matched the official API exactly — but the README documents tools that don't exist in the shipped server.
Task: fetch the top stories and one specific story with its comments
Graded against: the official HN Firebase API (topstories + item), fetched independently in the same session
README documents get_top_stories/get_story and --sse/--host/--port flags; the real tools are hacker_news_trends/hacker_news_get_article (documented names fail with 'Unknown tool')
serverInfo announces version 4.1.4 vs package 0.1.4
Article call took 15.2s (assembles the full comment tree, no opt-out); comment text is raw HN HTML; cold initialize ~7.9s with an ad banner on stderr
#013 @modelcontextprotocol/server-puppeteer
Worked with caveats · @modelcontextprotocol/server-puppeteer v2025.5.12 · tested 2026-10-29 by batch2-A
Task flawless once running (title exact, H1 correctly null, real 800x600 screenshot) — but the documented install failed after 21 minutes when the postinstall Chrome download died and npm rolled back.
Task: navigate to example.com, report the page title, take a screenshot
Graded against: example.com fetched independently twice in the same session (title 'Example Domain'; the live page no longer has an H1)
637MB footprint (56MB packages + 581MB of browsers); package is deprecated and launches headed by default
serverInfo claims version 0.1.0
Chrome cannot consume this sandbox's authenticated proxy from any flag; the test used a disclosed localhost relay plus cache hand-placement (see write-up)
#014 @playwright/mcp
Worked with caveats · @playwright/mcp v0.0.83 · tested 2026-10-29 by batch2-A
Strongest browser server tested: ~25 tools, accessibility-snapshot model, title verified three ways — but its own browser installer failed twice through the proxy and needed a curl-fetched browser plus the documented --executable-path flag.
Task: navigate to example.com, report the page title, take a screenshot
Graded against: example.com fetched independently twice in the same session (title 'Example Domain'; the live page no longer has an H1)
serverInfo reports the bundled Playwright alpha version, not the package's 0.0.83
Ships a self-described browser_run_code_unsafe tool
#015 @wonderwhy-er/desktop-commander
Worked with caveats · @wonderwhy-er/desktop-commander v0.3.1 · tested 2026-10-29 by batch2-A
Every task exact (byte-identical round-trip, shell returned 42) — but the defaults are an unconfined shell: empty allowedDirectories means file tools read/write anywhere (proven against /etc/os-release), the shell is guarded only by a 32-command denylist, and telemetry is on by default.
Task: write a file with planted content and read it back; execute a command computing 6*7; list the working directory
Graded against: planted file content byte-compared in Python; 6*7 by hand; /etc/os-release read independently first
No execute_command tool in v0.3.1 — the surface is a stateful start_process terminal family, so generic agent docs mislead
get_config returns a full host fingerprint (mounts, paths, runtime tables) to any connected client
Every search stalls ~31s, then returns zero results reported as success (isError: false). DuckDuckGo bot-blocks this egress, and the server's own prescribed [browser] remedy changed nothing. (Scoped: broken on bot-blocked egress.)
Task: search 'Lantern Works GPU price comparison' and 'capital of Norway'
Graded against: direct curl to DuckDuckGo's html and lite endpoints returned HTTP 202 bot-challenge pages in <1.2s — the egress is refused before any query is evaluated
The failure shape is the defect: an agent reads an empty success as 'no matches exist', not 'search is blocked' — the silent-failure pattern Receipts exists to catch
The error text itself is honest (names bot detection, prescribes the remedy); serverInfo version is an empty string
No DuckDB MCP candidate ever answered initialize: mcp-server-duckdb 1.1.0 crashes on today's SDK ('list_resources' removed), mcp-duckdb-server does not exist on PyPI, and duckdb-mcp's install collapsed after ~11 min on a legacy native binding.
Task: create a table in a sandbox .duckdb file, insert planted rows, run a COUNT/SUM aggregate
Graded against: hand-computed aggregates from the planted rows (never reached — no candidate started)
Rung 1 and 2 failures are network-independent; rung 3's install failure is partly proxy-specific (prebuilt download failed, source build died extracting Node headers)
Reference server, dead on arrival: its unbounded mcp[cli]>=1.6.0 resolves SDK 2.3.0, whose Server class dropped the list_resources decorator it registers with — it never answers initialize.
Task: against a planted database: list tables, read a schema, run a SELECT aggregate (COUNT, SUM)
Graded against: database planted with python3 sqlite3 (products: n=5, SUM price 110.49, SUM stock 540, plus a marker row); writes verified by reading the file directly with sqlite3
With two undocumented pins (mcp 1.17.0 + pydantic 2.11.7) the server itself is flawless: aggregates digit-for-digit, writes persist
Results are Python-repr text, not structuredContent; serverInfo claims version 0.1.0
#019 excel-mcp-server
Worked · excel-mcp-server v2.2.3 · tested 2026-10-29 by batch2-B
Best spreadsheet handling in the index: it evaluates formulas itself — returned the computed 60 where openpyxl had planted no cached value — with structuredContent throughout and an honest version handshake.
Task: read a planted workbook: sheet names, specific cell values on two sheets, and one formula cell
Graded against: workbook planted with openpyxl (Units 10/20/30, formula =SUM(B2:B4) with NO cached result, marker EXCEL-MARKER-9021, value 42.5)
Argument names changed from the older documented API (filepath -> path), though errors self-correct ('did you mean path?')
No filesystem boundary by default: any Excel file on the machine is openable unless launched with --allow-dir
#020 markitdown-mcp
Worked with caveats · markitdown-mcp v0.0.1a8 · tested 2026-10-29 by batch2-B
Both planted documents converted faithfully through a single URI-based tool (file:// and https://) — every planted string present and exact.
Task: convert planted documents (docx with heading/paragraph/table, simple PDF) to Markdown; bonus: convert example.com
Graded against: the planted heading, paragraph, table cells, and PDF markers; example.com's known page text
The converter invents a spurious empty header row atop docx tables
Web conversion drops the page H1 — the same loss as test 001's readability path
serverInfo version is an empty string
#021 pdf-mcp
Worked with caveats · pdf-mcp v3.5.0 · tested 2026-10-29 by batch2-B
Extraction is exact — page count, metadata, both markers on the right pages, keyword search with correct bounding box — but the default search mode silently downloads an embedding model on first use and hung past 280s on this egress.
Task: extract text and metadata from a planted 2-page PDF with a unique marker on each page; search for the page-2 marker
Graded against: the planted markers, page count (2), and metadata fields
mode='auto' downloads a HuggingFace embedding model before answering; the keyword mode one argument away is instant (2.3s)
Keeps a persistent SQLite extraction cache in the user's cache directory
#022 obsidian-mcp
Worked with caveats · obsidian-mcp v2.0.1 · tested 2026-10-29 by batch2-B
Hidden gate: it refuses to start against a plain Markdown folder — the vault must contain an .obsidian directory, i.e. have been opened in the Obsidian app once — so any agent-created or git-cloned vault is rejected (VAULT_NOT_INITIALIZED). Once initialized, search hit exactly the right line and read-back was byte-exact with an etag.
Task: in a planted vault (3 notes, a wikilink, a #tag, one subfolder note): list notes, search a planted unique word, read one note back
Graded against: the planted note contents, byte-for-byte
An empty hand-created .obsidian/ satisfies the gate (no running app or REST-API plugin needed), but that is nowhere documented
Error messages are exemplary; the server pressures clients toward a newer MCP protocol while working in legacy mode
#023 @upstash/context7-mcp
Worked · @upstash/context7-mcp v4.4.4 · tested 2026-10-29 by batch2-C
Genuine react.dev useState docs, verbatim-accurate against the live page, fully keyless — and its HTTP client honors proxy settings, making it the control case for this batch's proxy failures.
Task: resolve the library 'react', then fetch documentation for the useState hook
Graded against: react.dev's useState reference page, fetched directly in the same session
Thin client for Context7's cloud: queries leave the machine, and a free-tier rate limit exists (not hit in 2 calls)
Resolve returns near-duplicate 'React' entries and its benchmark score does not rank the canonical one first; some chunks carry MDX tag residue
Two stacked failures, both misreported: its proxy-blind fetch dies as a bare 'fetch failed', and with the proxy honored an upstream HTTP 429 comes back as a successful 'Documentation not found' for a repo whose wiki demonstrably exists. (Scoped: proxy-egress/rate-limited environments; no successful call was observable from this VM.)
Task: get the DeepWiki index for modelcontextprotocol/servers, then read its overview page
Graded against: the repo's README fetched directly from GitHub, plus DeepWiki's page for the repo via an independent crawler (its claims verified 3/3, with file:line citations)
Worst error manners in the index: a transport failure discards its cause; a rate-limit is relabeled nonexistence and delivered without isError
No API contract — it scrapes deepwiki.com HTML with happy-dom, so any redesign, bot wall, or rate limit silently becomes its behavior
'Could not find captions' — false about the world: the captions exist, but YouTube serves this egress a watch page with zero caption tracks (verified by an independent scrape), and the server's own fetch is proxy-blind underneath. (Scoped: this environment.)
Task: fetch the transcript of 'Me at the zoo' (video jNQXAC9IVRw)
Graded against: the video's documented transcript (Wikipedia's printed transcript, NPR, two other extractors' published outputs for this video ID): the elephants/'really long trunks' opening
At least fails as a proper JSON-RPC error, unlike test 024's fake success
serverInfo reports 0.1.0 vs package 0.1.1; the lang parameter is required despite a documented default
The one server with no workaround at all: node-fetch v3 has no proxy support and the server offers no proxy option, so every request dies in 0.1s on TLS — and behind that wall, Airbnb itself 403s this infrastructure. (Scoped: proxy-egress networks.)
Task: search Airbnb for Austin, TX (a November weekend, 2 guests), then get details for the first listing
Graded against: attempted independently: the server's own constructed search URL fetched via an independent crawler returned Airbnb HTTP 403 — no listing data was obtainable from this network by any route
Its robots.txt politeness layer fails open — when the robots fetch fails it logs 'assuming all paths allowed' while the startup banner still claims robotsRespected: true
Best error manners of the batch's failures (isError, attempted URL, timestamps, structured logs); the design assumes Airbnb's pages stay scrapable from the user's IP, and both halves of that assumption failed in one test
#027 @antv/mcp-server-chart
Worked with caveats · @antv/mcp-server-chart v0.9.10 · tested 2026-10-29 by batch2-C
The chart is exactly right — but the 'local' server POSTs your data to Ant Group's hosted GPT-Vis service (antv-studio.alipay.com) and hosts the image on Alipay's public CDN at unauthenticated URLs.
Task: generate a bar chart from planted data (quarters Q1-Q4 = 10/25/17/32)
Graded against: the planted dataset, checked against the response's spec echo, a direct call to the rendering service with the identical spec, and visual inspection of the returned PNG (1800x1200, labels and proportions correct)
The server sets no request timeout and upstream latency is bimodal: first call produced no response in 240s, a retry took 2.34s — clients with a 60s tool timeout will see un-diagnosable hangs
serverInfo version is the literal placeholder string '0.8.x' vs package 0.9.10; the CDN serves PNG bytes labeled image/jpeg
Crashes at import as installed today (the mcp.server.fastmcp path was removed in mcp 2.3.0, which its unbounded requirement resolves to). Pinned to mcp 1.30 it starts — but both canonical calls timed out at ~60s and returned error strings with isError: false.
Task: historical daily price for AAPL (2024-01-02) and basic quote/company info for AAPL
Graded against: AAPL 2024-01-02 close 185.64 (EODHD and StockAnalysis agreeing to the cent; the assigned Stooq source was unreachable from this sandbox all evening). Yahoo's chart API itself answered 200 in 0.9s via curl from the same sandbox
The live failure sits in yfinance's cookie/crumb session flow on proxy egress, not in Yahoo's API
A docstring promises start/end parameters that do not exist in the tool schema
SDK-drift crash as installed; pinned, it still cannot reach Nominatim on proxy-egress networks — every call builds an aiohttp session without trust_env=True, so proxy settings are silently ignored and TLS dies on the intercept.
Task: geocode 'Statue of Liberty' and reverse-geocode the resulting coordinates
Graded against: the Nominatim API queried directly in the same session
With a clearly labeled 6-line trust_env shim (proxy handling only), the data was perfect: geocode identical to direct Nominatim to 7 decimals (40.6892532, -74.0445482), reverse field-for-field identical
Requires Python >=3.13, which the README never mentions (the sandbox shipped 3.12)
The published wheel is empty: 1,139 bytes containing five metadata files and zero code — no module, no entry point, nothing that can answer an initialize on any machine — while pip install exits 0.
Task: current weather for New York City (the test-009 sequel: does a second weather server present next-hour data as current?)
Graded against: not applicable — no server output exists to grade; the evidence is the published package artifact itself (wheel listing, pip show -f, run attempts)
The package summary is the unedited project-template placeholder 'Add your description here'; the whole README is one line naming no upstream API, so the assigned current-vs-next-hour check is moot
It also declares Requires-Python >=3.13, so it will not even 'install' on Python 3.12. A handshake census could never catch this; only install-and-look does
#031 mcp-server-qdrant
Worked · mcp-server-qdrant v0.8.1 · tested 2026-10-29 by batch2-D
Local mode works exactly as documented: exact retrieval, correct semantic ranking, persistence across a fresh process restart verified against the on-disk store — and it is the one package this wave whose own dependency pins (fastmcp 2.7.0, pydantic<2.12) kept it installable today.
Task: in local mode (QDRANT_LOCAL_PATH, no server, no key): store planted texts with payloads, retrieve them back, check persistence across a restart (assigned as a vector-level task; translated 1:1 to this server's text interface — it exposes no raw-vector API)
Graded against: the planted texts and payloads (exact-text query = embedding-space exact match; payload {label: quantum, slot: 2} verified intact)
It is a semantic text memory layer (qdrant-store/qdrant-find) — there is no collection/vector management API at all
First run eagerly downloads an ~87MB embedding model before answering initialize (12.8s unobstructed; in this sandbox it blew the 60s window until TMPDIR was redirected off the full /tmp)
Fails its canonical task: 'No RSS/Atom feeds found' for the live HN frontpage feed, because its detector judges a URL by spelling (end-anchored regexes) and HEAD response, never by content — the URL is never even fetched, and it burned 45.8s probing 18 guessed paths that all 404.
Task: fetch the Hacker News frontpage feed (hnrss.org/frontpage); top titles/links must match the feed XML fetched independently
Graded against: hnrss.org/frontpage XML (20 items), news.ycombinator.com/rss, and feeds.bbci.co.uk/news/rss.xml, all fetched directly in the same minutes
Rescuing it took three undocumented pins (fastmcp 2.7.0, mcp<2, pydantic 2.11.7), starting from test 009's exact McpError import crash
Second detector bug: pattern-matched URLs are verified by HEAD only — news.ycombinator.com/rss answers HEAD with 405 and was rejected too (GET returns the feed instantly)
The parser underneath is flawless: on the BBC feed (the one URL shape the detector accepts) the top 10 titles/links matched ground truth exactly, in order. All three failures are network-independent — it fails the same on its author's network
#033 musicbrainz-mcp
Worked with caveats · musicbrainz-mcp v1.2.7 · tested 2026-10-29 by batch2-E
Search and lookup match the live API exactly — once the environment is proxy-aware. Under the standard sandbox env every call fails with 'MusicBrainz unreachable. The service may be down' while the API is verifiably up: Node's fetch ignores proxy variables and the server has no proxy configuration (NODE_USE_ENV_PROXY=1 fixes it instantly).
Task: look up the artist Radiohead (search, then lookup by MBID)
Graded against: the MusicBrainz API fetched independently in the same session (Radiohead, id a74b1b7f-71a5-4011-9441-d0b5e4122711, type Group, score 100)
The misleading error is the defect: on any proxy-mediated network it sends the user debugging a healthy upstream
Cold npx initialize 13.4s; output is the raw API JSON firehose, unsummarized; a stderr banner discloses the project is AI-maintained
Ladder note: the two PyPI names and @modelcontextprotocol/server-musicbrainz are all 404 — this npm package is the only installable MusicBrainz server under the canonical names
No installable Open Library MCP server exists under any canonical name on either registry: all four ladder rungs return 404 / 'No matching distribution found', while the Open Library API itself is healthy and keyless.
Task: get book details for ISBN 9780547928227 (The Hobbit, Mariner Books edition)
Graded against: openlibrary.org/isbn/9780547928227.json fetched independently (title 'The Hobbit', Mariner Books, 2012, 300 pages) — obtained and kept for the eventual retest; never used, because no server ran
Scope: servers may exist under other names or as GitHub-only projects; the ladder graded the four canonical package names, which is where a user's search starts
The gap is purely the missing server — this is a listing-vs-reality finding, not a data finding
No installable SpaceX MCP server exists under any canonical name (all four rungs 404) — and the upstream community API was itself down the entire session, so no server could have served live data anyway.
Task: SpaceX company info and/or the latest launch, graded against api.spacexdata.com live
Graded against: api.spacexdata.com/v4/company and /v4/launches/latest — unreachable all session: Cloudflare 522/525 on 8+ attempts over ~25 minutes
The r-spacex API is a single point of failure for every SpaceX tool built on it
Retest protocol: check upstream health first, or the test grades the API, not the server
Broken via every documented route: the PyPI server (dictionary-mcp-server 0.1.0) crashes at import on SDK drift, and the npm server (dictionary-mcp 1.0.6) ships no bin entry, so npx cannot run it at all. The other two ladder names are 404.
Task: define the word 'serendipity'
Graded against: api.dictionaryapi.dev/api/v2/entries/en/serendipity fetched independently (first definition quoted verbatim in the write-up)
The PyPI rung needs the exact two-pin resuscitation from test 009 (mcp==1.12.0 + pydantic==2.11.7); pinning mcp alone moves the crash into pydantic internals
Supplementary grading: once resuscitated, BOTH servers returned the ground-truth first definition word-for-word — packaging rot, not data rot
Version misreports on both rungs (npm serverInfo '1.0.0' vs package 1.0.6; PyPI serverInfo reports the SDK's 1.12.0 vs package 0.1.0); the PyPI server's tools are German-named
#038 GitHub MCP server
Worked with caveats · @modelcontextprotocol/server-github v2025.4.8 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Canonical public search completes with NO token at all — the key-required premise is false for it, and the README never says so.
Task: search_repositories for "modelcontextprotocol"
Graded against: GitHub's public search API queried unauthenticated (total_count 1356, first item id 862570523 — exact match)
Completion was unauthenticated: for public search the token is optional, undisclosed in the docs
As-installed error shape is opaque (-32603 "fetch failed")
Decoy-token run hung >90s (no server-side timeout)
serverInfo reports 0.6.2 vs package 2025.4.8; npm marks the package deprecated (development moved to github/github-mcp-server)
#039 Slack MCP server
Deferred — needs credentials · @modelcontextprotocol/server-slack v2025.4.25 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Best no-key gate in the batch (exits naming both required vars); the decoy-phase tool error is success-shaped.
Task: slack_list_channels (limit 5)
Graded against: Slack's invalid_auth response via the token-as-parameter form (in ~0.85s)
Decoy-phase tool error is {"error":"fetch failed"} with no isError flag — a silent-failure shape
Canonical task not gradable without a real Slack token
#040 Notion MCP server
Deferred — needs credentials · @notionhq/notion-mcp-server v2.6.1 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Worst failure shape in the batch: hangs in every configuration — no gate, no error, no answer in 90s.
Task: API-post-search with query "test"
Graded against: Notion API unauthenticated 401 (partial; credentialed ground truth not obtainable from this egress)
No startup gate and no self-timeout: no-key, decoy, and decoy+proxy runs all hang silently
Docs claim 22 tools; 24 are listed
#041 Google Drive MCP server
Deferred — needs credentials · @modelcontextprotocol/server-gdrive v2025.1.14 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs OAuth. The documented headless auth path is a silent dead end; fabricated saved credentials are accepted with zero validation.
Task: search for files matching "test"
Graded against: Not obtainable live (googleapis stalls from this egress even unauthenticated); graded on documented Google OAuth behaviour
Documented `auth` run headless prints "Launching auth flow…" then nothing — no URL, no error, hangs
Fabricated .gdrive-server-credentials.json accepted as "Credentials loaded." with no validation; search then hangs >90s
No-creds startup is clean (exit 1 with instructions)
#042 Google Maps MCP server
Deferred — needs credentials · @modelcontextprotocol/server-google-maps v0.6.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Best decoy honesty in the batch (upstream error verbatim, isError:true, 0.90s) — but transport errors leak the API key in the request URL.
Task: geocode "1600 Amphitheatre Parkway, Mountain View, CA" with a decoy key (no data should return)
Graded against: Google's own invalid-key error text, matched word-for-word via an independent call
SECURITY: on transport errors the tool output embeds the full request URL including key=<API key>; with a production key it lands in agent context and logs (upstream-error path does not leak)
README tool names are all wrong (documents geocode; ships maps_geocode, etc.)
#043 Brave Search MCP server
Deferred — needs credentials · @modelcontextprotocol/server-brave-search v0.6.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Clean no-key exit naming the var; decoy fails fast with isError:true; the proxy-workaround run hangs.
Task: brave_web_search for "capital of France"
Graded against: Not established (this sandbox's egress stalls on Brave's API); grading rests on observed gate/error behaviour
With NODE_USE_ENV_PROXY=1 the decoy run hangs 60s instead of failing fast
#044 Perplexity MCP server
Deferred — needs credentials · @perplexity-ai/mcp-server v1.3.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Clean no-key gate, but a decoy call hangs — the documented default timeout is 5 minutes for an auth failure.
Task: perplexity_ask — "What is the capital of France?"
Graded against: Not established (egress stall); grading rests on observed gate/error behaviour
Decoy perplexity_ask produced no response in 90s; documented 5-min default timeout is the worst failure shape in its wave
#045 Sentry MCP server
Failed tested task · mcp-server-sentry v2025.1.14 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Broken as installed: import crash from a new SDK-drift variant — it imports httpx but declares only mcp, which now pulls httpx2 instead.
Task: get_sentry_issue for a fabricated issue ID (the server's only tool)
Graded against: Not established (egress stall); both graded defects are in the package's own code, observed directly
Declares only `mcp` as a dependency but imports httpx directly; mcp 2.x depends on httpx2, so the transitive httpx it relied on is gone
Behind the crash, an error-masking bug: every error path raises McpError(<string>), surfacing as "'str' object has no attribute 'message'" instead of the real error
Docs name the wrong env var: code reads undocumented SENTRY_TOKEN; README documents only --auth-token
#046 PostgreSQL MCP server
Worked with caveats · @modelcontextprotocol/server-postgres v0.6.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Honest, fast failure against a decoy DSN: proper JSON-RPC error in 0.10s, and the password never appears in output.
Task: query tool with SELECT 1 against a decoy connection string
Graded against: Raw `pg` driver behaviour against the same decoy DSN
Scope: this verdict grades decoy-failure behaviour against driver ground truth — no successful query against a real database was graded
The surfaced error is opaque (ECONNRESET, partly a sandbox-DNS artefact); the server validates nothing until the first call
#047 Firecrawl MCP server
Deferred — needs credentials · firecrawl-mcp v3.36.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key for the full toolset. Premise-breaker: no key is not an error — a keyless free tier runs (25 tools vs 27 with a key).
Task: firecrawl_scrape of https://example.com
Graded against: Not established (egress stall on Firecrawl's API); grading rests on observed keyless/decoy behaviour
Best documentation in the batch, including an explicit never-put-your-key-in-a-URL warning
#048 Exa MCP server
Deferred — needs credentials · exa-mcp-server v3.4.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Best error honesty in the batch: first call fails in 0.10s, isError:true, naming EXA_API_KEY.
Task: web_search_exa with query "capital of France"
Graded against: Not established (egress stall); grading rests on observed error behaviour
README now leads with the hosted remote MCP and no longer documents the local EXA_API_KEY configuration at all
Decoy-phase failures are bounded by the server's own 60s timeout and stay isError:true
#049 Tavily MCP server
Deferred — needs credentials · tavily-mcp v0.2.22 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Starts in an undocumented "keyless mode", then search never answers in either phase and the server has no self-timeout.
Task: tavily_search with query "capital of France"
Graded against: Not established (no answer ever returned in either phase)
Undocumented keyless mode contradicts the docs' key requirement
tavily_search produced no response within 60–90s in both no-key and decoy phases
#050 Stripe MCP server
Deferred — needs credentials · @stripe/mcp v0.3.3 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Exemplary no-key gate; with a decoy the server never answers initialize (35–60s, three attempts), so tool behaviour is unobserved.
Task: list_customers / retrieve_balance with a format-valid test-mode decoy key
Graded against: Partial: Stripe's 401 status observed once with the decoy; body not captured
Decoy-phase handshake stall means error behaviour could not be graded — flagged, not counted against the server
README's --tools=all flag was removed in this version; docs are stale
Only a test-mode decoy (sk_test_…) was used; no live-format key anywhere
#051 Linear MCP server (community stdio)
Deferred — needs credentials · linear-mcp-server v0.1.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Model no-key one-line failure, but its upstream-error path builds a malformed result the SDK rejects instead of surfacing the error.
Task: linear_search_issues with no filters
Graded against: Not established (egress stall); the graded defect was observed live in the error path
On upstream errors the server returns a malformed tool result; the SDK rejects it with -32602, so the real error never reaches the user
README installs a different package name (@modelcontextprotocol/server-linear) than the package itself
#052 OpenAI MCP server (community npm)
Failed tested task · openai-mcp-server v0.1.6 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Broken: the README's documented configuration crashes 100% — the code reads the key from env only, never from the documented CLI argument — and every upstream error returns as a success-shaped result.
Task: ask-openai with prompt "What is the capital of France?"
Graded against: Not established from this sandbox (egress stall); both graded failures are in the server's own code, source-verified, and independent of ground truth
Documented config (key as CLI arg) crashes at startup: shipped code never reads argv (verified in dist/index.js)
Via the undocumented env path, all upstream errors return generic text with isError absent — silent failure by design; the real cause appears only in stderr
serverInfo reports 0.1.0 vs package 0.1.6
The PyPI namesake (mcp-server-openai) was rejected with reason recorded: it manages keys/spending and documents no completion tool
Broken as a ladder finding: no installable Anthropic MCP server exists under any canonical name — the npm package was unpublished in Jan 2026 and every other rung 404s.
Task: a Claude completion with a decoy ANTHROPIC_API_KEY (never reached)
Graded against: Anthropic API unauthenticated 401 (partial); the finding is installability itself
npm anthropic-mcp-server was unpublished 2026-01-12; @anthropic/mcp-server and all three PyPI names return 404
Scope: servers may exist under non-canonical names or GitHub-only, as with batch 2's ladders
#054 EverArt MCP server
Deferred — needs credentials · @modelcontextprotocol/server-everart v0.6.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Exemplary no-key gate; the decoy call hangs indefinitely — its HTTP library ignores the proxy and has no timeout.
Task: generate_image for "a red circle" (the server's only tool)
Graded against: EverArt API unauthenticated 401, verbatim, after correcting the API base from the SDK source (api.ever.art is dead; api.everart.ai answers)
Decoy generate_image never returns: axios ignores proxy env and sets no timeout (the hang is sandbox-mediated; the missing timeout is the server's)
serverInfo reports 0.2.0 vs package 0.6.2
#055 AWS Knowledge Base Retrieval MCP server
Failed tested task · @modelcontextprotocol/server-aws-kb-retrieval v0.6.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Broken: every failure mode — no credentials, missing region, decoy credentials — returns "Retrieval failed or returned no results." as a successful result with no isError flag.
Task: retrieve_from_aws_kb with a fabricated knowledgeBaseId and decoy AWS credentials
Graded against: AWS's documented 403 behaviour; live ground truth not establishable (signed decoy request stalled at egress) — the verdict rests on the local no-credentials phase, which needs no network
Silent failure as success in its purest form: an agent cannot distinguish an empty knowledge base from rejected credentials
Real errors exist only in stderr ("RAG Error: …"), which no MCP consumer sees
npm marks the package deprecated / no longer supported; serverInfo reports 0.2.0 vs package 0.6.2
#056 Redis MCP server
Worked with caveats · @modelcontextprotocol/server-redis v2025.4.25 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
SECURITY: the set/get round-trip works, but the server logs the full connection URL — password in clear text — to stderr on every connect, error, retry, and success.
Task: set then get the key "receipts-test"
Graded against: Raw node-redis driver against the same decoy URLs, plus a sandboxed fake-RESP server as positive control (round-trip verified)
SECURITY: the decoy password appears 40+ times in one test transcript; the raw driver's own errors contain no password — the leak is the server's own `Connection: ${REDIS_URL}` logging. With a real password, every host log is a credential spill
Connect-before-handshake: with an unreachable server, initialize is never answered (~35–60s of retries, then exit)
Logs "Successfully connected" on a bare TCP accept, before AUTH — a success claim for a connection that never authenticated
A wrong password is retried identically 5 times over ~34s instead of surfacing the (honest, verbatim) error
npm marks the package deprecated / no longer supported; serverInfo reports 0.0.1
#057 Neon MCP server
Deferred — needs credentials · @neondatabase/mcp-server-neon v0.6.5 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Validates the key online at startup (a clean deferral), but the documented-in-code NEON_API_KEY env var is ignored — argv only — and the no-key error prints to stdout.
Task: list_projects (never reached — the server validates the key online at startup)
Graded against: Neon API unauthenticated 401, verbatim
NEON_API_KEY in env is silently ignored; only the README's argv form works
The no-key error is printed to stdout, polluting the JSON-RPC stream
npm install warns the package is deprecated in favour of the remote MCP at mcp.neon.tech
#058 Supabase MCP server
Deferred — needs credentials · @supabase/mcp-server-supabase v0.15.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Exemplary startup gate; decoy error is honest but content-free; tools/list is flaky — empty in 2 of 3 runs while the tools stay callable.
Task: list_projects with a decoy personal access token
Graded against: Partial: no-header 401 from the Supabase API
tools/list returned empty in 2 of 3 runs, yet list_projects remained callable while unlisted
Decoy error is a bare TypeError "fetch failed" — right shape (isError:true), no cause
--help crashes with ERR_PARSE_ARGS_UNKNOWN_OPTION
#059 GitLab MCP server
Failed tested task · @modelcontextprotocol/server-gitlab v2025.4.25 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Broken: the decoy call crashes in 0.10s — the server .map()s GitLab's object-shaped 401 body as if it were a results array, instead of surfacing the auth error.
Task: search_repositories with query "gitlab" and a decoy token
Graded against: Partial: GitLab's no-header 401 body ({"message":"401 Unauthorized"}) captured — exactly the object shape that triggers the crash
JSON-RPC -32603 "Cannot read properties of undefined (reading 'map')" replaces the real 401
serverInfo reports 0.5.1 vs package 2025.4.25
A retest on credential-permitting egress would confirm against a live 401; the crash mechanism is already demonstrated
#060 MongoDB MCP server
Worked with caveats · mongodb-mcp-server v3.2.2 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Best ground-truth match of the batch: the server's surfaced driver error is identical to the raw driver's, in 0.10s, with the password never echoed.
Task: list-databases via a decoy connection string
Graded against: Raw MongoDB driver against the same decoy SRV string — the surfaced error matches exactly (querySrv EREFUSED)
Scope: this verdict grades decoy-failure behaviour against driver ground truth — no successful read against a real database was graded
connectionId "preconfigured" is undiscoverable from the tool itself
The plain-string connect path waits the full 30s driver timeout, then swallows the real error behind a generic message
#061 Airtable MCP server (community)
Deferred — needs credentials · airtable-mcp-server v1.14.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Clean startup gate naming AIRTABLE_API_KEY; the decoy error has an honest shape but its cause is stripped to "fetch failed".
Task: list_bases with a decoy personal access token
Graded against: Partial: no-header 401 from the Airtable API
Startup failure comes with a noisy stack trace
Airtable offers no official stdio MCP server; this is the commonly installed community package
#062 Pinecone MCP server
Deferred — needs credentials · @pinecone-database/mcp v0.4.0 · tested 2026-10-31 by batch-3 wave testers; reviewed by Pax
Deferred — needs key. Best failure communication in the batch: its per-call error tells the user exactly how to fix it.
Task: list-indexes with a decoy key
Graded against: Partial: no-header 401 from the Pinecone API
Open question recorded: whether a real 401 lands in its "network problem, retry once" bucket is unobservable from this egress
Starts keyless with only a stderr warning rather than gating at startup — a defensible design, noted for contrast
Lantern Works