spec: incorporate MeshCore discussion #2538 review feedback

Incorporates two of the three corrections raised by msmuenchen on discussion #2538: - Identity: document that the 64-byte Ed25519 private key is the expanded `a || RH` (orlp) form, not a 32-byte seed, and that it cannot be consumed by standard Ed25519 libraries. Touches 15-identity.md and 14-crypto.md. - Group: document the GRP_TXT/GRP_DATA inner plaintext format (timestamp + flags + message) and the GRP_TXT length limits in 08-payload-group.md. The third correction (GRP_DATA vs RAW_CUSTOM app differentiation) is deferred pending reviewer clarification. Corpus: replace the unrealistic group vectors with valid inner-format plaintexts; the ed25519 vectors now carry the 64-byte expanded key and the seed; the schema and validator gain seed fields. Also corrects two pre-existing wrong values found while verifying -- ed-001's public key and ed-002's signature now match RFC 8032. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 19:11:19 -06:00
1 parent 9232c16f7c
commit 9d7b1c6281
@@ -3,7 +3,7 @@
  "name": "Ed25519 Sign/Verify",
  "spec_ref": "src/15-identity.md",
  "protocol_version": 0,
-  "description": "Ed25519 signature test vectors. MeshCore uses Ed25519 for advertisement signing. These vectors use RFC 8032 test keypairs for cross-validation.",
+  "description": "Ed25519 signature test vectors. MeshCore uses Ed25519 for advertisement signing. These vectors use RFC 8032 Section 7.1 test keypairs for cross-validation. crypto_context carries both the 32-byte seed (sender_seed) and the 64-byte expanded private key (sender_private_key, the clamped scalar a || nonce prefix RH) that MeshCore actually stores; see src/15-identity.md, 'Private Key Format'.",
  "vectors": [
    {
      "id": "ed-001",
@@ -26,11 +26,12 @@
      },
      "binary": "3D 00 E5564300C360AC729086E2CC806E828A84877F1EB8E5D974D873E065224901555FB8821590A33BACC61E39701CF9B46BD25BF5F0595BBE24655141438E7A100B",
      "crypto_context": {
-        "sender_private_key": "9D61B19DEFFD5A60BA844AF492EC2CC44449C5697B326919703BAC031CAE7F60",
-        "sender_public_key": "D75A980182B10AB7D54BFED3C964073A0EE172F3DAA3F4A18446B0B8D183F8E3",
+        "sender_seed": "9D61B19DEFFD5A60BA844AF492EC2CC44449C5697B326919703BAC031CAE7F60",
+        "sender_private_key": "307C83864F2833CB427A2EF1C00A013CFDFF2768D980C0A3A520F006904DE94F9B4F0AFE280B746A778684E75442502057B7473A03F08F96F5A38E9287E01F8F",
+        "sender_public_key": "D75A980182B10AB7D54BFED3C964073A0EE172F3DAA62325AF021A68F707511A",
        "plaintext": ""
      },
-      "notes": "RFC 8032 Section 7.1 Test Vector 1. Private key (seed) is 32 bytes; the 64-byte format used by MeshCore includes the expanded key. Signature over empty message."
+      "notes": "RFC 8032 Section 7.1 Test Vector 1. sender_seed is the 32-byte RFC seed; sender_private_key is the 64-byte expanded form (a || RH) that MeshCore stores. Signature over an empty message."
    },
    {
      "id": "ed-002",
@@ -48,16 +49,17 @@
          "hashes": []
        },
        "payload": {
-          "data": "92A009A9F0D4CAB8720E820B5F642540A2B27B5416503F8FB3762223EBDB69DA085AC1E43E159C7E94B2BA35F6F6A1918E25682F9B0ADC5C33C2BF3C39F04E50"
+          "data": "92A009A9F0D4CAB8720E820B5F642540A2B27B5416503F8FB3762223EBDB69DA085AC1E43E15996E458F3613D0F11D8C387B2EAEB4302AEEB00D291612BB0C00"
        }
      },
-      "binary": "3D 00 92A009A9F0D4CAB8720E820B5F642540A2B27B5416503F8FB3762223EBDB69DA085AC1E43E159C7E94B2BA35F6F6A1918E25682F9B0ADC5C33C2BF3C39F04E50",
+      "binary": "3D 00 92A009A9F0D4CAB8720E820B5F642540A2B27B5416503F8FB3762223EBDB69DA085AC1E43E15996E458F3613D0F11D8C387B2EAEB4302AEEB00D291612BB0C00",
      "crypto_context": {
-        "sender_private_key": "4CCD089B28FF96DA9DB6C346EC114E0F5B8A319F35ABA624DA8CF6ED4FB8A6FB",
+        "sender_seed": "4CCD089B28FF96DA9DB6C346EC114E0F5B8A319F35ABA624DA8CF6ED4FB8A6FB",
+        "sender_private_key": "68BD9ED75882D52815A97585CAF4790A7F6C6B3B7F821C5E259A24B02E502E514566848291DACAF225CC63DEB348DA318E2C2E17B00B8160F9CE6BFA0472911D",
        "sender_public_key": "3D4017C3E843895A92B70AA74D1B7EBC9C982CCF2EC4968CC0CD55F12AF4660C",
        "plaintext": "72"
      },
-      "notes": "RFC 8032 Section 7.1 Test Vector 2. Signature over the single byte 0x72."
+      "notes": "RFC 8032 Section 7.1 Test Vector 2. sender_seed is the 32-byte RFC seed; sender_private_key is the 64-byte expanded form (a || RH) that MeshCore stores. Signature over the single byte 0x72."
    },
    {
      "id": "ed-003",
@@ -80,11 +82,12 @@
      },
      "binary": "3D 00 6291D657DEEC24024827E69C3ABE01A30CE548A284743A445E3680D7DB5AC3AC18FF9B538D16F290AE67F760984DC6594A7C15E9716ED28DC027BECEEA1EC40A",
      "crypto_context": {
-        "sender_private_key": "C5AA8DF43F9F837BEDB7442F31DCB7B166D38535076F094B85CE3A2E0B4458F7",
+        "sender_seed": "C5AA8DF43F9F837BEDB7442F31DCB7B166D38535076F094B85CE3A2E0B4458F7",
+        "sender_private_key": "909A8B755ED902849023A55B15C23D11BA4D7F4EC5C2F51B1325A181991EA95C6608C8666B9CDE2325F539D7D83386FE8187C6BE61D8A70C247190D64EDF5F1E",
        "sender_public_key": "FC51CD8E6218A1A38DA47ED00230F0580816ED13BA3303AC5DEB911548908025",
        "plaintext": "AF82"
      },
-      "notes": "RFC 8032 Section 7.1 Test Vector 3. Signature over the two-byte message 0xAF82."
+      "notes": "RFC 8032 Section 7.1 Test Vector 3. sender_seed is the 32-byte RFC seed; sender_private_key is the 64-byte expanded form (a || RH) that MeshCore stores. Signature over the two-byte message 0xAF82."
    }
  ]
 }
@@ -3,7 +3,7 @@
  "name": "Group Data Payload",
  "spec_ref": "src/08-payload-group.md",
  "protocol_version": 0,
-  "description": "Tests group datagram payload. Same wire format as GRP_TXT but payload type = 0x06.",
+  "description": "Tests group datagram payload. Same wire format as GRP_TXT but payload type = 0x06. The encrypted plaintext is timestamp(4, LE) followed by an application-defined binary blob.",
  "vectors": [
    {
      "id": "grp-data-001",
@@ -22,17 +22,17 @@
        },
        "payload": {
          "channel_hash": "72",
-          "cipher_mac": "D184",
-          "ciphertext": "948C819389BA4ED7B1194A0FF8E62073"
+          "cipher_mac": "EFFF",
+          "ciphertext": "70917BC984A0AD4A409C3906B2266144"
        }
      },
-      "binary": "19 00 72 D184 948C819389BA4ED7B1194A0FF8E62073",
+      "binary": "19 00 72 EFFF 70917BC984A0AD4A409C3906B2266144",
      "crypto_context": {
        "shared_secret": "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F",
        "encryption_key": "202122232425262728292A2B2C2D2E2F",
-        "plaintext": "47726F75704D736721"
+        "plaintext": "00F15365DEADBEEF0102030405"
      },
-      "notes": "Header 0x19 = GRP_DATA(6)+FLOOD(1). Same ciphertext as GRP_TXT since same key and plaintext."
+      "notes": "Header 0x19 = GRP_DATA(6)+FLOOD(1). Channel hash = SHA256(channel_secret)[0] = 0x72. Inner plaintext is the GRP_DATA format: timestamp 1700000000 (00F15365 LE) + the application-defined blob DEADBEEF0102030405. 13 bytes, AES-128-ECB zero-padded to 16. AES key = first 16 bytes of channel secret; MAC = HMAC-SHA256(channel_secret, ciphertext)[0..1]."
    }
  ]
 }
@@ -3,7 +3,7 @@
  "name": "Group Text Payload",
  "spec_ref": "src/08-payload-group.md",
  "protocol_version": 0,
-  "description": "Tests group text message payload: channel_hash(1) + cipher_mac(2) + ciphertext.",
+  "description": "Tests group text message payload: channel_hash(1) + cipher_mac(2) + ciphertext. The encrypted plaintext is the GRP_TXT inner format: timestamp(4, LE) + flags(1) + \"<sender_name>: <body>\".",
  "vectors": [
    {
      "id": "grp-txt-001",
@@ -22,23 +22,23 @@
        },
        "payload": {
          "channel_hash": "72",
-          "cipher_mac": "D184",
-          "ciphertext": "948C819389BA4ED7B1194A0FF8E62073"
+          "cipher_mac": "EC1C",
+          "ciphertext": "AE220A559CFF467B8F45948C2CABFD745BB36B14426A2127BB391DBBA51DE9E5"
        }
      },
-      "binary": "15 00 72 D184 948C819389BA4ED7B1194A0FF8E62073",
+      "binary": "15 00 72 EC1C AE220A559CFF467B8F45948C2CABFD745BB36B14426A2127BB391DBBA51DE9E5",
      "crypto_context": {
        "shared_secret": "202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F",
        "encryption_key": "202122232425262728292A2B2C2D2E2F",
-        "plaintext": "47726F75704D736721"
+        "plaintext": "00F1536500616C6963653A2068656C6C6F206D657368"
      },
-      "notes": "Channel hash = SHA256(channel_secret)[0] = 0x72. Plaintext 'GroupMsg!' (9 bytes). AES key = first 16 bytes of channel secret."
+      "notes": "Channel hash = SHA256(channel_secret)[0] = 0x72. Inner plaintext is the GRP_TXT format: timestamp 1700000000 (00F15365 LE) + flags 0x00 (TXT_TYPE_PLAIN) + the message \"alice: hello mesh\" (sender name \"alice\"). 22 bytes, AES-128-ECB zero-padded to 32. AES key = first 16 bytes of channel secret; MAC = HMAC-SHA256(channel_secret, ciphertext)[0..1]."
    },
    {
      "id": "grp-txt-002",
      "description": "Group text with tampered MAC must fail",
      "type": "invalid",
-      "binary": "15 00 72 0000 948C819389BA4ED7B1194A0FF8E62073",
+      "binary": "15 00 72 0000 AE220A559CFF467B8F45948C2CABFD745BB36B14426A2127BB391DBBA51DE9E5",
      "expected_error": "mac_invalid",
      "notes": "Same ciphertext as grp-txt-001 but MAC zeroed. Verification must fail."
    }
@@ -329,17 +329,25 @@
      "type": "object",
      "additionalProperties": false,
      "properties": {
+        "sender_seed": {
+          "$ref": "#/$defs/hex_string",
+          "description": "32-byte Ed25519 seed of sender; SHA-512-expanded and clamped to form the 64-byte private key"
+        },
        "sender_private_key": {
          "$ref": "#/$defs/hex_string",
-          "description": "64-byte Ed25519 private key of sender"
+          "description": "64-byte expanded Ed25519 private key of sender (clamped scalar a || nonce prefix RH)"
        },
        "sender_public_key": {
          "$ref": "#/$defs/hex_string",
          "description": "32-byte Ed25519 public key of sender"
        },
+        "recipient_seed": {
+          "$ref": "#/$defs/hex_string",
+          "description": "32-byte Ed25519 seed of recipient; SHA-512-expanded and clamped to form the 64-byte private key"
+        },
        "recipient_private_key": {
          "$ref": "#/$defs/hex_string",
-          "description": "64-byte Ed25519 private key of recipient"
+          "description": "64-byte expanded Ed25519 private key of recipient (clamped scalar a || nonce prefix RH)"
        },
        "recipient_public_key": {
          "$ref": "#/$defs/hex_string",
@@ -69,6 +69,59 @@ secret. Only a successful MAC verification confirms the correct channel.
   channel's shared key.
 4. The first channel that produces a valid MAC yields the correct decryption.

+### Inner Plaintext Format
+
+The decrypted ciphertext begins with a 4-byte little-endian `timestamp`. The
+remaining bytes depend on the payload type.
+
+**GRP_TXT** uses the same `timestamp ‖ flags` prefix as TXT_MSG (see
+[Section 6 — Plaintext Format (TXT_MSG)](06-payload-encrypted.md#plaintext-format-txt_msg)):
+
+| Field | Offset | Size | Type | Description |
+|-------|--------|------|------|-------------|
+| timestamp | 0 | 4 bytes | uint32_le | Message timestamp |
+| flags | 4 | 1 byte | uint8 | Bits 2–7: message type (`0` = plain text); bits 0–1: attempt counter |
+| message | 5 | remaining | UTF-8 | `"<sender_name>: <body>"` |
+
+For GRP_TXT the reference chat firmware always sets `flags` to `0x00`
+(`TXT_TYPE_PLAIN`, attempt 0). The `message` joins the sender's display name
+and the body as `"<sender_name>: <body>"` (for example, `alice: on my way`), so
+every channel member can attribute the message without a separate sender field.
+The receiver requires the plaintext to be longer than 5 bytes and reads
+`message` as a NUL-terminated string: trailing zero bytes from AES-ECB padding
+(see [Section 14](14-crypto.md)) act as the terminator, and a receiver MUST
+treat the first `0x00` byte at or after offset 5 as the end of the message.
+
+**GRP_DATA** carries `timestamp(4)` followed by an application-defined binary
+blob. MeshCore prescribes no structure for the blob.
+
+### Length Limits
+
+The ciphertext is AES-128-ECB output, so it is always a positive multiple of
+`CIPHER_BLOCK_SIZE` (16 bytes). With a 1-byte `channel_hash` and 2-byte
+`cipher_mac`, the encoder (`Mesh::createGroupDatagram`) rejects any plaintext
+longer than 168 bytes (`data_len + 1 + CIPHER_BLOCK_SIZE − 1 ≤
+MAX_PACKET_PAYLOAD`).
+
+For GRP_TXT the reference chat firmware applies a tighter limit: `MAX_TEXT_LEN`
+= `10 × CIPHER_BLOCK_SIZE` = **160 bytes** (`BaseChatMesh.h`). This bound is
+applied to the combined `"<sender_name>: " + body` text; the 4-byte `timestamp`
+and 1-byte `flags` are additional.
+
+Because the sender's display name is embedded in the plaintext, the body length
+available to an application is **not fixed** — it shrinks as the sender name
+grows. For sender names of 0–32 characters the usable body runs roughly from
+153 bytes (short name) down to 121 bytes (32-character name). Implementations
+MUST handle this variability and MUST NOT assume a single fixed maximum body
+length.
+
+> **Note (informational).** Upstream issue
+> [#2583](https://github.com/meshcore-dev/MeshCore/issues/2583) proposes raising
+> the GRP_TXT limit to 11 blocks (~172 bytes), since a 1-byte `channel_hash`,
+> 2-byte `cipher_mac`, and an 11-block ciphertext still fit within
+> `MAX_PACKET_PAYLOAD`. That change is not present in the reference firmware
+> this specification tracks.
+
 ### Cross-References

 - [Section 14: Cryptography](14-crypto.md) — Encrypt-then-MAC
@@ -15,10 +15,13 @@ in an encrypt-then-MAC scheme.
 |-----------|-----------|----------|--------|
 | Symmetric Encryption | AES-128-ECB | 16 bytes | Multiple of 16 bytes |
 | Message Authentication | HMAC-SHA256 | 32 bytes | 2 bytes (truncated) |
-| Digital Signature | Ed25519 | 32/64 bytes | 64 bytes |
+| Digital Signature | Ed25519 | 32 B public / 64 B private | 64 bytes |
 | Key Exchange | X25519 (ECDH) | 32 bytes | 32 bytes |
 | Hashing | SHA-256 | — | Up to 32 bytes |

+Ed25519 private keys are stored in expanded `a ‖ RH` form rather than as a
+32-byte seed; see [Section 15: Private Key Format](15-identity.md#private-key-format).
+
 ### AES-128-ECB Encryption

 MeshCore uses AES-128 in Electronic Codebook (ECB) mode with zero-byte padding.
@@ -14,10 +14,47 @@ X25519 (Ed25519 keys transposed to Curve25519).
 | Type | Size | Description |
 |------|------|-------------|
 | Public Key | 32 bytes (PUB_KEY_SIZE) | Ed25519 public key |
-| Private Key | 64 bytes (PRV_KEY_SIZE) | Ed25519 private key (seed + public) |
+| Private Key | 64 bytes (PRV_KEY_SIZE) | Expanded Ed25519 private key: clamped scalar `a` (32) ‖ nonce prefix `RH` (32) |
 | Signature | 64 bytes (SIGNATURE_SIZE) | Ed25519 signature |
 | Shared Secret | 32 bytes (PUB_KEY_SIZE) | ECDH shared secret |

+### Private Key Format
+
+MeshCore stores the Ed25519 private key in **expanded** form — the 64-byte
+value `a ‖ RH`, not the 32-byte seed that most Ed25519 APIs accept.
+
+A key pair is generated from a 32-byte random seed:
+
+1. Compute `SHA-512(seed)`, producing 64 bytes.
+2. Clamp the first 32 bytes: `b[0] &= 248; b[31] &= 63; b[31] |= 64`. The
+   result is the scalar `a`.
+3. The second 32 bytes are the nonce prefix `RH`.
+4. The stored private key is `a ‖ RH` (`PRV_KEY_SIZE` = 64 bytes).
+5. The public key is `a · B` — scalar multiplication of `a` by the Ed25519
+   base point — yielding 32 bytes (`PUB_KEY_SIZE`).
+
+This is the layout used by the [orlp/ed25519](https://github.com/orlp/ed25519)
+C library bundled with the reference implementation. Pre-expanding the key lets
+the firmware sign without recomputing `SHA-512(seed)` at load time.
+
+> **Interoperability warning.** The 64-byte `a ‖ RH` value is **not** a
+> standard Ed25519 private key. Conformant Ed25519 libraries store the 32-byte
+> seed and perform the SHA-512 expansion and clamping themselves at signing
+> time; they cannot consume `a ‖ RH` directly. Because SHA-512 is one-way, the
+> original seed **cannot** be recovered from a stored private key. An
+> implementation that holds only `a ‖ RH` therefore MUST use a signer that
+> accepts a pre-expanded key, or replicate the orlp signing procedure.
+>
+> Public keys are unaffected: a MeshCore public key is an ordinary 32-byte
+> Ed25519 public key, and signatures verify with any conformant Ed25519
+> verifier.
+
+See also: [ed25519 key formats](https://blog.mozilla.org/warner/2011/11/29/ed25519-keys/)
+(Mozilla); a portable Java/Bouncycastle implementation
+([meshcore-packets-java](https://github.com/msmuenchen/meshcore-packets-java));
+and the Bouncycastle feature request for orlp-style keys
+([bc-java#2302](https://github.com/bcgit/bc-java/issues/2302)).
+
 ### Hash Derivation

 A node's routing hash is simply a prefix of its public key:
@@ -47,7 +84,10 @@ match = memcmp(hash, pub_key, hash_size) == 0
 Used for advertisement signatures (see [Section 5](05-payload-advert.md)):

 1. Construct the message to sign (e.g., pub_key || timestamp || app_data).
-2. Sign using the Ed25519 private key.
+2. Sign with the expanded 64-byte private key: the scalar `a` (first 32 bytes)
+   is used directly as the signing scalar and the nonce prefix `RH` (second 32
+   bytes) as the deterministic-nonce input. No SHA-512 expansion or clamping is
+   performed at signing time (see [Private Key Format](#private-key-format)).
 3. The 64-byte signature is included in the advertisement payload.

 Verification:
@@ -61,7 +101,11 @@ Verification:

 To compute a shared secret between two nodes:

-1. Convert the Ed25519 keys to X25519 (Curve25519) format.
+1. Convert the Ed25519 keys to X25519 (Curve25519) format. For the private
+   side this is trivial: the X25519 scalar is exactly `a`, the first 32 bytes
+   of the expanded private key, which is already clamped. The peer's public
+   key is converted from its Ed25519 (Edwards) point to the corresponding
+   Montgomery (Curve25519) `u`-coordinate.
 2. Perform X25519 Diffie-Hellman: `shared_secret = X25519(my_private, their_public)`.
 3. The resulting 32-byte shared secret is used for:
   - AES-128 key: `shared_secret[0..15]`
@@ -72,7 +116,7 @@ To compute a shared secret between two nodes:
 Identities can be serialized for storage:

 - **Public Identity**: 32 bytes (public key only)
- **Local Identity**: 64 bytes (private key) + 32 bytes (public key) = 96 bytes
+- **Local Identity**: 64 bytes (expanded private key) + 32 bytes (public key) = 96 bytes

 Hex representation uses uppercase characters with 2 hex digits per byte.

@@ -216,8 +216,9 @@ def validate_vector_consistency(vector, file_path):
    if "crypto_context" in vector:
        ctx = vector["crypto_context"]
        for field in ("shared_secret", "encryption_key", "plaintext",
-                      "sender_private_key", "sender_public_key",
-                      "recipient_private_key", "recipient_public_key"):
+                      "sender_seed", "sender_private_key", "sender_public_key",
+                      "recipient_seed", "recipient_private_key",
+                      "recipient_public_key"):
            if field in ctx:
                errors.extend(validate_hex_string(
                    ctx[field], f"crypto_context.{field}", file_path, vid))