Lesson 1

How Redis picks its in-memory structure

HSET does not tell you what Redis just used to store that hash. For a small hash it is one contiguous run of bytes; past some threshold it becomes a hash table with a pointer for every field. Same command, same data type, several times the memory. OBJECT ENCODING is the command that says which one you got.

One key, two layers

Every Redis key has a type that you see (TYPE returns hash, set, zset, list, string) and an encoding that is how it is really stored (OBJECT ENCODING). You choose the type; Redis chooses the encoding, from the element count and the value lengths measured against the configuration thresholds.

112 combinations, measured

The rig builds each key with a single EVAL and then calls OBJECT ENCODING, across four collection types × seven element counts (1, 127, 128, 129, 511, 512, 513) × four kinds of value (3-byte string, 64-byte string, 65-byte string, integer). The thresholds are the Redis 7.4 defaults, except that list-max-listpack-size is set to 128 so there is a positive number to compare against.

docker exec rdlab redis-cli EVAL "
  for i=1,tonumber(ARGV[1]) do
    redis.call('HSET', KEYS[1], 'f'..i, ARGV[2])
  end
  return redis.call('HLEN', KEYS[1])" 1 h513 513 v
docker exec rdlab redis-cli OBJECT ENCODING h513
# hashtable

These are the rows out of the 112 that say the most:

The price of a 513th field

Two keys that differ by a single field:

One extra field makes the key take 4.65 times the memory, and the cost per field rises from 12.1 to 56.2 bytes. The reason is that a listpack stores the fields back to back in one byte block, with no pointers and no buckets, while a hashtable needs a bucket array plus a dictEntry for every field.

In practice: if your application holds millions of small hashes, keeping them under the threshold is the largest saving available without changing a line of code. The reverse is also true — raising hash-max-listpack-entries too far means every single-field read has to scan the whole listpack.

One small detail worth recording: h511 and h512 both take 6,192 bytes. That is not a sampling artefact of MEMORY USAGE — re-running with SAMPLES 0, which counts everything instead of sampling, still gives exactly 6,192. A listpack grows in steps, so sometimes one more field costs nothing.

Lists: the limit is in bytes, not elements

list-max-listpack-size defaults to -2, and a negative value means the limit is a size rather than a count. But even when you set it to a large positive number, a hard limit remains: one listpack node never exceeds 8,192 bytes.

Binary-searching for the boundary with list-max-listpack-size 100000:

All three boundaries stop at a MEMORY USAGE of 8,240 bytes: the 8,192-byte listpack plus roughly 48 bytes of object header. The lab below recomputes the listpack size as 6 bytes of header + 1 terminator + per element (contents + length prefix + backlen), and that formula reproduces all three boundaries exactly.

Conversion only goes one way

Redis converts from the compact encoding to the full one when a threshold is crossed, and never converts back. Measured:

hash with 513 fields                         → hashtable
CONFIG SET hash-max-listpack-entries 1024    → still hashtable
HSET one more field                          → still hashtable
HDEL down to 512 fields                      → still hashtable
a NEW key with 513 fields (threshold 1024)   → listpack

The same holds for sets: 129 strings become hashtable, and removing one to get back to 128 leaves it hashtable.

The lab

Set the type, the element count, the value length and each threshold. The algorithm running in your browser is a reimplementation of the rules above, checked against 112 matrix rows, three list byte boundaries and four string cases on Redis 7.4.11 — all of them agree.

Check yourself