That’s a lot of voxels

A typical Minecraft world, which is made of voxels, is 128 blocks high and 2048 blocks wide and deep, for a total volume of 536,870,912 blocks. We’ll call it 500 million. So now that we have LLM’s like Claude, we can get some advice on how to build bigger worlds, which I did. I asked Claude about using a filesystem based “database” to store worlds as big as 10,000×10,000×10,000 which is a trillion (10^12) blocks.

Claude agreed with me that traditional RDBMS won’t work for that. Claude also pointed out that it’s not typical to store a trillion files on a filesystem. However, for a 3D world with a trillion blocks, we could store 30 million files, each of which was 32x32x32. The math is:

10^12/32^3 = 30,517,578.125. About 30 million files.

As long as they’re in directories, it could work. FreeBSD UFS can store 32,767 files per directory or we could just use ZFS which can store 2^48 (281 trillion) files per directory. Linux ext4 can store roughly 10 million files per directory.

I asked Claude for a POC which it created in rust in about 8 minutes.

So, after a couple hours additional work and learning I now have kBlockDB.

• Capable of storing 1 trillion voxels, perhaps more.
• Each voxel can have an arbitrary number of key-value pairs. The keys are strings, and the values are string, float or int64.
• Arbitrary dimensions. Typically we think of a 3-dimensional world, however if we wanted to think about layers we could imagine 4 dimensions. KBlockDB supports any number of dimensions.
• There is a write-through cache to speed performance
• kBlockDB has an RESTful interface, and a custom binary interface which is about 50% faster

On my Macbook, the performance numbers are “ok”. 2500-3000 write operations per second and about 10,000 read operations per second over HTTP.

Next steps could include

• A query language in including concepts such as arbitrary N-dimensional regions, voxel key names and voxel values and timestamps
• I’d like to explore the idea of custom indices to enable fast querying
• Synchronization of kBlockDB databases across hosts, with appropriate version management.
• Potential implementation of compression to save space
• There is support for username and password login, but I’ll likely add LDAP support.