Repository: three.js
Issue: Allow >4 bones to skin a vertex
Labels: Enhancement

# Description

Currently, ThreeJS limits how many bones can _skin_ a vertex to at most 4. When a loader sees more than 4 it arbitrarily removes some of them and [emits a warning](https://discourse.threejs.org/t/is-the-limit-of-4-skinning-weights-per-vertex-a-hard-limit-of-webgl/801). This results in broken rigs.

Some file formats and mesh/rig authoring tools allow vertices to be skinned by more than 4 bones. I have some meshes with this need so I would like to add support to ThreeJS for more than 4 bone weights per vertex.

**I have started modifying ThreeJS to add this support and was wondering if it would likely be accepted?**

# How bone skinning currently works

Vertex shaders receive [per-vertex skinning weights + indexes](https://github.com/mrdoob/three.js/blob/cfc782958f39f096489ca12ce77aa4bea48b34a1/src/renderers/webgl/WebGLProgram.js#LL654C1-L660C1):

```glsl
attribute vec4 skinIndex;
attribute vec4 skinWeight;
``` 

`skinIndex` is used to [retrieve the bone matrices](https://github.com/mrdoob/three.js/blob/cfc782958f39f096489ca12ce77aa4bea48b34a1/src/renderers/shaders/ShaderChunk/skinbase_vertex.glsl.js#L4) and then `skinWeight` weights them [to get the final vertex](https://github.com/mrdoob/three.js/blob/cfc782958f39f096489ca12ce77aa4bea48b34a1/src/renderers/shaders/ShaderChunk/skinning_vertex.glsl.js).

The `skinIndex` and `skinWeight` vertex buffers are [directly created by loaders](https://github.com/mrdoob/three.js/blob/cfc782958f39f096489ca12ce77aa4bea48b34a1/examples/jsm/loaders/GLTFLoader.js#LL2110C24-L2110C24) at load time and [attached to `BufferGeometry`](https://github.com/mrdoob/three.js/blob/cfc782958f39f096489ca12ce77aa4bea48b34a1/examples/jsm/loaders/GLTFLoader.js#L4467).

# How I propose supporting >4 weights per vertex

Instead of sending bone indexes and weights to the vertex shader as vertex buffer data, it would be uploaded as a data texture and a single int per-vertex for the starting point in the bone-index-weight-pair texture.

```glsl
attribute int bonePairTexStartIndex;
uniform sampler2D boneIndexWeightPairs;
```

The shading code would read from the bone pair texture until it saw an index of -1 or the maximum number of vertices, `MAX_BONES_PER_VERT`, is reached:

```glsl
        #define MAX_BONES_PER_VERT 32;

        ...
        vec4 skinVertex = bindMatrix * vec4( transformed, 1.0 );
	vec4 skinned = vec4( 0.0 );
	vec4 skinnedNormal = vec4( 0.0 );
        for (int i = 0; i < MAX_BONES_PER_VERT; i = i + 1) {
		int bonePairTexIndex = bonePairTexStartIndex + i;
		vec2 boneIndexWeight = 
			texelFetch(boneIndexWeightPairs, 
			           ivec2(bonePairTexIndex % boneTexWidth,
			           bonePairTexIndex / boneTexWidth),
			           0).xy;
		int boneIndex = int(boneIndexWeight.x);
		if (boneIndex < 0) break;
		mat4 boneMatrix = getBoneMatrix(boneIndex);
		skinned += (boneMatrix * skinVertex).xyz * boneIndexWeight.y;
		skinnedNormal += 
			normalize(mat3(bindMatrixInverse)
			        * mat3(boneMatrix)
			        * mat3(bindMatrix) * objectNormal)
			* boneIndexWeight.y;
	}
```

Instead of loading skinning index/weight data directly to a vertex buffer on `BufferGeometry`, loaders should create a data texture if a flag tells it to. It could also detect if any vertex has more than 4 bone weights and make this decision dynamically but it's easier to keep it as a configuration flag. A new texture on `SkinnedMesh` seems like the right place for this to live.

This change in the shaders would be controlled by a new `#define` so that the existing behavior is maintained when the extra bones aren't needed.

# Alternatives

More shader attributes can be added to support additional bone weights but there's a limit on the amount of vertex data. [`MAX_VERTEX_ATTRIBS` for WebGL is 16](https://registry.khronos.org/webgl/sdk/tests/extra/webgl-info.html). The more attributes are used, the less likely a browser is to support the shader. Putting this data in a texture doesn't run into a similar limit although it might make performance worse.

# Additional context

Unity supports unlimited blend weights as a parameter: https://docs.unity3d.com/2019.2/Documentation/Manual/class-QualitySettings.html#BlendWeights

Someone previously had an example of FBX deleting weights when there were more than 4 per vertex: https://discourse.threejs.org/t/is-the-limit-of-4-skinning-weights-per-vertex-a-hard-limit-of-webgl/801

Their related Github issue: https://github.com/mrdoob/three.js/issues/12127