Different frequencies, one position vector

Position 3

SineCosine

Features 0 and 1: sin(position), cos(position)

Features 2 and 3: sin(position / 10), cos(position / 10)

Features 4 and 5: sin(position / 100), cos(position / 100)

Features 6 and 7: sin(position / 1000), cos(position / 1000)

Position vector 3

Feature 0

Feature 1

Feature 2

Feature 3

Feature 4

Feature 5

Feature 6

Feature 7

Each pair changes at its own frequency. The slowest waves barely change over this short range, and the displayed values are rounded. All eight values form the vector added to a token's embedding at this position.