Prologue #

Made possible by generous support from givemeanode

Some time ago, a post landed on my X feed showing off frontier LLMs being able to parse text from images for fewer tokens than the text itself costs. Projects such as Snapcompact and pxpipe demonstrate this ability by creating dense images with text, and passing them to models in order to cut down costs and subscription usage. Models such as Claude Fable 5 from Anthropic or Gemini 3.6 Flash from Google are remarkably good at using text from images, boasting 4.8x and 5x context efficiency increases respectively. Seeing them read text out of pixels so cheaply made me curious about the inner workings of the vision component of LLMs, and how they could bring such benefits. By the end of this article, you’ll see how I reimplemented the foundational ideas of VLMs in order to fine-tune a small model that packs code into 4x fewer context slots and can return it byte-identical.

This article is a description of the implemenation of my project Exact Latents on GitHub.

Images in an LLM #

When text is passed to an LLM, a process must occur before it hits the model itself. The text is tokenized: divided into subword chunks present within the LLM’s vocabulary, and these tokens are converted to their respective embeddings which are then passed into the model. Images instead are chopped into pixel patches (typically 14x14 or 16x16), and passed to a dedicated vision encoder neural network, which outputs embeddings as well. The key difference is where these embeddings come from. A text token’s embedding is indexed from a massive table by its id, and the same token always produces the same vector. The vision encoder instead computes fresh embeddings, vectors that exist in no table anywhere. The key similarity is that both land in the same context window with the model reading them identically.

This is also why the image path can be cheaper. A text token costs one context slot no matter how little it says, meaning an indentation gets the same slot as a rare identifier, as its embedding must be one of the values the table was born with. The encoder plays under no such rules. It sees the whole patch, decides what matters, and can pack the information of many characters into each vector it writes. Fewer slots, denser vectors, same context.

So why not code? #

Whether it’s your coding agent or online chatbot, an LLM reads code the same way: the code is tokenized through the process outlined above. Every token is still given that same slot in the context window, no matter how rare or common it is. Code is especially expensive in this regard, as it’s mostly composed of structure the model already knows. Indentation, function definition, colons, boilerplate, all of these are second nature to the model, yet each one pays the same context price the real meat of the code pays.

This is why feeding images of code to VLMs has seen the compression gains it has, as the necessity to embed every token is thrown out and instead the vision encoder decides what the truly important information to pass forward is. Vision-encoder compression is fascinating as-is, but there are two glaring issues.

First, the requirement to turn the text into an image before being decoded into the final embeddings is a painful step. Rasterizing text just so a neural network can untangle it. Second, most VLMs’ ability to read text is benchmarked on similarity rather than verbatim accuracy. The 98% accuracy DeepSeek-OCR boasts sounds great until you realize a program with 98% of the correct characters is a broken one. So what if you instead made a code encoder? Text goes directly into the encoder, and out come embeddings the LLM understands. No pixels, no benchmark forgiveness: the code has to come back exact.

To a tee #

Beginning with the architecture, I decided to use two Qwen3 1.7B models with one acting as an encoder and one as the decoder connected by a simple two-layer MLP projector. The encoding model has the last two layers cut off so the resulting embeddings are its hidden thinking state about the context as a whole, instead of for predicting the next token. These hidden states are then fed into the projector, which has the simple task of translating the vectors from the encoder context to the decoder context. The decoder model is the actual LLM the user talks to, holding the compressed vectors alongside ordinary tokenized text (such as queries or the system prompt) just like a standard VLM with the image and text in the same context.

However, I left out one key detail, which also happens to be the more novel component of the architecture, the pooler.

On paper, its job sounds simple. It sits between the encoder and the projector, with the encoder handing it one hidden state vector per input token, and it has to return a quarter as many without losing the actual meaning of the code. The easiest solution is to simply split the tokens into groups of four and average each group into one vector. This works to a degree, but it treats every token as equally valuable, and it has no learned component at all, so it can never get better at the one decision that matters. Indentation and standard syntax are cheap to predict, so allocating the same “room” for them as for a specific named function call misses the point of the project.

Instead, the solution I created was a 17 million parameter learned module, built specifically for pooling. The module treats each group’s average as a starting point, using it as a query over the entire context and blending in the token vectors relevant to it. This means a slot near the function’s signature can stay mostly “at home”, holding the information of the signature, while another can reach across the function for the one rare identifier it’s responsible for remembering. Alongside this, a distance penalty makes slots favor their own neighborhood unless there’s a good reason to look further, and a built-in table of token rarity tells the module from day one which tokens are common boilerplate and which ones are expensive to get wrong. Finally, the whole scanning mechanism starts training with its influence dialed all the way down. On the first step the pooler is the plain average, and the learned allocation only grows in as training proves it out, so the module can never do worse than the baseline it replaced.

How to train to a tee #

The training loop is simple. Get a sample of Python code, split it into individual functions (via AST), pass them through the compressor, and then through the decoder LLM with nothing in its context but the vectors. Every token the decoder emits is graded against the original and the errors are pushed back through the weights. The key point to pay attention to here is the bar you grade against. Similarity makes it easy to achieve high-looking scores, but it is deceptive about true exactness. A model that gets 98% of tokens right sounds nearly done, yet at that rate a 300 token function comes back fully correct about 0.2% of the time. Exactness has to be graded all or nothing, and everything downstream of this choice, including which checkpoints even counted as progress, flows from refusing to grade on similarity.

When it came to deciding what components of the architecture I wanted to train, I tried to start with as minimal a setup as possible. I kept the encoder and decoder completely frozen, with the only learned component being the projector. From this, I was able to get 5/20 functions code-exact (I’ll use this term more, but code-exact means identical apart from comments and docstrings). From here, I realized that attempting to keep the encoder and decoder frozen would simply not work, so I turned to LoRA for both. LoRA trains small add-on matrices while the original weights stay untouched, making it a significantly faster and cheaper way to fine-tune an LLM. Through this, grading the decoder token by token puts accuracy at a pretty impressive high-nineties percentage. Writing freely from the vectors alone though, only 61% of functions came back code-exact. However, I got stuck at this exact 61% figure for multiple runs as I tried implementing different features to improve performance to no avail. For scale, the frozen setup trained 8.4 million parameters, a quarter of a percent of the system, and LoRA only raised that to around 40 million. The full fine-tune would raise it to 3.37 billion. Ultimately, turning back to the very VLMs which inspired this project in the first place, I realized the one thing I had failed to replicate was their use of full fine-tuning. Add-on matrices can only bend a model so far, and full fine-tuning doesn’t have that limit. I performed the full fine-tune, and it yielded the exact results I was looking for.

A few training decisions matter enough to mention here, as their fingerprints show up all over the results. The loss was told to care three times less about the wording of comments and docstrings (identified by an AST parser) than about code, since paraphrasing a comment is forgivable while paraphrasing an identifier is not. Training also alternated between 4x and 8x compression instead of committing to one rate. A slice of the batches also asked the model to answer a question about the function from the vectors alone instead of reconstructing it, alongside a handful of smaller auxiliary tasks. A tenth of the training batches were plain uncompressed code with no vectors at all, purely so the decoder wouldn’t forget how to be an ordinary language model while learning its new job.

All of this ran over 2.6 million Python functions for 16,000 steps on an 8xH100 node rented from givemeanode at roughly $32 an hour. The entire project, every failed run included, came in under $1,900, nearly all of it covered by credits givemeanode granted me.

Results #

On the final full fine-tune, with 600 out-of-distribution functions from repos created after the training data’s cutoff, the model achieved 96% code-exact (577/600) and 88% byte-identical (527/600) at 4x compression. For comparison, handing the same model the raw uncompressed text and telling it to copy scores 97% code-exact, so reading vectors gives up almost nothing over reading the text itself while using a quarter of the context. To be clear about what 4x means, it’s four times fewer context slots rather than four times fewer bits, as each vector is a full-width embedding taking up one slot like any token would.

Below is a diff of an example input (the unit is a single Python function) compared with the output.

@_partial(jax.jit, static_argnums=(2, 3))
def _ses_ensemble_via_utils(resids: jnp.ndarray, alphas: jnp.ndarray, smooth: bool, order: int) -> jnp.ndarray:
    """SES ensemble or rolling mean based on smooth flag.
    
    Args:
        resids: Residuals array
        alphas: Pre-computed array of alpha values for SES ensemble
        smooth: If True, use SES ensemble; if False, use rolling mean (static)
        order: Window size for rolling mean minus 1 (static)
    """
    def smooth_path():
        def one(alpha):
            _, fitted = utils._ses_forecast(resids, alpha)
            f = jnp.nan_to_num(fitted, nan=0.0)
            mask = jnp.isnan(fitted)
            idx = jnp.maximum.accumulate((~mask).astype(jnp.int32) * jnp.arange(resids.size))
            return f[idx]
        mats = jax.vmap(one)(alphas)
        return jnp.mean(mats, axis=0)
    
    def rolling_path():
        rm = _rolling_mean(resids, order + 1)
        return rm.at[:order + 1].set(resids[:order + 1])
    

    return lax.cond(smooth, smooth_path, rolling_path)

Does it think? #

The QA quality was evaluated on 1,268 questions about the 600 OOD functions, of a type excluded from training, vectors at 4x achieved 39.0% accuracy as opposed to the text achieving 39.6%. The stock model achieved only 16.9%. This shows that whilst the model also got better at answering code questions through training, there is a statistically insignificant loss of information through the vectors as opposed to the raw text. The table below shows how the two arms split those questions. Despite landing at the same overall accuracy, they disagree on 23.7% of them, each answering questions the other misses.

Dynamic dial #

One nuance of the way the model was trained is the fact that the compression ratio is a parameter to the model, meaning compression ratios not trained on can in theory be utilized.

The model holds up at ratios well outside the interval it was trained at, so it really did learn compression as a dial rather than memorizing the two rates it saw.

Function stacking #

One component of training I included was cases with multiple functions compressed individually and placed within the same context window in an attempt to simulate a more realistic use case (after all, what function is so big it cannot fit within the window?). Stacking was used both for reconstruction and QA training.

Duplicate names are not atypical in Python, for example, there can be many functions by the name __init__ in the same .py file.

Accompanying every function is a text tokenized header, which includes the name of the file and the function. This leads to cases where the model has a hard time differentiating between functions with the same duplicated names.

Hidden thinking states #

Another thing I was curious to look into was the similarities in thinking states between text and vectors. Through this PCA projection (of a handful of Python functions, not the whole test set), it’s clear that the vector positions and text positions largely differ throughout the first layers, but converge to nearly identical positions by the last layer. The numbers agree with the picture. Comparing the decoder’s mid layer states for vectors against text of the same function, CKA similarity measures 0.72, where states for entirely different functions bottom out at 0.08.

Epilogue #

Ultimately this project turned out a massive success. Picking up on the VLMs that inspired it in the first place, it suggests the root of their efficiency lies in their use of latent tokens. With VLMs able to achieve the scores noted at the start of the article, and my small model achieving relatively similar scores, it truly makes me wonder what the frontier edge of this architecture could look like.

Whilst this project alone is not practical for any use, I believe it stands as a proof of concept to be implemented into some larger model at a proper scale, and also paints the picture of the value of latent tokens in these discrete language models. With a larger decoder and a smaller, more intentional encoder, I think this architecture could become fairly useful the same way VLMs have, and there are applications for latent tokens beyond code, such as numerical data. I’d also be curious about the possibility of nondiscrete output tokens, such as for extended reasoning or other applications.

If you’d like to play around with the project, the full code alongside an easy to set up playground is available on GitHub, with the fine-tuned weights on Hugging Face.

I’d also like to give a massive thanks to Evan Conrad from SF Compute for providing usage credits through his new platform givemeanode. I had a great experience using the platform, whether it be its rapid support or agent native tooling, and couldn’t have done the project without their support. I recommend you check out their platform here.