Understanding THOR: Running LLM Inference on Ciphertext

Tech & Insight — The Homomorphic Encryption & Private AI Series (4 parts)

• ① Understanding Fully Homomorphic Encryption (FHE): Computing on Data Without Ever Opening It
• ② Understanding THOR: Running LLM Inference on Ciphertext — current post
• ③ Understanding the GL Scheme: Homomorphic Encryption Redesigned from the Ground Up for Matrices — coming soon
• ④ Understanding Private AI: Using AI Without Handing Over Your Data — coming soon

At a glance (TL;DR)

• THOR is a framework that runs the entire transformer inference process on top of fully homomorphic encryption (FHE). Because the user's input stays encrypted throughout the LLM computation, not even the server operating the model can see what the user asked or what answer they received.
• A joint research result of DESILO and Professor Miran Kim's team at Hanyang University, it was presented at ACM CCS 2025 (Taipei), a top-tier venue in computer security. First author Jungho Moon took part while at DESILO.
• The core idea is a redesign of matrix multiplication to fit the nature of homomorphic encryption. Compared with the previous best techniques, it is 5.3× faster for plaintext–ciphertext matrix multiplication and 9.7× faster for ciphertext–ciphertext matrix multiplication.
• As a result, it completes the entire 128-token inference for a BERT-base model in 2 minutes on a single GPU. That is the product of continued optimization from the 10 minutes reported when the paper was released, for a task that took 2.7 hours in the immediately preceding work using the same approach. Accuracy stays within about 1 percentage point of the plaintext model.

Table of contents

1. Why "encrypted inference"
2. What makes it hard: the gap between transformers and homomorphic encryption
3. Core technique #1: multiplying matrices along the diagonal
4. Core technique #2: taming nonlinear functions with polynomials
5. Core technique #3: a pipeline that conserves bootstrapping
6. Results: 2 minutes, and 1%p
7. Security model: what it guarantees
8. After THOR: the Private AI roadmap
9. Frequently asked questions (FAQ)
10. References

---

1. Why "encrypted inference"

The more widely generative AI is used, the clearer one fact becomes: the moment you enter something into an LLM, that content is handed straight to whoever operates the model. For organizations handling sensitive data — work documents, customer information, medical records — this structure is fatal, and it is exactly why AI-adoption discussions keep stalling in regulated industries such as finance, healthcare, and the public sector.

Fully homomorphic encryption (FHE) is the most fundamental answer to this problem. Because it can compute on data while it stays encrypted, LLM inference on ciphertext is possible in principle. The catch was speed. Applying FHE to an LLM with at least hundreds of millions of parameters meant a single inference could take hours, or only worked on very small models.

THOR is the result of closing that gap. Its name comes from Transformer and HOMomOrphic encRyption. It was developed jointly by DESILO and Professor Miran Kim's research team at Hanyang University, with researchers from UTHealth Houston (the University of Texas Health Science Center at Houston) also taking part. The paper was presented at ACM CCS 2025, held in Taipei in October 2025. CCS is counted among the four top-tier venues in computer security, alongside IEEE S&P, USENIX Security, and NDSS.

For homomorphic encryption itself, see 「Understanding Fully Homomorphic Encryption (FHE)」 in Insights, which covers the technology as a whole.

---

2. What makes it hard: the gap between transformers and homomorphic encryption

The transformer is the core architecture of generative AI, widely used across open-source projects and AI services, and it consists of two broad kinds of computation. One is enormous matrix multiplication. From the query/key/value computations of attention to the feed-forward layers, most of the compute is matrix multiplication. The other is nonlinear functions — functions mixing exponentials, division, and square roots, such as softmax, GELU, and LayerNorm, repeated at every layer.

From homomorphic encryption's point of view, both are formidable. A ciphertext is a long vector holding tens of thousands of values, and the supported operations are only vector-wise addition, multiplication, and rotation. The free access to rows and columns that plaintext allows becomes, in ciphertext, a chain of expensive rotations and key-switching operations. Nonlinear functions cannot be computed directly at all and must be replaced with polynomial approximations, and bootstrapping — which reinitializes the noise that grows as operations accumulate — is the single most expensive procedure in homomorphic encryption.

Prior work proceeded along two lines.

THOR's goal is clear: keep the non-interactive structure that requires no communication, while pushing speed into the practical range.

---

3. Core technique #1: multiplying matrices along the diagonal

The first of THOR's core ideas is a homomorphic-encryption-friendly matrix-multiplication algorithm. The starting point is to encode the matrix by diagonal rather than by row or column. Each diagonal of the product AB is expressed exactly as the element-wise product of rotated upper-diagonal vectors of A and the lower-diagonal vectors of B, summed together — a structure in which matrix multiplication is completed using only the operations homomorphic encryption does best: vector rotation and element-wise multiplication.

Two further designs build on this.

Dense packing. A single ciphertext has tens of thousands of slots, but storing just one diagonal leaves most of them wasted. THOR uses an encoding that layers the diagonals of multiple matrices into one ciphertext, filling the slots and processing the 12 heads of multi-head attention in parallel.

Multiplication-specific algorithms. Transformers involve two multiplications of different character.

• Plaintext × ciphertext multiplication (PC-MM): the product of model weights (plaintext) and activations (ciphertext). By splitting the matrix into blocks and multiplying blocks on the same diagonal in parallel, it cuts the required rotations down to the block-size level rather than the full matrix size.
• Ciphertext × ciphertext multiplication (CC-MM): the case where both sides are ciphertext, as in the attention score (QKᵀ). A Baby-Step Giant-Step strategy defers the rotation of the input ciphertext so it is applied only once to a partial sum, halving the number of rotations.

The results are borne out in the numbers. Across every matrix operation required for multi-head attention, it consistently outpaced the previous best techniques (BOLT, Powerformer).

Measured by the number of key-switching operations — the basic operation the paper uses as its complexity metric — it cut 89–98% in the attention-head computation compared with prior methods.

---

4. Core technique #2: taming nonlinear functions with polynomials

What homomorphic encryption can compute is, in the end, only addition and multiplication — that is, polynomials. The exponential and division in softmax, GELU, and the inverse square root in LayerNorm all have to be converted into polynomial approximations. An inaccurate approximation breaks model accuracy; a heavy one breaks speed.

Softmax: two-stage approximation. In practice, BERT's softmax inputs are spread across a wide range on the order of [-44, 50], making the exponential hard to approximate in one shot. THOR solved this by combining a normalize-and-square strategy [4] — which narrows the range to approximate the exponential stably, then repeatedly squares and normalizes to return to the original range — with an adaptive technique [5] that tunes the convergence rate at each step of Goldschmidt's iteration. As a result, the number of iterations dropped from 7 to 3, and the most expensive bootstrapping from 8 to 3, for a 2.64× speedup over the previous best technique. Converted to a single softmax, that is about 14.5 ms.

GELU and Tanh: composition of low-degree polynomials. Instead of one high-degree polynomial, it obtains the same precision from the composition of two low-degree polynomials, with fewer operations. Compared with the prior method, the number of multiplications dropped from 56 to 39 and multiplicative depth from 14 to 11.

LayerNorm. The inverse square root needed for normalization is likewise computed with the same adaptive iteration.

---

5. Core technique #3: a pipeline that conserves bootstrapping

Even if individual operations are fast, sloppy assembly makes the whole slow. THOR also embeds design at the level of the end-to-end pipeline.

A representative example is transpose encoding. Computing the query as Q = XW directly would require rotating the ciphertext X many times, but computing it in the transposed form Qᵀ = WᵀXᵀ shifts the rotation onto the plaintext weights, so ciphertext rotation disappears. THOR designed the entire inference process consistently in this transposed form.

The placement of bootstrapping is handled the same way. THOR concentrates bootstrapping at the point where the number of intermediate ciphertexts is smallest. Even so, bootstrapping accounts for 56% of total inference time (in the paper's implementation) — that is how large this procedure's share is. For comparison, the prior non-interactive work (NEXUS) requires a total of 6,016 bootstrapping operations because of its encoding structure, estimated to take about 1.7 hours on that alone. The choice of encoding structure sets the number of bootstrapping operations, and the number of bootstrapping operations sets overall performance.

---

6. Results: 2 minutes, and 1%p

Accuracy was verified on three GLUE benchmark tasks. There are three points of comparison: the plaintext original model; the plaintext approximate model, in which every nonlinear function is replaced with a polynomial approximation and run without encryption; and the encrypted inference (THOR), which runs that approximate model on ciphertext. The gap between encrypted inference and the original is about 1 percentage point.

The middle column is what to watch. The plaintext approximate model, with only the approximation applied, matches or even exceeds the original on all three tasks. That means the source of the slight accuracy difference is not the approximation technique but the numerical error of the homomorphic computation — the approximation itself is essentially lossless. The full source code is available on GitHub.

---

7. Security model: what it guarantees

There are two parties in THOR's scenario: the server (service provider) holding the fine-tuned model, and the client requesting inference. The client encrypts the embedding of its input sentence and sends it; the server computes on ciphertext alone from start to finish and returns an encrypted result. Only the client, who holds the secret key, can decrypt it.

The threat model assumes an "honest-but-curious" server — one that follows the protocol but tries to peek at the data passing through. The security goal is that the server can learn nothing about the client's input, and this is guaranteed by the semantic security of the CKKS scheme, i.e. the mathematical hardness of the Ring-LWE problem. Because the protocol structure is such that decryption results are never shared externally, it is also safe against the known key-recovery attacks on CKKS.

Being non-interactive matters in practice too. Unlike MPC-family approaches, there is no need for the server and client to exchange data every round of computation, so there is neither tens of gigabytes of communication nor any dependence on network latency. The client sends the ciphertext once and receives the result once.

---

8. After THOR: the Private AI roadmap

What THOR showed is that "encrypted LLM inference" has moved beyond proofs of concept on tiny models or hours-long demos into the practical range of minutes. This is exactly the technical foundation on which Secure RAG — where a user's prompts and documents stay encrypted throughout retrieval and generation — and DESILO's DESILO Private AI product family stand.

The directions for expansion are open as well. As the paper notes, THOR's matrix-multiplication and nonlinear-approximation techniques can be applied beyond BERT-style encoders to GPT-style decoder models and other transformer-based tasks. The remaining performance gap keeps closing along three fronts: algorithms, GPU optimization, and dedicated hardware. In particular, the fifth-generation GL (Gentry-Lee) Scheme, which redesigns from the ground up the matrix multiplication that accounts for most of the computation in large AI models, is a card to accelerate the direction THOR proved to the next stage.

---

9. Frequently asked questions (FAQ)

Q1. Can the server really see nothing?

Correct. The inputs, all intermediate computations, and the final result are handled by the server as ciphertext only. The ability to decrypt lies solely with the client holding the secret key, and this is guaranteed not by a policy such as access control but by the mathematical hardness of lattice problems.

Q2. Can it get faster?

More than half the time (56%, in the paper's implementation) is taken by bootstrapping, which reinitializes noise. Improvements to bootstrapping itself, GPU and dedicated-hardware acceleration, and new schemes that redesign the matrix-multiplication structure are the three paths that will narrow the gap. Given that the immediately preceding non-interactive work took 2.7 hours, the paper's 10 minutes is a starting point rather than a destination — and indeed, continued optimization after the paper's release has since brought it down to about 2 minutes.

Q3. Doesn't accuracy drop?

On the GLUE benchmark, the gap from the plaintext model is about 1 percentage point. Since the approximation-only plaintext model is sometimes even better than the original, the source of the slight difference was analyzed as the numerical error of homomorphic computation, not the approximation.

Q4. Does the model have to be retrained?

In principle, no separate retraining for homomorphic encryption is required. That said, replacing operations that are heavy under homomorphic encryption — exponentials, division, and the like — with lighter polynomials and then fine-tuning to preserve the original model's performance enables more efficient inference under homomorphic encryption, and a good deal of related research is underway.

Q5. How far does it extend?

The paper completed end-to-end verification on the BERT-base model and shows that the proposed techniques can extend to GPT-style decoder models and other transformer tasks. Scaling to larger models is the next task, one that hinges on the interplay of algorithms and hardware acceleration.

---

10. References

1. Jungho Moon, Dongwoo Yoo, Xiaoqian Jiang, Miran Kim, "THOR: Secure Transformer Inference with Homomorphic Encryption," ACM CCS 2025: https://doi.org/10.1145/3719027.3765150
2. THOR source code: https://github.com/crypto-starlab/THOR
3. Liberate.FHE (now DESILO FHE): https://fhe.desilo.dev/latest/
4. Wonhee Cho, Guillaume Hanrot, Taeseong Kim, Minje Park, Damien Stehlé, "Fast and accurate homomorphic softmax evaluation," ACM CCS 2024: https://dl.acm.org/doi/10.1145/3658644.3670369
5. Jungho Moon, Zhanibek Omarov, Donghoon Yoo, Yongdae An, Heewon Chung, "Adaptive successive over-relaxation method for a faster iterative approximation of homomorphic operations," Cryptology ePrint Archive 2024: https://eprint.iacr.org/2024/1366

---

Tech & Insight — The Homomorphic Encryption & Private AI Series (4 parts)

• ① Understanding Fully Homomorphic Encryption (FHE): Computing on Data Without Ever Opening It
• ② Understanding THOR: Running LLM Inference on Ciphertext — current post
• ③ Understanding the GL Scheme: Homomorphic Encryption Redesigned from the Ground Up for Matrices — coming soon
• ④ Understanding Private AI: Using AI Without Handing Over Your Data — coming soon

---

Coming up next

The next installment turns to the fifth-generation GL (Gentry-Lee) Scheme, which redesigns from the ground up the matrix multiplication that bottlenecks large-scale AI computation. Co-invented by FHE's founder Craig Gentry and DESILO, how far does it push the "minute-scale encrypted inference" that THOR proved? We'll break it down right here on this blog next week.