In May 2025, Sakana AI introduced Continuous Thought Machines (CTMs), a neural model that unfolds computations over explicit micro-ticks, weaving a temporal axis into every neuron's update rather than relying solely on feed-forward passes.[1]
What Makes CTMs Different?
Standard deep networks map inputs to outputs via fixed, layer-by-layer transformations. CTMs instead define an internal time dimension where hidden states evolve continuously:
Internal Time Dimension
Inputs trigger a sequence of micro-ticks during which the network re-evaluates and refines its hidden states, similar to continuous-time recurrent networks studied in the literature.
Neuron-Level Dynamics
Each unit retains a brief activation history, enabling temporal filtering and resonant behaviors akin to the third-generation spiking neuron models[2].
CTMs also harness neural synchronization, encoding information in the relative timing of peaks across neuron groups, a principle shown to boost memory robustness and multiplexing in biological circuits[3].
"The Continuous Thought Machine is a new kind of artificial neural network which uses the synchronization between neuron dynamics to solve tasks."
— Sakana AI Research Team
Adaptive Computation: Thinking Longer for Harder Problems
Building on the Adaptive Computation Time concept, CTMs include a learned halting mechanism that allocates more micro-ticks to challenging inputs and fewer to simple ones, trading extra "thinking steps" for greater accuracy only when needed[4].
This mirrors human strategy, quick judgments on easy tasks, deeper deliberation on tough ones, while optimizing compute.
Impressive Task Performance
Even as a research prototype, CTMs produce competitive results across a range of challenging benchmarks. What makes these results particularly notable is that they're achieved with a fundamentally different architecture than traditional neural networks. While most cutting-edge AI systems rely on established designs with years of optimization, CTMs represent a completely new approach that's still in its early stages of development.
The Sakana AI team has tested CTMs on diverse tasks requiring different types of reasoning, from recognizing objects in images to navigating complex environments and solving logical problems. In each case, the time-based processing approach has demonstrated promising capabilities that suggest this new paradigm could eventually compete with or even surpass conventional approaches in certain domains.
Image Classification
On ImageNet-1K, CTMs attain 72.5% top-1 and 89.9% top-5 accuracy, a strong showing given the novel temporal paradigm[1].
This means CTMs correctly identify objects in images 72.5% of the time on first guess, and nearly 90% when allowed five guesses. Unlike traditional models that process an entire image at once, CTMs examine different parts of the image over time, similar to how humans might scan a picture, making these results particularly promising.
Maze Solving
CTMs form an internal spatial representation and solve 2D mazes without explicit embeddings, generalizing to unseen layouts beyond training[5].
What's fascinating is that CTMs navigate mazes without being programmed with spatial concepts. They develop their own "mental map" through thinking over time. Researchers can actually visualize the CTM tracing potential paths, much like how you might follow routes with your eyes when solving a paper maze.
CTMs also excel at several other challenging computational tasks[1]:
- Sorting: CTMs can organize jumbled sequences of numbers into the correct order, showing an ability to compare values and determine their relative positions, a fundamental problem in computer science.
- Parity Checks: These models can determine whether a binary sequence contains an odd or even number of 1s, demonstrating their capacity for logical operations that require tracking information across an entire input.
- Sequential Question-Answering: When presented with information followed by questions about that content, CTMs can maintain context over time to provide accurate answers, suggesting potential for more complex language understanding tasks.
- Reinforcement Learning: In environments where decisions must be made based on limited information and delayed rewards, CTMs have shown an ability to learn effective strategies through trial and error, much like humans learning from experience.
What's particularly remarkable about these diverse capabilities is that they emerge from a single architectural approach, the time-based processing that defines CTMs, rather than requiring specialized adaptations for each type of problem. This versatility suggests that temporal processing may be a more fundamental aspect of intelligence than previously recognized in AI research.
Natural Interpretability
Because CTM reasoning unfolds visibly over micro-ticks, researchers can trace representation changes step by step, offering transparency akin to spiking-neuron analyses[6].
One of the long-standing challenges in AI has been understanding how neural networks make decisions. Traditional deep learning models often function as "black boxes" where it's difficult to understand why they reach particular conclusions. CTMs provide a refreshing alternative by allowing researchers to observe how the network's internal representations evolve at each micro-tick, similar to watching someone solve a problem while thinking out loud.
This inherent interpretability has important implications for building trustworthy AI systems. In applications where understanding the reasoning behind decisions is crucial, such as healthcare diagnostics or autonomous vehicle navigation, CTMs could provide both high performance and the necessary transparency to verify that the system is working as intended.
Current Limitations and Future Directions
Dynamic-time models like CTMs incur higher training costs and require new tooling for sub-tick schedules. Sakana AI's open-source release aims to spur ecosystem support[7].
Introducing multi-timescale heterogeneity, varying tick granularity across layers, could further enrich temporal representations, as demonstrated in recent multi-compartment spiking models[8].
Despite CTMs' promising results, their computational demands present practical challenges for widespread adoption. The extra "thinking time" that gives CTMs their unique capabilities also means they typically require more processing power than traditional models of comparable size. This is analogous to how human deliberation consumes more mental energy than instinctive reactions. Sakana AI has released open-source code and documentation to encourage community contributions toward more efficient implementations.
Looking ahead, researchers are exploring how different parts of a CTM could operate on varying timescales, much like how the human brain processes information at different speeds in different regions. For example, some layers might update rapidly to capture fine details, while others change more slowly to maintain stable, high-level representations. This multi-speed approach could potentially enable CTMs to simultaneously track both immediate events and longer-term patterns, further enhancing their ability to handle complex, dynamic environments.
References
[1] Luke Darlow, Ciaran Regan, Sebastian Risi, Jeffrey Seely, and Llion Jones, "Continuous Thought Machines," arXiv preprint arXiv:2505.05522, May 2025. Online
[2] W. Maass, "Networks of Spiking Neurons: The Third Generation of Neural Network Models," Neural Networks, vol. 10, no. 9, pp. 1659–1671, Dec. 1997. Online
[3] V. L. Galinsky and L. R. Frank, "Critically synchronized brain waves form an effective, robust and flexible basis for human memory and learning," Scientific Reports, vol. 13, article 4343, Mar. 2023. doi:10.1038/s41598-023-31365-6 Online
[4] A. Graves, "Adaptive Computation Time for Recurrent Neural Networks," arXiv preprint arXiv:1603.08983, Mar. 2016. Online
[5] E. Parisotto and R. Salakhutdinov, "Neural Map: Structured Memory for Deep Reinforcement Learning," in ICLR, 2018. Online
[6] H. Chen, J. Ding, et al., "Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics," Nature Communications, vol. 14, article 44614, 2023. doi:10.1038/s41467-023-44614-z Online
[7] Sakana AI, "continuous-thought-machines," GitHub repository, 2025. Online
[8] Z. Wang, R. G. Smith, and M. A. Farkho, "Heterogeneous quantization regularizes spiking neural network dynamics," Scientific Reports, vol. 13, article 96223, 2025. doi:10.1038/s41598-025-96223-z Online