Image: Generative AIs (GPT-3, Copilot, Gemini, Gopher, Chinchilla, PaLM, Human, etc.) train on very large data sets, which can contain billions of texts, images, music or videos.
These deep neural networks are composed of many parameters. For example, OpenAI's GPT-3 model contains 175 billion parameters.
How can the blind increase of these parameters give rise to a form of intelligence?
There is still something deeply disturbing in this cognitive manifestation because it tells us that intelligence can emerge automatically!
* In the short scale used in English-speaking countries, 1 billion is equal to 109 or a thousand million or 1 billion.
Image posted by Sharan Narang and Aakanksha Chowdhery, software engineers, Google Research.
Neural network parameters are the internal variables of an AI model. These parameters are automatically adjusted during learning from input data.
For example, OpenAI's GPT-3 model has 175 billion parameters. OpenAI's DALL-E model has 12 billion parameters. Google's Gemini Ultra model is said to have 540 billion parameters.
The number of parameters depends on the structure of the network, that is to say the number of layers, the number of neurons per layer, and the type of connection between the layers.
The number of parameters is given by the following formula: P = (d+1)h + (h+1)o where d is the number of input neurons , h is the number of hidden neurons, and o is the number of output neurons. The term +1 corresponds to bias, which is an additional parameter added to each layer to avoid the natural human tendency to favor certain outcomes.
The number of parameters influences the learning capacity of a neural network and therefore the performance and behavior of the model. The more parameters there are, the more the system has the capacity to produce correct and consistent results. However there is a limit!
A phenomenon called “overfitting” penalizes the system when the number of parameters is too large compared to the number of data available.
If we want to increase the number of parameters in a neural network, we must also increase the training data.
This explains why operators' appetite for our data is insatiable.
In artificial neural networks (ANN), each neuron performs a calculation that is a weighted sum of its inputs, then applies a thresholded activation function to determine its output which it provides to the next layer.
From this simple mathematical process, a singularity appears.
The network only predicts the next word or token that will follow in the sentence. And yet, an ordered, rational and coherent sentence emerges even though it is the result of a probabilistic process.
For this magician of language who juggles with words without worrying about their meaning, the notion of truth is irrelevant. The system does not seek to provide correct answers, but rather probable sentences.
In other words, a system which has no connection with our reality, devoid of meaning and knowledge and which does not distinguish "true" from "false", can provide an "intelligent" response.
It is thanks to its immense learning corpus that the AI gives the impression of understanding the context of the sentence, the author's intention and the nuances of the language.
There is something deeply disturbing about this manifestation!!
How can a phenomenon as complex and sophisticated as intelligence appear in a virtual environment?
Examples of emerging concepts:
- Just after the Big Bang, the universe is extremely hot and dense. In this extreme environment, from pure energy emerges matter, in accordance with Einstein's equation, E=mc². Thus, elementary particles such as quarks, electrons, neutrinos, which did not exist before, emerge from the early universe.
- Life is an emergent phenomenon, it results from the interaction of simpler components, such as the chemical molecules that constitute it. However, it presents new and irreducible properties to these components. From a certain molecular organization it appears in an environment where it did not exist before.
An emergent concept arises from a more fundamental concept while remaining new and irreducible to it. In other words, new properties appear with the concept emerging from an environment where it was not previously present. These new properties seem to be a natural response to the specific physical conditions of an environment.
IA models before 2017 were trained on much smaller datasets than those used today. They were far from perfect, generative AIs didn't work very well.
As the data available for learning increases, data scientists have intuitively increased the number of parameters. From a miraculous threshold, they noted the appearance of a significant improvement in results.
This phenomenon occurred in 2017 with the GPT-2 (Generative Pre-trained Transformer 2) model which marked a turning point in the field of text generation, by demonstrating its ability to produce human-quality texts.
What happened ?
Before 2017, the scale of models (learning data and neural architectures) increased but nothing happened, performance was poor and they stagnated. Then suddenly when the scale reached a threshold, there was a phase transition. In other words, a change in the physical state of the system, caused by the diversity of data and parameters.
Suddenly, richer, deeper, more complex interactions between neurons appeared.
The remarkable thing about this miraculous evolution is the emergence of more sophisticated cognitive abilities that now seem "intelligent" to us.
Scientists have a hard time explaining this phase transition. However, an “intelligence” does indeed emerge mathematically from the interaction of very simple components, such as data, algorithms, models and parameters!!
What does this emergence from a machine tell us about the nature of intelligence itself?
Machine learning is a non-linear process, meaning that small changes can lead to large changes in the model's behavior. At the moment, we don't understand how the models make their decisions and this makes it difficult to predict their future behaviors.
The field of AI is evolving very quickly, with new technologies and architectures emerging all the time. From the increasing complexity of models, other unexpected properties could emerge, such as creativity, art, understanding of reality and even consciousness.
AI was originally designed to mimic the capabilities of the human brain. To do this, she was inspired by biological neuron models to create artificial neural networks.
It is likely that in the future, AI and brain research will enrich each other.
By using AI and letting it evolve on its own, it is possible that it will provide us with the keys to unlocking the mysteries of the human brain.
“Chance is the god of inventors.” - Pierre Dac (1893-1975) French humorist.
Why Did the Genus Homo Nearly Go Extinct 900,000 Years Ago? 
AlphaGo vs AlphaGo Zero: A Revolution in Artificial Intelligence 
The Next Step for Intelligent Machines 
The First Step Towards the Emergence of Life 
Phylogenetic Tree of Life 
Formal neuron 
The
shadow biosphere 
Decline of Anthropocentrism 
Artificial intelligence: the explosion of gigantism 
When AI models train on their own data, they go mad! 
Emergence of artificial intelligence: Illusion of intelligence or intelligence? 
The
horseshoe crab, a living fossil! 
Biosignatures or presence of life in the Universe 
Challenge and threat of Artificial Intelligence 
How do machines understand, interpret and generate language in a similar way to humans? 
How does an artificial neural network work? 
Origin of life on Earth: Panspermia theory 
Origin of life on Earth: White smoker theory 
Why 37 degrees Celsius? 
Thermodynamics of the sandpile 
Are we alone in the universe? 
Trace of frozen life in Siberia 
Ice cores tell us about our past 
Life evolves in the shelter of glaciations 
Organ regeneration, the salamander 
Cosmic rays and the mutation of species 
Mephisto, the little worm of the depths 
Discovery of solid buckyballs in space 
Bipedalism in
hominids 
Kamchatka
giant crab 
The passage between the inert and the living 
From particles to biochemical life 
Egocentric vision, the man at the center 
Megapod
uses volcanic heat 
Ardi is 4.4 million years old 
Natural selection, the birch moth 
The explosion of life in the Ordovician 
Liquid water, an accelerator of chemical reactions 
Neandertal 
Asimo the
future humanoid 
Conditions for the appearance of life 
Fermi's paradox or Plato's cave 
The Tardigrade, the immortal animal 
Toumaï,
7 million years old 
Border between inanimate and living 
The incredible life of the abyss 
Cyanobacteria create toxic gas 
The short history of the evolution of life 
The smallest frog in the world 
The explanation of the Little Ice Age 
Ashen light, the proofs of life 
Bioluminescence
of living organisms 
Beyond our senses, the great scientific revolutions 
The primitive
soup