Tech from the Trenches Chapter 10

Germán is a 41-year-old entrepreneur who doesn’t identify himself with the term ‘entrepreneur.’ His passion for Artificial Intelligence (AI) began in 2004-2005 when he took courses related to AI during his Erasmus program. Later, he decided to pursue a Ph.D. in machine translation, which he completed in 2012. During this period, he also specialized in machine learning.

In 2014, after completing a two-year postdoctoral position, Germán and his partner Vicent Alabau founded Sciling. They decided to start the company for two main reasons:

a) What was being developed in the university didn’t reach the market in terms of production.

b) They observed a brain drain starting in 2014, resulting in a decapitalization of the country and a lack of opportunities.

Initially, they focused on developing their own machine translation product for e-commerce, but they didn’t achieve the expected success. In 2017, they made the decision to change course and become a “boutique for Artificial Intelligence solution development.” Germán explains that they specialize in taking technologies that are in the laboratory phase and applying them to real use cases, aiming for high impact and a high level of specialization in research and development, as well as operationalization of AI models.

Currently, Sciling has a team of approximately 22-23 people who share a strong vision of the impact of AI on society.

How is it possible for machines to understand text?

Germán mentions that the impact of ChatGPT has surprised everyone. Not only ChatGPT or Stable Diffusion (for images), but all machine learning used to be very ad hoc. However, in 2014, there was a significant shift with the introduction of neural networks. Google released Transformers in 2017, which was revolutionary. Since then, Transformers have evolved with the incorporation of more and more data, with a daily cost of several million euros for maintenance.

Between ChatGPT 3 and 4, something interesting called ChatGPT 3.5 has emerged, which is a fine-tuning of ChatGPT 3 where the output is adapted to what humans expect to receive as a result. GPT-3 and 4 are large Language Models (both discriminative and generative) based on neural networks.

According to Germán, at a basic level, there is no difference between neural electrical impulses and the binary 0s and 1s of computer models. This leads us into a philosophical scenario.

What is currently being done in Natural Language Processing (NLP)?

According to Germán, the current focus is on the ability to relate documents to documents in the field of NLP. Until now, this problem has been approached using conventional approaches, such as the use of embeddings from large language models.

However, it is expected that generative models will be used in the future to solve this type of challenge. The goal is to be able to link contracts with specifications, job offers with resumes, among other examples. In this field, there are many opportunities both in the document linking field and in the development of chatbots.

Chatbots have experienced a significant impact thanks to AI, as much of the prior work in which a decision tree had to be defined to generate responses has been eliminated.

The concept of ‘grounding’ refers to providing ChatGPT with a set of documents and then asking questions about those documents to build a conversation based on that dataset. Having a corporate database with all the knowledge and being able to ask questions about it in relation to your work is a very powerful tool.

Do language and image models pose a significant risk to society?

Germán points out that there is a dose of exaggeration regarding the risks and dangers of language and image models, but it is also true that we are not fully aware of the impact of a technology that is less than 6 months old. Even the developers at OpenAI are not fully aware of all the associated risks. Some of the risks mentioned by Germán include the spread of fake news, among others.

Germán raises concerns that the internet will be flooded with AI-generated content, which can pose a long-term problem. In this scenario, search engines will struggle to generate reliable information. Training models like ChatGPT 4 requires a large amount of high-quality data. If most of the available information is self-generated, we could face a scenario where we cannot train models due to the lack of reliable data. In the case of Stable Diffusion, there are assumptions and problems related to this issue.

In Europe, we are advanced in regulating AI. Is this good or bad to avoid falling behind in a global world?

Germán advises the European Union (EU) on AI issues, and in his opinion, the EU is concerned about having competitive technology compared to ChatGPT. Although there are more advanced models, such as Blum or open-source models, none reach the level of ChatGPT 4. The EU is aware of the need to be competitive in this field.

Germán mentions his friend Francisco Zamora, the director of a biometric identification company, who faced problems with European legislation and couldn’t collect data from European citizens. This led the company to acquire data from Asian countries, resulting in models with a significant bias towards Asian features.

ChatGPT models are transforming jobs as we know them. Germán emphasizes that technology hasn’t destroyed jobs but transformed them. If we consume technology but don’t generate it, we risk losing jobs in the process.

What could we do in Europe in this scenario?

Germán mentions that he doesn’t know exactly what we could do in Europe in this scenario, but it’s something that concerns him. So far, Europe has played more of an “Arbitrator” role than a “Ronaldo” role in the field of AI. Germán highlights the need to try to position the strategy we want to pursue in this field.

In Europe, the market is highly fragmented compared to the United States, and this is an underlying problem in the European Union. These problems are endemic and require attention to effectively address them.

Are we facing the last AI Winter?

Germán responds that he doesn’t know if we are facing the last AI Winter. His AI predictive system is broken, so he can’t provide an accurate answer. However, he points out that the entry of ChatGPT has generated significant attention in the field of AI. There has been a strong boom, and now we are in a calm stage, but in the near future, we will see how many things will transform.

Both individuals and companies are starting to use ChatGPT from the ground up, which is optimizing tasks but will also drastically change the production model. Germán mentions a paradigmatic example in the field of education: teachers use ChatGPT to create exams, and students use the model to answer those questions. In this sense, humans have become an interface between AI models.

In summary, while we cannot predict with certainty if we are facing the last AI Winter, it is evident that we are witnessing a significant transformation in various fields driven by models like ChatGPT.