multimodal-ai January 29, 2025

Get to know what multimodality entails at Microsoft AI

Smiling person standing in front of a backdrop with green leaves. Text overlay reads "Nando de Freitas, AI Science at Microsoft AI.

AI isn’t just a technology to Nando de Freitas — it’s a lifelong pursuit of understanding intelligence itself. From training neural networks 30 years ago to leading cutting-edge research today, he empowers teams to push the boundaries of multimodal AI. As a Vice President at Microsoft AI, Nando is committed to shaping AI that is both thoughtful and human-centered, believing in its potential to improve lives.

How would you describe your job and what you do? 
I am part of a team that aims to advance AI and empower users with it.

How did you first become interested in AI and what motivated you to pursue it as a career? 
I started training neural networks more than 30 years ago because I wanted to know how brains work, what is intelligence, and what is consciousness. It was one of the possible final year undergraduate projects available to me at Wits University in South Africa. Back then, this was a very esoteric topic and very few people cared about it, but I felt that understanding intelligence and harnessing it wisely was the most important thing to do. I still believe this. I had great teachers too. The project was fun, and allowed me to connect math, code, and hardware in building diagnoses tools for control valves used in the South African mines. I sold the ideas and made a little bit of money, which for a 24-year-old student was a huge deal. I no longer had to work long hours selling beer to pay for my studies, I could now simply train neural nets and get scholarships. Pretty sweet I thought.

“Everything will change. AI is the most transformative thing that humanity has experienced since language.”

Nando de Freitas, VP of Microsoft AI

 

What educational background or training do you have that has helped shape your skills? 
At Wits University, I studied electrical engineering. The degree back then involved a lot of math, physics, signal processing, control, information theory, electromagnetics, hardware, and software. It was the perfect background to understand neural nets and to train them. However, what made a big difference is that I had amazing teachers, who empowered me to grow.

Despite this, I was planning to abandon it all and get a job to bring home the bacon because I used to believe that that was the responsible thing to do, and what people expected of me. Then, two big things happened. First, I was introduced to a professor from Imperial College of London, England. I would have never dreamt of going abroad and doing a PhD, but when I spoke to this prof, who to me was like a god descended from Mount Olympus, I realised that I could be as smart as those people in the UK or USA.  For a Portuguese boy, often the victim of racial abuse during Apartheid, this was a huge revelation. I still remember a nice old lady telling me that everyone in the town was proud of me because they never thought a Portuguese boy could go to university and do well. A great many people suffered far more than me.

The second life-changing event was my mother telling me, right after the violent murder of my dad, that he had wanted me to continue with graduate school instead of getting a random job. I had got it all wrong. My loved ones had believed in me, but I hadn’t.  So, after many rejections, I got into Trinity College, Cambridge, and later UC Berkeley. Eventually I became a machine learning professor at UBC and Oxford University. In Cambridge, I learned a lot of probability, statistics, speech, and computer vision. At Berkeley, I learned computer science. Again, I was fortunate to have amazing teachers and mentors. Good teachers are heroes to me. I have tried to give back by helping to lay the foundations for education efforts in Africa and Latin America, including the deep learning Indaba and Khipu AI.

What are you currently working on? 
I am a member of the multimodal team, which focuses on perception and generation, e.g. to understand what an image is about, or to generate a fully interactive movie or game.

Can you share an overview of your role and how it specifically relates to Microsoft AI? 
I am a VP, which basically means I do my best to empower a group of exceptionally talented engineers and researchers to achieve ambitious goals. The team is everything. I learn so much from everyone.

I also like to take time to do my own work often. I love keeping up with AI papers, learning, coding, and understanding what people in my team are doing.

What role do user feedback and usability play in your process? How do you incorporate feedback to improve your work? 
Feedback is super important. I won’t get into details here, but I strongly recommend the book “thanks for the feedback”.

With the introduction of AI, what changes or shifts do you anticipate within the industry? 
Everything will change. AI is the most transformative thing that humanity has experienced since language.

“If harnessed well, AI could empower every person to live are more meaningful and healthier life.”

Nando de Freitas, VP of Microsoft AI

What excites you as a VP with the integration of AI? 
If harnessed well, AI could empower every person to live are more meaningful and healthier life. As a VP at MAI, I try to put users first when creating new products and experiences.

How do you stay updated on current tech trends and emerging practice?

These days it is nearly impossible to go anywhere where people aren’t talking about AI. Sometimes, I find it a bit overwhelming.

Looking back at your journey, what advice would you give to aspiring product marketers who are just starting out on their own path? 

Believe in yourself, dream big, and know you too are responsible for shaping the future of humanity and our planet. You are not perfect, and you will likely make many mistakes, but if you have a dream, curiosity, openness to feedback, self-reflection, empathy, and compassion, you will likely succeed.

A modern office space with large windows and a staircase. Three people walk in different directions while a person works at a desk near the window. The interior is bright, with natural light streaming in.