Geoffrey Hinton, one of the so-called godfathers of artificial intelligence, urged governments on Wednesday to step in and make sure that machines do not take control of society.

Hinton made headlines in May when he announced that he quit Google after a decade of work to speak more freely on the dangers of AI, shortly after the release of ChatGPT captured the imagination of the world.

The highly respected AI scientist, who is based at the University of Toronto, was speaking to a packed audience at the Collision tech conference in the Canadian city.

The conference brought together more than 30,000 startup founders, investors and tech workers, most looking to learn how to ride the AI wave and not hear a lesson on its dangers.

"Before AI is smarter than us, I think the people developing it should be encouraged to put a lot of work into understanding how it might try and take control away," Hinton said.

"Right now there are 99 very smart people trying to make AI better and one very smart person trying to figure out how to stop it taking over and maybe you want to be more balanced," he said.

Hinton warned that the risks of AI should be taken seriously despite his critics who believe he is overplaying the risks.

"I think it's important that people understand that this is not science fiction, this is not just fearmongering," he insisted. "It is a real risk that we must think about, and we need to figure out in advance how to deal with it."

Hinton also expressed concern that AI would deepen inequality, with the massive productivity gain from its deployment going to the benefit of the rich, and not workers.

"The wealth isn't going to go to the people doing the work. It is going to go into making the rich richer and not the poorer and that's very bad for society," he added.

He also pointed to the danger of fake news created by ChatGPT-style bots and said he hoped that AI-generated content could be marked in a way similar to how central banks watermark cash money.

"It's very important to try, for example, to mark everything that is fake as fake. Whether we can do that technically, I don't know," he said.

The European Union is considering such a technique in its AI Act, a legislation that will set the rules for AI in Europe, which is currently being negotiated by lawmakers. 

'Overpopulation on Mars'

Hinton's list of AI dangers contrasted with conference discussions that were less over safety and threats, and more about seizing the opportunity created in the wake of ChatGPT.

Venture Capitalist Sarah Guo said doom and gloom talk of AI as an existential threat was premature and compared it to "talking about overpopulation on Mars", quoting another AI guru, Andrew Ng.

She also warned against "regulatory capture" that would see government intervention protect the incumbents before it had a chance to benefit sectors such as health, education or science.

Opinions differed on whether the current generative AI giants - mainly Microsoft-backed OpenAI and Google - would remain unmatched or whether new actors will expand the field with their own models and innovations.

"In five years, I still imagine that if you want to go and find the best, most accurate, most advanced general model, you're probably going to still have to go to one of the few companies that have the capital to do it," said Leigh Marie Braswell of venture capital firm Kleiner Perkins.

Zachary Bratun-Glennon of Gradient Ventures said he foresaw a future where "there are going to be millions of models across a network much like we have a network of websites today."