It’s not a term that trips easily off the tongue. Nor does it paint a pretty picture of the astounding visuals it delivers. But Gaussian splatting is the revolutionary technology for how we see the world
Currently, there are three major approaches to volumetrically modeling 3D visuals. The oldest is photogrammetry, where a camera circles the intended object and creates a mesh of interconnected dots. Where there aren’t specific dots, the program fills in the area with its best guess.
The strength of photogrammetry is that it scales and retains spatial data like GPS coordinates. This is important for things like mapmaking or architectural applications, as it keeps data on the actual size of things in, say, miles or inches or millimeters.
The second form of 3D modeling is NERF, short for neural radiance fields. As you may have gathered from the word neural, there is AI involved in this. Specifically, deep learning. Why NERF is different from photogrammetry is that it not only maps the object, but it will also give you data about other objects in the frame. For instance, the sky or distant mountain ranges. In contrast to photogrammetry, NERF provides a more comprehensive model of things.
This is why NERF is popular for virtual production. Even with a very rough video, you can return to the studio and work with the scene, greatly enhancing and improving it beyond the original capture.
The latest entrant is Gaussian splatting.Gaussian splatting achieves similar things as NERF but takes a much different approach.
G-splat starts with a cloud field, where each dot is almost like a point in a starfield. Each dot then is mapped in relationship to each other dot. As we move around one point all the other points change relative to our point of view.
This is done through some very clever and efficient math. Every time we move, every 3D point changes its shape (covariance), color and opacity (alpha).
From one perspective a point can look round, from another it becomes stretched and spheroid. From one perspective a dot in the cloud can be very bright, from further away it becomes dim. In g-splatting everything is relative. Everything is interconnected. Not unlike how gravity connects the planets and stars our universe.
The world through the eyes of Gaussian splatting
To know how g-splatting came about it helps to know something about Carl Freidrich Gauss. Sometimes called “The prince of mathematicians,” Gauss was born in 1777 and is frequently grouped with Newton and Archimedes as one of the history's greatest mathematicians.
One of Gauss’s great contributions was in determining how the stars and planets move. Building on Newton’s theories of celestial mechanics, Gauss created complex astronomical calculations, or algorithms, for how the planets tug and pull on each other, and speed elliptically across our solar system.
In a time where telescopes weren’t very good, smaller planets would come and go from out vision. No one knew where they would appear next.
The minor planets, or asteroids, Ceres and Pallas presented such a problem. Science had no way of locating them at any given time. Gauss made these two disappearing and reappearing planets his problem. And he cracked the code.
After running Gauss’s numbers, there was Ceres, exactly where he said it would be. This made Gauss a kind of rock star. Because while most couldn’t not understand his other math, everyone got the planets. Ceres became the first minor planet, or asteroid, to be discovered.
It was still a wonder how he did it well into the 20th century. From “Men of Mathematics” by E. T. Bell, published in 1937:
“The mere arithmetic necessary to establish an orbit with accuracy sufficient to ensure that Ceres on her whirl around the sun should not be lost to telescopes might well deter and electrically-driven calculating machine even today; but to the young man whose inhuman memory enabled him to dispense with a table of logarithms when he was hard pressed or too lazy to reach for one, all this endless arithmetic – logista, not arithmetica – was the sport of an infant.”
Carl Freidrich Gauss, the Prince of Mathematicians
What Gauss created was a way revealing the interconnectedness of the planets. Where each would be and how it’s journey would appear from any perspective. And he did it with pure math, which is exactly what our modern computers excel at. Number crunching.
Because G-splatting employs highly elegant math, it currently provides the fastest and most accurate way to process visuals. Because of this, it has incredibly fast refresh rates, meaning it can process large scenes at up to 600 frames per second.
As it only takes 24 FPS to trick us into thinking we are seeing reality, this leaves a lot of extra information to play with. In a virtual world, g-splatting can instantly create virtual worlds that we can seamlessly move through with photorealistic resolution. The natural application is video games. But it goes beyond this
In film or content production it means that if we create point clouds for anything – trees, cars, hamburgers, and we can then dynamically animate them into scenes that are also the by-product of point clouds.
What this means is that the importance of physically filming things in the field is greatly reduced. Films, commercials, and digital content can be made on with photorealistic fidelity on a laptop.
This changes everything.
Content will no longer be made in a linear fashion, coming off a production line.
It will be made by a process of simulation. And stories will be ladled out from fluid data visualizations enabled by g-splatting.
Creation will soon be the process of pouring out solutions.
Here is a technical paper that was release at Siggraph,2023 if you would like more: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Also check these guys out. They are stoked: https://www.youtube.com/watch?v=juRMRej2d5c
Welcome to Nextness. More than a newsletter, a mindset.
This film caught my eye because I’ve been there. The over-enthused maker blathering on about the new world of AI.
But these folks do a better job than I do.
Brought to us by the people at Runway who make the state-of-the-art text to video solution, we get a modern film that is a reflection of our current state: an ever-changing blend of AI and human creativity.
The second we think we have things figured out, everything shifts, and we find ourselves in a new land.
It’s a simple film that is complimented with cutaways and interstitials that I presume are mostly AI generated.
It’s a liberating approach for any of us who have spent hours in the edit looking for just the right stock image. It’s a reminder that whatever we use it should be interesting and the viewer will make what they will of it.
If it is exactly what the viewer expected why would we use it in the first place? It’s the original sin of corporate filmmaking that AI has pleasantly and surprisingly dispatched.
Although the film seems a little forced at first – I was having flashbacks of “My Dinner with Andre” – things soon turn fluid and conversational
The conversation soon turns to themes we all find ourselves also contemplating. Is AI just a better version of what already exists? Or will it spawn something new? Can we really train AI to be just like us when we ourselves change all the time? Why will humans always matter?
What is interesting is that this short episode relies more on live action film than generative video.
Right now, the best blend is the old and the new. To make something completely satisfying, generative video isn't quite here yet. And the people at Runway probably get this.
But it's coming. Fast.
In the end, I discovered that the couple are real people with a background in AI. Claire Evans is a musician who made a machine-generated album. And Stephen Marche is an author wrote the first AI-generated book to be reviewed by the New York Times.
This week I saw what in my mind was the first legitimately made AI commercial. It was for the BMX Xi and it featured an AI influencer. Everything about had the overly rounded and smoothed out beauty lens feel we associate with generative AI imagery.
And it mostly worked. Because that's the way car commercials are. Slightly fantastic, overly slathered with video effects, and strangely distant. Like buying a luxury car puts you on a different plane of reality.
Perfume commercials, cosmetics, beauty and jewelry spots are no different. So they will are all prime candidates to lead the vanguard of text-to-video advertising.
Where AI has been less is convincing is slice of life spots. You may remember that weird beer spot that made the rounds recently. Or the video I shared a few versions back called 'Creepy Heidi."
Capturing “reality” with AI is a lot different than serving up fantasy.
This week, Michael Matassa, a commercial editor, took a shot at the food category with “Meat Heads.”
As a video editor Michael finds himself with a front row seat to the revolution that is taking place in the visual arts:
"I really feel like I'm in the early days of, of cinema."
Yet, he doesn't fully understand why everyone doesn't feel this way:
"It's interesting how people are angry at it and are very dismissive of it. And I'm really shocked because I'm like, oh my god, this is this is a whole new medium. We are living in a time, that is like when cinema started, or when radio started, or color television. And here we are being given this amazing opportunity to take these tools and create something entirely new with them."
That's what Michael is doing.
“Meat Heads” is what he calls his sophomore effort in generative commercial making. His first was a funny short for a mythical music festival called Absurd-A-Fest. It got picked up by a few sites who special in posting AI films and spurred him on to try something else.
That it was lunchtime, could have also played a role in his vision:
“I imagined the classic burger place. I was looking for Route 66. In the 60s, driving down the highway, you're finding your burger joint. And it's out there. And you go, and it's this great little dive place that you go to, and you pull up to and you get a great burger.”
From there, Michael philosophy is to use whatever tool works:
"When I'm making these commercials, I'm trying to figure out how to get the image that I want. And I'll something And if that's not working, I'll move onto something else. To me, that's the fun part. It's a tool, but it's a tool with 20 with paint brushes. I can either use the small one or the large one. That's what these apps are to me. Sometimes I just get, I use one app for one little thing, and never use it again. So I'm constantly toying with these different apps to find the result that I'm envisioning."
"Meatheads" began with creating the physical presence of the restaurant:
“I did that in Ideogram. Ideogram knows how to spell. So I made the Meat Heads logo, the storefront, the whole picture, because it does really cool image generation as well.”
Taking his inspiration from road trips, Michael imagined a radio in a dashboard, then wondered what it might be playing. This led to a soundtrack for the spot he could edit to:
“I wanted to test something called Chirp. Chirp by Suno AI is a text-to-song service that make incredible jingles. You can put the lyrics in, and it'll sing them for you. Or you can let it write the lyrics for you. You can just say, 'There's a burger chain called Meat Heads. Make me a jingle about that.' Then you could pick punk rock, or classical or whatever. I was kind of going for that hip hop, R&B, Beastie Boys vibe, for that driving down the highway feel.”
After this, it became the familiar task of filling in the timeline, building the story as he went. Soon he had an almost complete spot. But wanted something special for the ending. The spot concludes with a visual button he wasn't happy with until his girlfriend laughed.
“I forget what I was looking for, but Midjourney gave me four images. it started as a hamburger crushing a guy. I was like, 'Oh, that's fantastic. That's hilarious.' And then I went backwards, into the inpainting tool in Midjourney, and I erased the burger from the person. But when you erase the burger from the person, it painted in the rest of the guy. From there, I was able to make those two come together.”
While clearly enjoying AI, Michael isn't immune to the frequently unspoken fears all creatives have about AI:
“I think it's kind of cool. And in a way, it's terrifying. Probably the biggest thing that I haven't figured out is, how are we going to make a living? You know, how is this going to get paid for? Who's going to do the work? Is the CEO going to do the work? Maybe. I mean, I get people all the time now telling me, the CEO wants to cut that. I've walked into edit sessions lately, and the executive creative director says, 'Here's my first cut. Can you make it better?' You know, can you fix it?”
It's an incredibly confusing time. As production tools get democratized, AI does more than we could have ever imagined, and everyone is broadening their offering, what does success look like for a modern creative?
There is not doubt it will broaden the idea of what is possible and help create many more combinations of things we could never have previously imagined.
“I think of Andy Warhol. He would just be blown away, What would he create? He would love this. It is totally something he'd be a part of. It's mixed media. You're just taking pieces from all these different places and creating art."
A few days ago, I spent way too many hours attempting to prove false a major pop-cultural myth.
People told me I was a simp. That I didn’t’ get it. That I was clueless.
That my story could be easily disproved by one of America’s pre-eminent art museums.
But they were wrong.
The myth-busting circulates around a letter from Mick Jagger to Andy Warhol.
In it, Mick is commissioning album cover art for, as the story goes, the legendary album Sticky Fingers. The album cover with the zipper.
People frequently cite this letter as the perfect creative brief. Or a paragon of creative communication.
Fans of this letter (I was one myself), frequently start their posts with something like this:
“I’ve always loved this letter. The legendary Mick Jagger writing to the legendary Andy Warhol about the now-legendary Sticky Fingers album cover from the Rolling Stones…"
But it’s total B.S.
Or, to put it more diplomatically, it’s one of those persistent and fascinating cases of misinformation hiding in plain sight.
The letter is not a request from Jagger to design Sticky Fingers.
I was a request to create cover art for Hot Rocks, the much less celebrated compilation LP.
Cue the sad descending horn music.
If you know just a little bit about the history of the Rolling Stones, five things quickly indicate this isn’t a letter about Sticky Fingers:
1. Mick writes this is for their “hits” album. Sticky Fingers is not a hits album. The album Hot Rocks, which was being compiled at the time, is the “hits” album.
2. Mick notes he has enclosed a copy of the record. Sticky Fingers was not completed until 2 years after this letter was written. Maybe they had a few rough tracks in the can but hardly enough to have pressed an album.
3. Mick states that Al Steckler will be following up. Al Steckler is the marketing executive at ABKCO put together the song list for Hot Rocks. One of the rare times anyone outside of the Stones did this.
4. In the final paragraph Mick indicates Mr. Steckler is in a hurry. He’s not rushing for a partially completed album that won’t arrive in the stores for 2 years, but for a very real hits compilation that he is deeply motivated to get out the door.
5. At this time, the Stones were separating from Allan Klein and ABKCO. After performing his Hot Rocks duties, Steckler was no longer had anything to do with the Stones or the marketing of Sticky Fingers.
So.
Who the fuck cares?
It’s a fair question.
I care. And I don’t know exactly why.
The first place my mind goes to is Brandolini’s law, where calling out bullshit requires more work than what went into the original BS.
We all have a responsibility to call out things that are wrong, or it just gets worse.
The 2,000 or so people who liked Sir John Hegarty’s post from yesterday are now using this as a glowing example of a great brief.
Someone else is penning another love letter to the letter.
And in the Warhol Museum in Pittsburgh, I am told this letter sits paired with the album cover from Sticky Fingers.
But I think there’s something else.
I spend a lot of time thinking about AI and worrying about all the false and misleading realities it might create.
Then, at the same time, I look around and realize we shouldn’t take our eyes off the truths that are very much in our control.
We should always keep questioning things. Anything.
We shouldn’t get lazy just because today’s world is so f’ing overwhelming.
Sponser
AI changes everything. Including storytelling. Nextness Volume 11 is brought to you by Storymachine. As a leader in the AI video space, Storymachine scripts, films and delivers everything from branded content and commercials to corporate masterclasses and training films. If you are looking to unleash a new kind of storytelling, Storymachine just might be your jam. storymachinefilms.com
