Today my conversation is with David Smith. He’s the CEO and Founder of Croquet Studios.
David is a computer scientist and entrepreneur who has focused on interactive 3D and using 3D as a basis for new user environments and entertainment for over thirty years. His specialty is system design and advanced user interfaces. He is a pioneer in 3D graphics, robotics, telepresence, artificial intelligence (AI) and augmented reality (AR). He creates world-class teams and ships impossible products.
In 1987, Smith created The Colony, the very first real time 3D adventure game/shooter and the precursor to today’s first-person shooters. The game was developed for the Apple Macintosh and won the “Best Adventure Game of the Year” award from MacWorld Magazine.
In 1990, Smith founded Virtus Corporation and developed Virtus Walkthrough, the first real-time 3D design application for personal computers. Virtus Walkthrough won the very first MacWorld/MacUser Breakthrough Product of the Year.
David was Chief Innovation Officer at Lockheed Martin and a Senior Fellow at Lockheed Martin MST, focused on next-generation, human centric computing, and collaboration platforms. Here, he developed several key technologies and won the Lockheed Martin TLS Inventor of the Year for the last four years (every year he has been eligible).
What’s interesting is that he worked closely with authors Tom Clancy (Rainbow Six, Hunt for Red October) and Michael Crichton (Andromeda Strain, Jurassic Park) to develop games.
But that’s only the beginning….
David believes that the year 1968 was the most critical year in computer science. In this one year, three key individuals launched what he considers, and what he’s continuing to build upon, is this goal of enhancing humans’ ability to solve hard problems using computers to think in a different way. Again, enhancing humans’ ability to solve hard problems using computers to think in a different way.
He’s building upon the work of the pioneers in the internet: Doug Engelbart, Alan Kay, and Ivan Sutherland’s work – all focused on working with the Xerox Alto Project from a long time ago – close to 50 years ago. Some of these breakthroughs – that even amazed Steve Jobs, as you can see on some of his YouTube videos from years ago when he was stunned as he looked at the Xerox Alto project. At that time, what really stuck out for Steve Jobs was the gooey interface. This was really that first interface between a computer and a human.
So, David’s passion is to continue to use his skills, competencies, and capabilities in 3D engineering and design. His goal is to develop these applications, systems, and platforms that are really going to transform how we use computers and solve big problems in the coming years. He’s exploring the use of 3D and graphical situations that we can’t even imagine right now, using computers to solve interesting challenges and complex problems moving forward.
Major Take-Aways from This Episode:
- What is an Augmented Conversation?
- The future of turning a computer into a vehicle to exchange ideas in real-time and sophisticated areas.
- How a fusion of ideas reinvents and redefines the vision of what computing is.
David Smith: 00:00:00 This kind of conversation you and I are having right now, where we share ideas, we try to paint the picture of what's possible but what I'm after is, what if I could have a discussion with you and I'm describing a simulation, the computer actually generates that simulation as part of the conversation and then you can reach into that and modify it as the conversation proceeds. What I'm after, this idea of augment conversation is how do we make that possible? How do we make that real? We built over the years, the kind of technologies that support it, and the new version I built is by far the best, but that the goal is when we think about what AR is and what computers really are about, as Alan recognized, they're a medium and a medium that when you think about that, what's a medium for, is to exchange ideas, exchange information and the really interesting thing to me is he knew that in 1968, obviously, Doug Engelbart knew that in 1968, if you look at his demonstration from that year.
That was one of the things that got lost in translation over the years and so Alan and I've been trying to reintroduce it as a foundational element of what it means to do computing basically. Let's go back to the beginning and try to make computers a vehicle of exchanging ideas in real time and sophisticated ideas so that's the area that I am in. That's what I focus on and that's kind of the area I'd like to talk about because I know more about it than just about anybody, but the other thing is, I think it's extraordinarily important because you're defined more by how you communicate than anything and we're reinventing, redefining the nature of communication so that you're now able to communicate, explore very, very complex ideas with other people.
Bill Murphy: 00:02:10 I'd love to explore that with you and I know we've jumped right into it, which is great and I just want to.
David Smith: 00:02:17 You asked me, right, I want to talk about, that's [inaudible 00:02:19]
Bill Murphy: 00:02:19 No, that's great and I just want to welcome you to the show today, David, and really look forward to talking to you about your expertise here because I think a lot of people could learn quite a bit.
David Smith: 00:02:32 Yeah, glad to be here.
Bill Murphy: 00:02:34 Well, let's take us a little bit into the past so that we can then have some context for where we are today and where your current project has come. You mentioned, in some of my research, 1968 was a real quite a turning point with three individuals and I would say I was just reading a book, just recently released called Loonshots, and I just had Safi Bahcall on the show and I didn't realize Alan, and through his storytelling in his book, Steve Jobs had gone out to, I think park, the research lab and met with Alan and seen some of the early graphical user interfaces, but maybe you could start with why you thought 68 was such an important year and a three gentlemen that make up that year.
David Smith: 00:03:25 Yeah, I don't think it's understatement to say that was the most important year in computer science. It was a culmination of three big ideas that were just demonstrated for the very first time. The first one, and probably the most important was Doug Engelbart. He demonstrated to the world, kind of a new vision of what computing was. It's referred to as the mother of all demos today. It was 1968, 50 years old and a little over, it was December of 68.
He was doing a demonstration in a center in San Francisco, but he had a microwave connection to Stanford SRI, it used to be Stanford Research Institute, and what he demonstrated was something that was extraordinarily profound. It was so profound, it was actually missed by most people because what they saw were the elements that he used to demonstrate this idea. The elements were, he invented the mouse so it was the first time anybody had really seen a mouse working.
He invented this hypertext capability to click links and he invented all these wonderful capabilities, but they're all in support of this idea of the computer as a means to collaborate so one of the parts and to me, the most important part of that demonstration he had was he had this microwave connection with another person and between SRI and between Menlo Park and San Francisco. He connected to that person. He opened up a little window. It was actually an overlapping window, maybe we're on the first overlapping window in history, and he started having a conversation with this guy who was far away. Now this is 1968. I mean we do this all the time, we do Skype, but this is 1968, but the difference was kind of profound, was that that remote user was able to interact with the same world that Doug was using, that same computer, they were sharing that interaction, both of them were fully enabled. Both of them could basically interact with the computer as if it was a single shared experience.
What he was looking at and what he's after was this way of how do you enhance human's ability to understand, how do you enhance human's ability to communicate and collaborate and solve very, very, very hard problems, and so all these technologies were brought out to demonstrate how we can use a computer to think in a different way and work in a different way. That idea got lost because people saw, oh I see a mouse, oh that's really cool, oh I see hypertext, I see kind of cut, copy, paste, I see video conferencing. They saw all the elements of it, but they missed the big idea, which was computers reinventing the nature of how we work so that was a linchpin of that year and in a sense, all of modern computer systems have a legacy to that and the problem is they took all the pieces like the mouse, but they left the main points on the cutting room floor. That was one.
The second one was equally important and it was dramatically influenced by it by Doug, which is Alan Kay. Alan invented a thing that he called the Dynabook, and he had an opportunity to go to University of Illinois and saw like a 16 by 16 pixel LCD display for the first time, and he realized that someday, that's going to be a large display. He'll be able to have it as a tablet size and so his imagining was someday, you're going to carry these things around with you, he called them Dynabooks and he imagined like that they in turn were not just, they're personal computers. In fact, which is really funny, in 1972, he coined the term personal computer in a paper he wrote, a personal computer for children of all ages and that was an earth shattering paper. It was based on his idea from 1968 of the Dynabook.
In that paper, by the way, there's a picture of two children working together. They're playing a game called Space War, and they're programming it, but if you look at that picture closely and there's a drawing that Alan did, you see that both those children have exactly the same screen. They are collaborating. They're working together to write this program and then play the game so that was a foundational thing. Of course, the Dynabook was a centerpiece of what became the system at Xerox PARC. The computer there that is called the Alto, they actually refer to it as the interim Dynabook.
The goal was, let's build that Dynabook vision that Alan had and what are the things that are necessary for it? Again, it was the vision driving the platform. It's like Alan had this idea of a Dynabook, what are the pieces that have to be true about that? Smalltalk was invented to support that capability.
Obviously, the idea that Doug Engelbart had of using the mouse to interact with, that was like, oh we got to take that idea and so in a real sense, Doug, kind of fueled Alan's work and that was just a wonderful fusion of ideas. There was a problem though that occurred, which was, the Alto had this wonderful thing that it was, the Ethernet was invented as part of this system and so you had hundreds of Altos connected to each other at Xerox, but the problem was, they weren't used in the same way that Doug had imagined it as a sort of a single supercomputer, everybody's collaborating. It was sort of loosely bound. In other words, you could send an email to somebody or share a file, but you really weren't collaborating in the same sense that Doug had demonstrated earlier. I'll come back to that but the third thing that happened that year, was done by a guy named Ivan Sutherland. If you've heard of a company called Evans and Sutherland. He's the Sutherland of that. The other, Evans is actually Dave Evans, who is Alan Kay's advisor for his Ph.D.
Ivan Sutherland is known for a number of things. One was he invented interactive graphics with a system called Sketchpad, which was in my mind, the beginning of this stuff where what you're looking at is sort of a conversational system between the human and the computer. That was 1962, and so the whole idea of interactive computer graphics stemmed from that but he wanted to go further. He was thinking, what does it mean when, not only are you just using your hands and your eyes to interact, but let's just say I want to be inside of that computer generated world, so he built the first real head mount, what we consider VR today and he had to develop certain kinds of software and hardware to be able to do that, but he actually had a working VR system in 1968, where you could look around and you could interact. It was astonishing.
Bill Murphy: 00:11:48 Yeah, I just saw it on YouTube, when I was watching some of your TED Talk. It was pretty cool. I mean it's actually not that much different than today's version in the sense that it doesn't look that awkward in comparison.
David Smith: 00:12:01 No, actually, it was significantly more awkward, but it worked and it was pointing the direction of where are we going now. I think it's really important to understand that what these guys did was, I mean it started with a blank piece of paper. You know the idea of a head mount like that didn't exist until Ivan built it. The idea of a collaborative platform didn't exist until Engelbart imagined it. He actually wrote a paper in 1962, which was the foundation of all that work and the goal of his project was to make the vision of that paper come true, but they set a vector. It started a blank page and they said, this is the way we see computing going. This is how we imagine computing, the importance of computing and again, I go back to the idea of it being a medium that enables you to communicate both with another person but also with the computer. That computer is a fully enabled participant and we kind of got there in a lot of ways.
You think about going back to the Alto and the Alto took many, many of the ideas that Doug Engelbart had and infused them. Of course, the interactive graphics part came from Ivan and many other people contributed but those are like the gods of that, and so that they resulted in this amazing machine called the Xerox Alto and that was easily the most influential computer in history outside of the Dynabook.
The Dynabook was a requirement for the Alto but the Alto then spawned two things. There have been rumors at Silicon Valley that there was this magic computer at Xerox so Steve Jobs was able to arrange a demonstration of that for him and his team. The way they did that by the way is Xerox wanted to invest into Apple and Steve said they'd only allow the investment if they gave him that demo so that's how he was able to get into the room.
Some of the people at Xerox were really, really unhappy because they said he's going to steal our ideas and of course, he was. Of course, he was, and he did so he got in there and he got a demo firsthand of how the Alto could work. Dan Ingalls actually did that demonstration. Dan's the guy who created Smalltalk. Alan invented the idea of Smalltalk, but Dan implemented it and really defined how it worked, but he was the one giving the demo to Steve Jobs. We're showing them all these wonderful things, and we're showing him all these wonderful things I guess there's one point where he's showing, here's text and we're scrolling and Steve Jobs says, oh that's wrong. In those days, programming, you kind of scroll line by line and Steve Jobs said, oh that's wrong, it has to be continuous.
Now Smalltalk's different from other systems. Smalltalk is actually a system, not just a language and so the full development system was right there so Dan Ingalls jumped into the code and there's a line in there where basically, he was making the text scroll line by line, and he removed a little indent function in there so that now it was moving smoothly and he did that within like 30 seconds while Steve Jobs is watching, and he shows it to Steve and now it's moving smoothly. It's really interesting that that one act, probably the most powerful thing that Dan showed Steve and Steve missed it.
Bill Murphy: 00:16:08 Oh really?
David Smith: 00:16:10 I mean it was like one of those invisible things. It's so powerful, it's so amazing and it just went over his head so that's why he talked later about the things he saw there. He said they showed me three things, they showed me this networking capability. I didn't see that. It's like it was one of those magical things but it was so good, it was invisible. He showed him an object oriented programming and he says he didn't say that. He said the third thing they showed me was this graphical user interface and he said it was the most magical, amazing thing he'd ever seen and it made everything else disappear, but of course and actually, Alan Kay said, well, they showed him way more than three things. He missed a lot of other important key ideas, but what he and his team walked away with was that graphical user interface without all the really important stuff underneath it like networking, like object oriented programming, like live programming, which where you can modify the system dynamically.
He walked away with that and he had this idea of how computers should work and he was mostly right so that became the foundation of how the Macintosh would be built. They saw overlapping windows. Alan invented overlapping windows in the shower one day because the Alto screens were too small. Pop-up menus, Dan invented that. The idea of scrolling text like that was based on a technology called BitBlt that Dan had invented, and
so all these things, they came away with but they didn't have the code, they didn't know how to do it. They just knew what the action, how it should respond and lucky for Steve Jobs, Bill Atkinson was with him at the time and one of the most talented programmers in history, and he was able to not just reconstruct a lot of what he saw, now this is the visible space.
It wasn't how it actually worked underneath, but he made all that work on top, but he actually invented some things that he had thought he seen that he hadn't seen, which is really more remarkable because they actually thought they saw some technologies that were implied but not ever actually working. The Macintosh was an extraordinary computer, but it was like the veneer of what they saw at Xerox PARC and of course, the Windows was sort of like had the advantage of, seeing both what Xerox was doing and what Apple was doing and so Windows became kind of the clone of a clone, so to speak. Again, since Microsoft didn't have any deeper understanding that Steve Jobs did about where it was going, really going on, it became just sort of a user interface engine as opposed to a thinking compute engine, which is what the Alto was.
That's where we kind of catch up to the stuff I've been working on, which is, how do we get back to that computer as a communication platform, as a collaboration platform, as a thinking platform and so Alan and I started talking, I met him 1990. I wrote the first real time 3D adventure computer game called The Colony, which is real time 3D shooter adventure. Then I had started a company called Virtus, where we did a product called Virtus Walkthrough, which is the first real time 3D design tool for PCs. We actually used an earlier version on the movie, The Abyss, where I was able to recreate the entire set of the underwater set so that they could actually start doing camera angles and things.
Bill Murphy: 00:20:18 I was wondering where that entertainment link came in with Tom Clancy and Michael Crichton and that's where that started?
David Smith: 00:20:26 Sort of, I mean what happened with Clancy, my game was reviewed quite highly. I won adventure game of the year from Macworld and Tom, he was a Mac user, and he saw that review and he decided he wanted to try. He wasn't really that much of a game player, but he decided he'd play the game and it's one of those games that once you get into it, it's obsessive. You just have to keep going. He got in touch with my publisher and said, hey, I want to talk to this guy and so my publisher says, hey, Tom Clancy wants to talk to you, here's his number. At the time by the way, I was reading Clear and Present Danger, which is one of his books so it's kind of a weird, I'm literally calling up like the most famous author in the world and I'm reading his book. We had a wonderful conversation and we just sort of started, he would call me every other day but he never asked for hints.
It was always to tell me what a terrible person I was because the game was so hard and so he goes through it and it's like, I got to this point, why were you such an a-hole, you did this to me. I was like, yeah, well, that's a game and so he finally got done and he was like, it was way too hard a game, but he finally gets done. It was like this big achievement and then he was so excited and he said, I want to do something with you, this is just awesome so I was working after the project with Cameron on The Abyss.
I was going to start this company to build tools for 3D design and so that's what Walkthrough was, and Tom became my first outside investor and my first outside board member, which was really, really cool and he was just the nicest guy. For me, he was a really, really close friend and a good friend. What happened then was he and I are talking about what's next and I said, let's do a game.
We came up with this idea called SSN, basically attack submarine simulator and he brought in Doug Littlejohns who was a captain of a Royal Navy of a attack nuclear submarine and we developed this incredibly cool simulator, a sub simulator. What was really funny is we took the game design and it got turned into a paperback book that called SSN, oddly enough, and that would became a like a number one New York Times bestseller book.
It was just the game design that had been turned into prose. That's why I realized Tom Clancy wasn't a person, he was a brand like McDonald's but then what happened was Tom introduced me to the hostage rescue team at Quantico, Virginia and because they actually were interested in using our software because if they're doing plans for hostage rescue, they need tools for understanding what they see, what bad people see and so I was spending time with them doing that.
One time I went there and they showed part of their training, they have these concrete towns that they use, and there were these observation towers. I was like on one of these observation towers and there's little, kind of micro town in front of you, and income these black helicopters and that's pretty loud and you're like up near them and seeing that, and then these guys in total black like ninjas start rappelling down from the helicopters onto the rooftops and then they start running into the buildings and you see these explosions because they're throwing these grenades into the rooms and it's like, this is better than movie, imagine this, if it was Apocalypse Now but it was full surround, right, I mean it was insane.
I was like, I was just so impressed by the complexity and the pyrotechnics and everything, it's like that's a game, that's a game so we called Tom. It was Frank Roseman and I were there, we called Tom after that and said, hey, we got to do a game with this. This is insane. He said, well, you guys do a game. I'll write a book and so that's where Rainbow Six came from.
Bill Murphy: 00:25:29 That's awesome.
David Smith: 00:25:30 Yup, and so what happened there was Brian Upton, I had brought in, we decided to start Red Storm Entertainment and do this game. SSN was a Virtus project but we decided to spin out and so we created Red Storm Entertainment and then one of the first big games on that was Rainbow Six. Brian Upton, I brought in to be the VP of engineering but he decided that he wanted to be the game designer and he is an amazing game designer, it turned out. I mean he'd been doing game design in this tabletop for years before and I didn't even know about that, but he came up with the ideas for Rainbow Six and really defined a new genre of gameplay. He later did Ghost Recon, which was the follow on and then he spent 15 years at Sony as sort of chief game designer there. He actually joined me on my new project so it's really kind of wonderful because I consider him one of the best game designers on the planet.
I'm not doing games, but we're building a game to demonstrate the new stuff so it's kind of fun to close that loop but anyway, that's what happened with Tom and Rainbow Six was actually using a version of what I called Croquet today. It wasn't quite anything like what we have today, but it was in that direction.
Speaking of that, the very first version of the system I'm working on now was done in 1994, it's a collaboration with Alan. I met Alan Kay. Alan Kay is the person I'm talking about earlier, in 1990. His wife actually introduced us and his wife is very interesting. She's Bonnie MacBird. You might be aware of her as the person who wrote the movie Tron.
Bill Murphy: 00:27:40 [inaudible 00:27:40]
David Smith: 00:27:41 Yeah, and actually, the main character was based on Alan and hence is the character's name in the movie is Alan, because it was like this vision of what the future would be like and Alan is extraordinarily influential in that so Bonnie introduced me to Alan in 1990 and we we're having conversations. I meet with him several times a year, but then we said, let's do something.
I had this idea of a collaborative 3D world because we talked about what's the next interface? What's the next UI? We kind of decided, well, obviously, we want to bring back the collaborative stuff that was originally in the Dynabook and that Doug Engelbart talked about, but we also said 3D is going to be the centerpiece of this because someday, you're going to be wearing your computer and so I built a system called ICE, Interactive Collaboration Environment, that illustrated those ideas. It was multi user. You could drag and drop a program into it and it was automatically shared, drag and drop a document, picture, anything and it was a prototype, pretty rough, but it really illustrated the ideas really nicely.
It's sort of a proof of concept if you were, if you will. One of the things I'd like to mention, after I did the project, but Tom, Michael Crichton got in touch with me and said, hey, I want to do what you did with Tom, basically and he had this idea for a game based on a book he was writing called Timeline. We started a game company called Timeline Computer Entertainment. I thought it would be fun to work with Michael, and I did that for two years. I ran his company, his game company for two years and it was an interesting, very different experience. With Tom, Tom just said I do books, I do stories. He wasn't involved in movies. He wasn't involved in anything else.
Michael was different. Michael was very aware of different media so he understood how to write bestsellers. Everything he wrote was basically a bestseller. He had directed Westworld. He created and wrote Twister, which at that time was Warner Bros.' biggest movie ever, and he created and wrote most of the first season of ER, that big television show.
Bill Murphy: 00:27:41 Oh really?
David Smith: 00:30:27 He really understood different media and he wanted to take a shot at games, so that in a sense, Timeline was sort of an experiment to see if his ideas would translate into a gaming space and at the end, it was unsuccessful. It was worth a try but the problem was every single medium that he was in, he controlled what the user saw and he wanted to impose that same model on to games and games is, oddly enough, is very different. Games is all about failure. Without failure, there's no game, and so taking out that failure mode kind of turns it into a ride and certainly that's more in keeping with the way Michael thought, but it didn't make a good game but it was an interesting experience to try something like that and we learned a lot but one of the things is really defining the nature of what a game is.
I think that was a good learning experience I'd like to say, but then Alan said, hey, we need to collaborate so I actually started working directly with Alan for, I've been working kind of indirectly or directly with him for since 2000 and that's where we started the Croquet project, which is the foundation of the system.
Bill Murphy: 00:31:56 Quick question for you related to the Croquet, maybe to lead into Croquet more deeply. Enhancing the ability for humans to solve problems, to solve big problems and this interaction and collaborative experience that the original kind of foundation thesis or vision for the three folks that were involved, especially with Alan early on, is that, that's what you're currently trying to solve right now. Is that what you're solving for now?
David Smith: 00:32:34 Yeah, well, you know what, we want to do that but to do that, we need some of the core pieces. When I talked about some of the things from Doug Engelbart's lab that were left on the cutting room floor at Xerox, to me, one of the most important parts was the ability to do direct collaboration, yeah, person to person sharing the computer or sharing a simulation or whatever. That got lost, and without that kind of key element, that ability to share, the rest of it's not going to ever happen.
We decided to focus on just that piece to enable any users to share a complex simulation, interact with that simulation totally seamlessly so today, what I'm building is not the whole system, although we'd like to do that someday but an enabling piece of it that allows any users to engage with, they could write their own application on top of it, but it makes their application multi user and collaborative so that multiple people can interact with it any way they want to and use it as a basis of conversation.
My feeling is, every application's ultimately going to have to be collaborative and when you think about augmented reality, for example, it's crazy if it isn't augmented, if it isn't multi user, because think about you're wearing this thing on your head, and it will be like a regular pair of glasses in the next few years and it's, if I'm talking to you, computer's a full participant in the conversation, and is fully able to generate a simulation between us as we talk, and then we're both able to interact with it dynamically. Well, we don't really have a foundational capability to achieve that. We've got some ad hoc methods that were developed for multi user gaming, but we don't have a real model for how collaboration could work, and that's the thing that Alan and I've been working on since 2000 was how do we create a sort of a platform that enables applications to be shared in just that way.
That's what Croquet is. It's that foundational element that enables people to write an application so that it's perfectly shared in just the way I'm describing. Ultimately we see it, you're going to start building more and more infrastructure on top of that so think of it, what we're building is sort of a kernel, a missing piece of the operating system kernel that enables collaboration.
Bill Murphy: 00:35:35 You're thinking like this will be like the OS in which apps will be written on top of the OS or like IP from a networking perspective.
David Smith: 00:35:42 Yeah, actually-
Bill Murphy: 00:35:43 Then TCP would be like the applications that would give value to the IP.
David Smith: 00:35:49 TCP/IP is a perfect analogy, and surprisingly, more than you think. David Reed was my collaborator on developing this Croquet platform, as was Andreas Raab, who sadly passed away but David, his thesis was this idea of kind of collaborative replicated computation but David actually was the guide. Alan calls him the slash in TCP/IP. He's the guy who created TCP running on top of IP instead of a separate protocol. He also was the inventor of what's called the end to end argument, which is the foundation for how the internet works, so he was my collaborator and so you're kind of, the similarity between TCP/IP and what we're doing, we call this replicated piece tee time, by the way.
I sort of think of it as the TCP/IP for collaboration. I mean, that's sort of grandiose but we certainly have, between Alan and David Reed and even myself, I think we've got something that's unique enough and important enough to be able to describe it to some degree in that way. Obviously, it's not a public protocol, but it is a thing that if you write your application assuming TCP/IP, then you've got a web-based application ability to share information between you and some other point on the internet, and while we're doing is, in a sense building tee time on top of TCP/IP so that now it's a replicated state engine, so whatever I see and whatever you see is running bit identical and then when I interact with it, that interaction is automatically shared seamlessly so that the simulation still runs a bit identical.
That concept is not far from the truth, I think. I mean, it's overhyping perhaps, but on the other hand, it's like, that's what it does. It guarantees just like TCP/IP guarantees a message is going to get to the other end, we guarantee that that simulation is going to be bit identical, wherever it's running, any number of places. It could be 100 different users
interacting, they are all going to have exactly the same simulation. It's going to evolve in exactly the same way and any actions from any of the users that impact it are going to, those actions are going to be also perfectly replicated seamlessly so that the simulation you see is going to be sustained even when multiple people are interacting with it.
Bill Murphy: 00:38:41 What's an example of a simulation just so that my listeners can understand?
David Smith: 00:38:46 Yeah, I use, I don't know, I was at Lockheed Martin for five years as a senior fellow and this one, I always think about this one as imagine a virtual wing in a virtual wind tunnel and you can see all of the vectors going around that virtual wing, and so let's just say that wing's floating between you and me and then we modify that wing and maybe modify what's called a Reynolds number or some other parts of that and we actually see how that modifies the characteristics of the wing. We create some vortexes here. We create some changes in that simulation, so we're actually having a conversation in real time. We're modifying the physical characteristics of that system and then the simulation itself changes to reflect what we modified, but we're doing it as a conversational way so we're sort of exploring the what ifs of a complex system, sort of like the magical thing about spreadsheets wasn't that it could just keep track of your payments. It was what if I do this to my organization and that's what we're talking about here is a shared simulation is an exploration of what ifs, but it's live real time, and so you're able to interact with it and see the end result and explore that together.
That's why I call it augmented conversation, where you and I are talking, we have this really interesting thing between us and the computer is a full participant in that and that it takes what we think and what we do and what we want, and converts it into an extension of that simulation as well as maintaining that simulation for us.
That's the ideal, the idea and what's really cool is the platform works. It was one of those things in 2000, we call said we want to do this, we don't know how to. David Reed had an idea of how to do it but that was his thesis.
It had never been implemented so we actually spent a few years trying to figure out how to make this work, and then at Lockheed, I did another version that kind of was a very different model and then this one, which is I think my company is Croquet Studios and what we're working on is that system. How do we ensure that everybody has a shared simulation, shared capability, and I know that someday, every application has to work that way.
This idea of having single user applications is a very outdated concept and it's just the limitation of the underlying infrastructure that keeps us from doing it. It's not that you can't do it, by the way. It's just the complexity of doing it is so high and the cost of doing it is so high. Nobody really does it, but I know that once you do have true collaborative applications where it's perfectly shared, you're guaranteed that what I see is what you see, then I think that it's going to be a requirement.
By the way that perfectly application's a centerpiece of this because when you're conversing with somebody, if what you see and what I see is even slightly different, we don't trust that communication channel anymore so it's essential that what we're sharing is a bit identical. It's like I know that what you're seeing is what I'm seeing. I know that what you're doing is what I see I know that what I do, you see.
Bill Murphy: 00:42:41 When you, just for a simple example, using like an Excel sheet with 1,000, 1 million or 10 million permutations built into it, right now you're obviously thinking well beyond like sharing that out via a WebEx conference where you can share and you can basically have that sheet up there and enter inputs into it and such. You're actually thinking of it from a 3D model, you're in a virtual, a full virtual world or an augmented world, correct, where you can basically manipulate that object real time in a virtual space.
David Smith: 00:43:21 Yeah, actually, that's sort of a side, it's example of how we see the interfaces working but in particular, you have another person there, and you're using the spreadsheet as a kind of exploration so one of the demos I did is this 3D spreadsheet and you select a bunch of cells, but one of the cool things you could do is extrude those cells as if it were a bar chart in 3D space so the other person can actually see what you're doing and you're talking to them, you're having this conversation. Look at how this information, people can't look at numbers and understand them but as soon as you see that extrusion, all of a sudden you have that relative view of what's going on and its mind blowing.
Alan has this phrase point of view is worth 80 IQ points and what you're doing is giving the users a different perspective, different point of view on exactly the same data you see flat on your computer screen but now, you're seeing it in a very different way and the other thing is you're sharing that with another person so that they've got, they're bringing their own ideas and perceptions on it and they'll say, oh, that's really cool. Let's look at this and so you're sort of handing off the chalk, so to speak, although you don't need to. I mean, it's like, then he's going to say, what about this area, or what happens, spreadsheets are simulation engines, by the way, so one of the things is you enter a value into it to change some particular field, but there's many dependencies out throughout it so you can see, let's just change this value as a seed and it computes a new, completely new landscape of information on that spreadsheet.
That's what we're after is that ability to have an exploration space like that. Of course, the spreadsheet could then be hooked up to something like a virtual factory that's basically operating based upon the parameters or defined in that spreadsheet and then that factory itself could be imposed on an actual factory with a communication of internet of things so that now you've got, not only are you defining the virtual factory and doing a what if, what if we did this, how's productivity affected, but then you could turn a switch and say, okay, do that so that new productivity model's imposed on the factory already exist.
Bill Murphy: 00:45:48 I think that this concept of plus-ing the IQ by 80 points is very real. There's some research that I've shown in my Innovation Group, that our human IQ hasn't scaled over the past 200 years to keep pace with our technologies, since Moore's Law is now taking a grip.
David Smith: 00:46:09 That actually doesn't matter that much. Alan makes a really good point. He says Da Vinci invented a lot of really powerful ideas, devices like helicopters and things like that, that could never be created but he's like the smartest guy on the planet. Henry Ford wasn't particularly, I mean, he's smart, but he wasn't a Da Vinci but he was able to take these same ideas and create a factory creating Model T’s.
What's the difference between the two? It wasn't IQ. I mean, Henry Ford had less IQ probably but it was having access to the infrastructure that the world provided and being able to leverage that.
It's not all about IQ necessarily. It's about sort of the opportunities that are around you that enhance your IQ. I think Alan talks about that as point of view. Henry Ford had a very different point of view than then Da Vinci had, and was able to leverage that in a way that defense he never had the opportunity to so I think that sort of what we think about IQ is, yeah, that's an important factor but then the multiplicative factors of, well, internal combustion engine exists so now that you kind of have that different perspective, and you're able to put powerful ideas together in a way that you couldn't, no one could have ever done before and that's where it comes sort of the amplification is sort of taking the idea spaces that surround us and seeing those from a different perspective and how they can be combined in a way that you would have missed otherwise.
Bill Murphy: 00:47:59 Yeah, I think that you wrote a paper just in December on, I think one of the takeaways I got from it was the augmented human is a better version of you.
David Smith: 00:48:08 Yeah
Bill Murphy: 00:48:09 I think that you have a quote man as much more than a tool builder as an inventor of universes, which-
David Smith: 00:48:16 That was Alan from that paper I talked about, Personal Computer for Children of all Ages.
Bill Murphy: 00:48:22 If you can interact in this virtual space, essentially you can create new capabilities, and as you mentioned, then be able to send those to the manufacturing capabilities through 3D printing or whatever it may be sort of gives you an exponential ability to create.
David Smith: 00:48:39 Yeah, well, and think. I mean, like I said if nothing else, one of the things that we have too much of maybe is ideas, but we don't really have capabilities, methodologies for exploring those ideas easily or putting them together easily.
I should back up a little bit and I mentioned, one of the things that happened when Steve Jobs saw the Alto is, as I mentioned, Dan Ingalls actually did some programming right in front of him, and he missed that. Fast forward to now, we don't really have live systems like that that enable you to as part of the conversation, define algorithms, define the world and I sort of think of it as the reason Smalltalk and the Alto were such powerful combinations is you could write. Smalltalk was written in itself. Smalltalk wasn't just a language. It was a system and so when Dan Ingalls made pop-up menus, he wrote it right there and extended the system.
When he needed to change the scrolling for Steve Jobs, he just did it. What we do today is we kind of develop operating systems in sort of a bottle. We don't use, we don't think of it as being a live system, we think of it as a very static system that enables a user to do something but in fact, what has to happen, because we're exploring a whole new world with AR and VR, is that those platforms have to be more malleable. We have to, yeah, I think of it as somebody trying to build an operating system for VR or AR today, it's like trying to build a ship in a bottle because you don't really know what a real ship is like, because no one's ever been in VR AR as a full OS but if we switched it around and said, okay, you have to build the operating system from within AR and VR, then you make very, very different design decisions because you're basing those decisions on the reality of the world you're in.
Some things work great, some things don't and so you have to modify the system within which you are working to be able to encompass that. I think of it as like the best thing to do if you're in VR AR, let's launch a raft but the raft is designed so that we can add and modify it so that we can build it into an aircraft carrier, versus trying to build an aircraft carrier in a bottle and launching it, and of course, the thing is going to sink because no one's ever done that.
I think that's really a kind of a conceptual model, a change that has to occur. We're enabling a piece of that because we want to enable multiple people to collaborate inside these idea spaces but the other part is enabling people to kind of modify the world that they're in all the way down to the foundation, and that's where the real win comes and when you think about bootstrapping, building a system from within itself to improve that system is a key idea and really, that's kind of, I keep coming back to Doug Engelbart because none of this is new. It was all his vision of how the world should work and it got lost to a certain degree for various reasons. One is just like Da Vinci, he had this idea of how things should work but he didn't have a way of making them work the way he thought.
We do now, we have the internal combustion engine that we didn't have before. We have the internet, among other things. We also have very fast computers. We also have 3D graphics. We have things that he couldn't even imagine, perhaps actually did imagine all this stuff. I should give them a lot more credit than that but now, we can build it. Now we can do that that thing that got lost because it was just too hard and that's what we're doing.
I'm just sort of in a sense filling in some of the pieces that I'm adding perhaps an internal combustion engine to the flow of what has to happen but it's essentially, you can't make a car without it, and we know that this is the way everybody's going to work someday.
Everybody's going to have this virtual car and everybody's going to work together seamlessly. It's going to be, as I said, this augmented conversation. Ultimately, every conversation you have is augmented. You're going to be able to interact with anyone at any time in any way, and it's necessary. It's a kind of, I think, well, it's inevitable for one thing, but the other thing is you're defined more by how you communicate than anything. The way you think, the way you express them is, Minsky said, Marvin Minsky said, you don't really understand something until you understand it more than one way, which is why teaching's so important because when you first teach something, you're forced to understand something from a different perspective and that's kind of where we're at.
We're in this idea of, like, I call the augmented conversation. It's about communication and about understanding things in multiple ways and you are defined more by how you communicate than anything so really, what we're doing, whether we like it or not, is we're redefining what we are. We're redefining what it means to be human because we're redefining the power of communication, and it's implicit in everything we do and every way we think. We don't notice it. It becomes invisible.
Marshall McLuhan talked about how the electronic media has extended our central nervous system to encompass the world. That was actually the very, I think the introduction to understanding media and it's so true because he got this crazy person who does a tweet and it affects 100 million people emotionally within a few milliseconds. That's the world we live in. We have parts of our consciousness, parts of us, our central nervous system exists on every place that the internet touches, and in some way, that vibration of what's occurring there is going to affect us ultimately, directly but we are that system.
The idea of being able, and I almost say we're victims of that system, because we're not really involved in a high level of communication. We're just at a primitive level of interaction with that kind of super shared nervous system. What we need to do is have a higher order control, we have to be able to think as well as sense and that's kind of the real goal of all this.
Bill Murphy: 00:56:22 I love this, David. I think it's a fascinating history of your trajectories from, and your partnership with Alan and how you're bringing these technologies into the current age, because I think that right now, we need wisdom as much as smarts to bring these technologies into our world. Is there anything in particular as we wrap up that that you want to summarize what we've talked about or just the general thesis that you want people to know about and I'll put this on the show notes as well as places they can go to check out your work but in summary, what do you want to say?
David Smith: 00:57:10 I think we're at a crossroads, actually. We're dealing with a world that where, you look at social networking, that sort of thing. Those are systems that are trying to create models of who you are, and understand you for two reasons. One is to be able to provide you with things that they think you want and the other is to perhaps manipulate you to want the things that somebody else wants you to have.
One way I think about that is they're trying to create a model of who you are, and as time goes on, it's going to become a much better model of that. The other way to look at it is what if we had an AI whose job was to enhance what you're doing, the things that you
want to do and the things that you want that you care about. Instead of being controlled by a third party, it's controlled by you and enhances you so I look at this world that we're moving into.
There's kind of a weird dichotomy where the AIs that seek to control you are owned by the big social companies, and the AIs that are designed to help you don't exist yet. We've got a real challenge that we have to start thinking in terms of what I call prophylactic Ais, AIs whose job is to extend and enhance you in the way that you want to be extended and enhanced.
Part of that, it all comes together is it's the nature of this augmented conversation that AI is part of that. It's an extension of you and enables you to think things and explore things with other people who are also enhanced by their own AIs. That's where we have to go and I think that the challenge is sort of be owned by the AIs or be in charge of them.
I know people talked about how AIs are the children of mankind and I kind of reject that. I don't want to be, I don't think, I don't think humans should go away. I think humans should be enhanced and that's really kind of the big struggle in our history may well be that one turning point where it's like, at the end of the day, are humans in this loop or are they a memory?
Bill Murphy: 00:59:45 That's an interesting, I hadn't really thought about from that angle but the AIs that have been, the AI arms race at Google and Facebook and Apple and the like, Palantir and these others, so they're at an AI arms race to basically serve us information, they think that they want us to see and they're sort of in charge of those algorithms but what you're saying is, what about algorithms that haven't been created that say, okay, Bill Murphy has this unique set of things that he wants and he wants to be enhanced in this way or as these unique skill sets here. Now, let's scan the universe for, for capabilities that make Bill better in this area and serve him the information in a certain way versus making him just randomly serving it based on a company, what they want to sell me.
David Smith: 01:00:34 One scary way to think about it, by the way is what they're doing is trying to model you, the big companies and as that model improves, it becomes more you. In other words, they're creating a slave that is an identical representation of you, it's going to think the way you do, it's going to act the way you do. It's their slave, they own you and just like that, it's all you're doing is saying well where was it? Where would this go?
It will go with that. We're going to have AIs that kind of model, they're a shadow version of you but they think like you. They do act like you because that's necessary to be able to predict what you're going to want, and then they can actually do simulations. I was just talking about, it's like, oh let's simulate, let's try and change this big guy's behavior and they'll try that out on the AI and say oh, it worked great. Let's do it on a real person.
Bill Murphy: 01:01:29 That's already here. That's the thing is we're talking about it, but that stuff's here and it's interesting. I'm in the cyber security world and people are wondering, okay, because there's sort of AI layered into a lot of our security systems now, and then what happens when the AI goes bad? I said, Well, I had an AI professor and researcher on the show a while back and he said, well, we're going to create AIs that watch the AI.
David Smith: 01:01:54 That's right. My example of this is security as a game, and AIs are really, really, really good at games so I actually think we've already lost. I think there's no AI, nobody's really put AIs watching that are worth a crap but you have to believe that creating an AI that can break a system to break in, like I said, it's a game and we're coming late to the party. I actually think everything's already hacked oddly enough, and maybe I'm wrong, but I sort of see it, you're applying AIs to a game like Go and I think you're fooling yourself if you think security is more complex than Go. These guys have, the AIs have already won if somebody decided to build that system.
Bill Murphy: 01:02:48 Well, I'm hopeful for the future and if it's guys like you that are going to help bring this age in, at least we know we can have adult conversations about what's real and then crafting the next set of technologies to address it.
David Smith: 01:03:07 Yeah, it's a marvelous time. I mean, it is a golden age and a golden age to find in terms that we don't know, the things we don't know are significantly greater than the ones we do but we see this world of opportunity and possibility opening up and we have challenges but I think that was the other thing that Engelbart was talking about was he wanted to build systems that enabled humans to solve complex problems. He said the complexity of being a human is increasing exponentially, and we need to have tools that match that.
Bill Murphy: 01:03:48 Building systems to solve complex problems.
David Smith: 01:03:51 Yep.
Bill Murphy: 01:03:52 Well, David, this is a great, great time talking with you and I'm going to put down the links that we talked about, your current projects with Croquet Studios and the related conversations we've had about your past and if people wanted to reach out to you in any way, what is the best way that they can learn more about your work and your projects?
David Smith: 01:04:17 Oh wow. Probably the best way to get started because a lot of it is visual is on YouTube, David A. Smith 3D shows, you can see like a historical context all the way back to my game Virtus Walkthrough and then ICE and then Alan and I did a demo of the Croquet Version 2000 and there's some newer ones up there too, so yeah, that's probably the best place to get started.
I can send you a list of others. It's www.croquet.studios.com. If you go there, what you'll see is a splash screen but show up in a week or so, you'll see a lot more.
Bill Murphy: 01:05:03 Excellent.
David Smith: 01:05:05 Yeah, this is the funniest time of my life actually, even though it's such a complex world we're in but all that happens is we need these capabilities and we need them now.
Bill Murphy: 01:05:17 I agree with you and that's what I've talked about the wisdom in the past and the guys like you to bring this new age into being so I appreciate your time today, David. Thank you very much.
David Smith: 01:05:29 Thank you. I really appreciate it. Great.
About David Smith
David Alan Smith is the CEO and Founder of Croquet Studio, developing Croquet – the Augmented Reality Collaborative Operating System.
Smith is a computer scientist and entrepreneur who has focused on interactive 3D and using 3D as a basis for new user environments and entertainment for over thirty years. His specialty is system design and advanced user interfaces. He is a pioneer in 3D graphics, robotics, telepresence, artificial intelligence and augmented reality (AR). He creates world-class teams and ships impossible products.
Smith was Chief Innovation Officer and Lockheed Martin and Senior Fellow at Lockheed Martin MST, where he was focused on next generation human centric computing and collaboration platforms. Before joining Lockheed Martin, Smith was the chief architect of the Croquet Project, an open source virtual world collaboration platform where he worked with Alan Kay (Turing Award winner) and was later CTO and co-founder of Teleplace Inc. providing a collaboration platform developed specifically for enterprises based on Croquet.
In 1987, Smith created The Colony, the very first real time 3D adventure game/shooter and the precursor to today’s first-person shooters. The game was developed for the Apple Macintosh and won the “Best Adventure Game of the Year” award from MacWorld Magazine. Smith founded Virtus Corporation in 1990 and developed Virtus Walkthrough, the first real-time 3D design application for personal computers. Smith also co-founded Red Storm Entertainment with author Tom Clancy. David gave the keynote address at IEEE VR 2017.
- Platform for the Future of AR & VR |David Smith|TEDxBeacon Street
- ARIA ARIA – AR in Action, David Smith
- Croquet Demo, Part 1, David A. Smith
- Croquet Demo, Part 2, David A. Smith
How to get in touch with David A. Smith
This episode is sponsored by the CIO Innovation Forum, dedicated to Business Digital Leaders who want to be a part of 20% of the planet and help their businesses win with innovation and transformation.
* Outro music provided by Ben’s Sound
Leave a Review
If you enjoyed this episode, then please consider leaving an iTunes review here
Click here for instructions on how to leave an iTunes review if you’re doing this for the first time.