Written by Tom Krazit
Amazon’s first real web service brought us everything from Pinterest to coronavirus vaccines. Fifteen years later, insiders tell Protocol how it grew to store more than 100 trillion objects.
In late 2005, Don Alvarez was just another software entrepreneur struggling to get a new business off the ground when a friend working at Amazon invited him to check out a secret project that would change the world.
Alvarez’s startup, FilmmakerLive, was designing online collaboration applications for creative professionals and faced a common problem for that time: storage. Tech startups were just starting to claw their way back from the excesses of the dot-com era, and buying expensive hardware was a risky bet for a startup. Buy too little and your site crashes. Buy too much and you go broke. For the chaotic life of a startup, that was a risky bet.
He was skeptical about what he could learn about movie collaboration from an ecommerce company, but took the friend up on his offer.
“Rudy Valdez blew my mind,” Alvarez told Protocol. Valdez was then the head of business development for AWS, which at that time offered only a handful of basic services. He gave Alvarez, now director of engineering for Mural, a taste of Amazon’s first and arguably most fundamental product: S3, a cloud-based object storage service.
S3, or Simple Storage Service, made its debut 15 years ago this weekend. It would be years before “the cloud” became one of the most disruptive forces in the history of enterprise computing. Amazon didn’t even use the term when it announced S3 on March 14, 2006. But the storage service’s launch instantly solved some very tricky problems for entrepreneurs like Alvarez, and would come to change the way all businesses thought about buying information technology.
Startups like Pinterest, Airbnb and and Stripe flocked to AWS in the coming years, and older companies like Netflix — then a DVD-mailing operation — also took the plunge to retool their operations for the internet.
“Amazon was putting infinite disk space in the hands of every startup at an incredibly low and pay-for-what-you-need price point, there was nothing like that,” Alvarez said. “The second piece was that their API was so simple that i could just pick it up and build something useful in it, in the first 24 hours of using an unreleased, unannounced product.”
S3 is now a vital cog in the AWS machine, which generated more than $45 billion in revenue last year. It has evolved in many different directions over the last 15 years, yet has kept a set of design principles drawn up by a team led by Allan Vermeulen, Amazon’s chief technical officer during the earliest days of AWS, at the heart of its strategy.
“We knew what [customers] wanted to do then,” Mai-Lan Tomsen Bukovec, vice president for AWS Storage and the current head of S3, told Protocol. “But we also knew that applications would evolve, because our customers are incredibly innovative, and what they’re doing out there in all the different industries is going to change every year.”
Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage.Photo: Amazon Web Services
Building for flexibility
“When people think bigger and faster in computers, they think of this,” said Vermeulen during an interview in 2014, drawing a line in the air up and to the right. But storage technology has evolved differently, he said, over a period of long plateaus followed by sharp increases in capabilities: “It’s the difference between driving my Tesla and flying my airplane.”
S3 was one of those sharp breaks from the status quo. It was a godsend for developers like Alvarez, who no longer had to worry about buying and maintaining pricey storage hardware just to do business.
“There was nothing that we had access to that provided anything remotely like what S3 could do,” Alvarez said. “I felt like somebody had just given me the keys to the candy store.”
Like much of AWS, S3 was born from Amazon’s experience building and scaling Amazon.com, which taught it a lot of hard lessons about the limits and possibilities of distributed computing.
“A forcing function for the design was that a single Amazon S3 distributed system must support the needs of both internal Amazon applications and external developers of any application. This means that it must be fast and reliable enough to run Amazon.com’s websites, while flexible enough that any developer can use it for any data storage need,” AWS said in the original launch press release for S3 in 2006.
In the early days of the cloud, performance and reliability were a huge concern. And those concerns were especially fraught when it came to data, which even 15 years ago was understood to be one of the most important assets in a company’s arsenal.
“When we launched S3 15 years ago, S3 had eight microservices, and we have well over 300 now.” Tomsen Bukovec said, referring to the then-novel software development practice of breaking up large chunks of interdependent code into smaller, independent services.
Building around microservices allowed AWS to decentralize points of failure for S3 while also creating a system designed to acknowledge that distributed cloud services will fail on occasion, and that such failures shouldn’t take the entire system down.
It also allowed the company to layer on future enhancements without having to disturb the core pieces of the system: AWS now claims that S3 offers “11 9s” of reliability, or an astonishing 99.999999999% uptime that exceeds self-managed storage equipment by a large margin. (Other cloud storage vendors have matched this standard.)
S3 began life as a holding pen for simple web elements like images and video that website operators would pull down from AWS to your browser when you loaded a page. Over time, as companies became more comfortable with cloud storage, they started putting all kinds of data in S3.
And that’s when things started to get a little messy.
Amazon Web Services’s booth at the Microsoft PDC event in Los Angeles in 2008.Photo: D. Begley/Flickr
Plugging leaky buckets
If you look back at any number of security incidents over the past several years, a large number of them can be attributed to “leaky buckets,” referring to the core unit of S3 storage. These incidents happen to other cloud providers as well, but given AWS’s market share it’s a problem the company has had to deal with on many, many occasions.
AWS operates under a “shared responsibility” model for security: AWS will prevent anyone from physically accessing its servers or infiltrating its network, but customers are expected to protect their accounts to a reasonable extent. In other words, you can’t blame the rental car company if someone steals your laptop from the back seat of an unlocked vehicle.
Yet time and time again, cloud customers have left sensitive data belonging to their own customers in unprotected storage buckets open to anyone who can find them, which is easier than you might think. It’s just one example of how AWS has had to evolve some of its core products to meet customers where they are, especially later-arriving customers accustomed to accessing everything they need from private, internal networks.
“In a business application world, you don’t need to have access outside the company, or really outside a group of users within the business,” Tomsen Bukovec said. But it was clear that AWS needed to do more to help its customers help themselves, which led to the development of tools like Block Public Access that could lock down all storage buckets associated with a corporate account.
It was also clear to outsiders in the fast-growth early days of AWS that Amazon’s famous “two-pizza teams” were “both a strength and a weakness,” Alvarez said.
“It enabled every one of those services to rocket forward at a speed none of those competitors could match. And in the early days, it meant there was a lot less consistency [and] that was hard to puzzle through and manage,” he said, noting that the experience has improved over time.
Additional security tools have followed that let customers scan their accounts for unauthorized access from the public internet, or assign different levels of access to people with different roles within a company.
“Where we’re seeing customers go with their migrations is that they often have hundreds of buckets and lots and lots of [different] roles,” Tomsen Bukovec said of the newcomers to the cloud who seem most prone to these mistakes. “When we think about what to build to help customers secure the perimeter of their AWS resource, we think about how they would like to audit and how they would like to control” access to their storage resources inside S3.
Getting to 100 trillion
S3 continued to evolve in the years following its debut, and it also got a lot cheaper: By the time AWS got around to having its first major re:Invent developer conference in 2012, one of the major announcements from that week was a 24% to 28% percent reduction in S3 storage prices, the 24th such price cut the company had made up to that point.
Those price cuts were possible because AWS was able to upgrade the underlying S3 service on the fly, as Alyssa Henry, then vice president of AWS Storage Services, explained during a keynote address in 2012.
S3 was originally designed to hold 20 billion objects in storage, but it grew more quickly than anyone had anticipated, hitting 9 billion objects within the first year. The company upgraded the underlying storage service with more capacity in mind without any disruption to the original S3 customers, and By 2012 it had scaled to 1 trillion objects in storage, and by 2020, 100 trillion.
“What’s really cool about this is customers didn’t have to do anything: You didn’t have to go out buy the next upgrade — v2 of Amazon S3; you didn’t have to do the migration yourself; you just got it all for free, it just worked, things just got better,” Henry, who is now executive vice president and head of Square’s Seller unit, said at the 2012 event. “That’s one of the differences with the cloud versus how traditional IT has been done.”
A similar upgrade rolled out just last year, when AWS introduced strong consistency across S3.
Consistency is a data-storage concept that can rattle your brain a bit the first time it shows up; older storage systems such as the original S3 were designed around “eventual consistency,” meaning that a storage service wouldn’t always be able to tell you right away if a new piece of data had settled into its designated storage bucket, but it would catch up before long.
Now that modern applications move much faster, however, anything that makes a query to a storage service really needs to know the exact, current list of available data to perform at the expected level. So over the last couple of years, AWS rebuilt S3 around strong consistency principles, which other cloud providers offer but were able to roll out against a much smaller user base.
“That is a very complicated engineering problem,” Tomsen Bukovec said, and it was one of the stand-out announcements from the re:Invent 2020 among the geekier set of AWS users.
As they head into a new decade, Tomsen Bukovec and her team are looking at ways to make it easier to do machine learning on top of S3 data, and to improve the performance and capabilities of data lakes that allow for fine-grained analysis of internal and customer data among AWS users.
In fact, the Moderna vaccine for COVID-19 was developed with the help of a S3 data lake, Tomsen Bukovec said.
“We have this unique view that we built up over 15 years of usage, where we can determine what our customers are trying to do, and how we can build [S3] in such a way that it keeps true to that simple, cost-effective, secure, durable, reliable and highly-performant storage,” she said.
