"Back when we started, we worked with four terabyte drives. "We replace our hardware every four years, but have at least a couple of new generations in those four years," Le said.
The company helps design its own custom servers, cramming more and more storage into its data centers. Some of the big players still don't use SMR." "They need to build a new file system, etc. "This is one of the examples where we were able to work closely with hard drive companies and were able to move much faster than some other companies," Le said. SMD disks can be much denser by writing new tracks that overlap part of the previously written magnetic track, somewhat like overlapping roof shingles. This allowed us to gain a lot of efficiencies from avoiding the file system, but also move quite a bit faster."įor example, the company could adopt shingled magnetic recording (SMR) hard disk drives without waiting for drivers to support them. We moved from using a file system to just managing the drive directly - we literally open the drive as a block device and then we have our own formats. "We also significantly improved the architecture. "It's not only the language we changed," Le said. This involved a huge amount of software work - including switching from programming language Go to Rust mid-way through to reduce memory use - and getting deeply involved with the hardware to ensure that every ounce of possible storage was squeezed out of a rack. Over a two-and-a-half-year period, the company built its own massive on-premises platform, officially launching it in 2015. This article appeared in Issue 42 of the DCD>Magazine. This, Le said, has allowed for significant cost savings and more control - but is not something that most other companies could easily replicate. The result was Magic Pocket, one of the largest data migrations off the cloud in web history. "It was only a few years later, when we really believed we could tackle this problem better for our needs, that we even tried."
"We used AWS S3 because storage at scale was an extremely hard problem to solve," Le said. It didn’t take long for the company to wonder whether it made more sense to do it themselves. The company has always had its own data center presence, but Dropbox needed more capacity and soon grew to become a major customer of Amazon S3 (Amazon Simple Storage Service) after joining in 2013. When file hosting service Dropbox first announced its hybrid cloud effort Magic Pocket in 2016, many saw it as a sign that the company was done with Amazon Web Services and was betting on an on-premise future.īut the reality is more nuanced, lead developer Preslav Le told DCD.
0 kommentar(er)
