Sometimes an exciting new technology or platform brings changes that were hard to foresee. The major benefits are easy to see and eagerly anticipated, but other effects are only obvious in retrospect. Of course, there are the notorious "unexpected consequences", which are the negative side-effects of a change, but happily there are also unexpected benefits.
I had a chance recently to recount some experiences in migrating large data structures, warehouses, and similar large-scale data resources, from an on-prem data center to a cloud platform. In more than one case, when the long-endured limitations on capacity are released, the data admins, engineers and architects have surprised themselves by “doing less”: smaller warehouses, fewer special purpose “data marts”, fewer and simpler modules and components, not what was expected when the move was planned.
The engineers and administrators responsible for these data centers have spent their careers carefully forecasting demand and planning capacity. It takes time and upfront money to have the hardware, licensed software and other infrastructure in-place, and budgets never cover *all* the predicted, or hoped for growth. So the data center at a growing company is always running at or near capacity, and managers make some tough decisions about priorities. When the time comes to migrate those resources to “the cloud”, the seemingly limitless capacity can be intoxicating. However, when the migration is complete, the data warehouse, or data lake or whatever it is called, has become leaner. The reason, of course, is that a cloud platform is not only “limitless” but also “elastic”.
When new capacity, for storage or for compute processing, can be scaled upward with a mouse-click, one does not need to have the data resources pre-built and ready. If and when the new demand arrives, when the customer base grows or the new products take-off, the data center can expand as required. Until then, there is no need to build out and instantiate resources that are not needed. Another old saying from the consulting world, ”Use today’s dollars to cover today’s needs, and leave tomorrow’s needs to tomorrow’s dollars”.
The important caveat, however, is that the plans need to be in place and ready to go. A strong suggestion I’ll offer is to build out at least a POC version of the resource you expect to need and store the structure as a script. Look into “infrastructure as code”. When it's time to instantiate the new resource and make it live for use, you will have a proven structure ready to implement. And you will have a backup as well, should you need to rebuild. Make this a part of your cloud strategy from the start!
