Data architecture is being disrupted, echoing the evolution of software architecture over the past decade. The changes coming to data engineering will look and sound familiar to those who have watched monoliths be broken up into microservices: DevOps to DataOps; API Gateway to Data Gateway; Service Mesh to Data Mesh. While this will have benefits in agility and productivity, it will come with a cost of understanding and supporting a next-generation data architecture.
Data engineers and software architects will benefit from the guidance of the experts in this eMag as they discuss various aspects of breaking down traditional silos that defined where data lived, how data systems were built and managed, and how data flows in and out of the system.
We would love to receive your feedback via editors@infoq.com or on Twitter about this eMag. I hope you have a great time reading it!
Free download
The InfoQ eMag - Modern Data Engineering include:
- The Future of Data Engineering - Chris Riccomini examines the current and future states of the art in data pipelines, data streaming, and data warehousing. He presents a six-stage evolution that data ecosystems follow from simple monolith to a complex data-microwarehouse architecture as the data engineers who manage them solve problems and clarify their roles as infrastructure engineers rather than data stewards.
- Beyond the Database, and beyond the Stream Processor: What's the Next Step for Data Management? - Databases have been around forever with the same shape: you make a request to your data and then you receive an answer. Now, stream processors came along with a different approach: data isn’t locked up, it is in motion. Understand how stream processors and databases relate and why there is an emerging new category of databases that focus on data that stays in place as well as data that moves.
- Data Gateways in the Cloud Native Era - Data Gateways act like API Gateways but focus on access to the data aspect. A Data Gateway offers abstractions, security, scaling, federation, and contract-driven development features. There are many types of Data Gateways, from the traditional data virtualization technologies, to light GraphQL translators, cloud-hosted services, connection pools, and fully open source alternatives.
- Combining DataOps and DevOps: Scale at Speed - DataOps is an extension of DevOps standards and processes into the data analytics world. It's about streamlining the processes involved in processing, analyzing and deriving value from big data.
InfoQ eMags are professionally designed, downloadable collections of popular InfoQ content - articles, interviews, presentations, and research - covering the latest software development technologies, trends, and topics.