The official PASS Blog is where you’ll find the latest blog posts from PASS community members and the PASS Board. Contributors share their thoughts and discuss a wide variety of topics spanning PASS and the data community.

Is DataOps More Than Just DevOps?

I’ve been drawn in by hype before, but the experience has me casting a critical eye every time there’s a new trendy buzzword going around. DataOps had that effect on me, I raised eyebrow and laughed when I first heard it at the start of 2018. It started appearing on hype lists later in the year. For me, the easy portmanteau of “Data” and “DevOps” (already a conjunction of Development and Operations) was unnecessary. Surely, whether it applies to data, web dev, app dev or any other kind of coding, the idea of DevOps applies the same.

But then… if having a special name for it, just for the “Data People” means that people actually embrace DevOps principles and think about reducing their cycle time, eliminating blockers and removing risk from code deployments, then sell me a t-shirt and call me a convert. DataOps forever. But, DataOps advocates argue that DataOps is about a whole lot more than just DevOps…

Have You Built ‘the DevOps’ Yet?

Here’s my issue - DevOps done right is an organisation change. It’s an elimination of the barriers between developers, quality assurance, and operations; the creation of a kick-ass team that has the skills to develop software, provision resources, and build automated tests. Now, creating a cross-functional team alone couldn’t achieve what DevOps sets out to do. You need to underpin it with technological ideals such as automated testing, infrastructure as code and decoupled release procedures.

Figure 1 - Azure “DevOps”... Really?

Unfortunately, it’s the tech that’s misunderstood. Because a series of tools have evolved to serve these needs, they’re commonly labelled as the “DevOps” element of a project. Sure, if you want to attain the velocity and development cycles of modern teams, you need to put effort into building out the deployment pipelines and codifying your infrastructure. But, too often, I’ve seen projects where CI/CD elements are tacked on at the end. They’re a “feature” to be delivered – as soon as they’re done, someone ticks a box saying: “We’re now doing DevOps”, without the underpinning cultural epiphany. Having these tools will save you time in the long run, but it’s not a magic fix for your organisational problems. For anyone who hasn’t yet, the Phoenix Project is a great read on applying the theory of constraints to IT operations, it’ll help the whole idea sink in.

What Makes DataOps Different?

Why then, is DataOps such a different beast? As a word of warning, if you go delving into the murky depths of the internet, you’ll find many DataOps diagrams that are DevOps diagrams with the names changed.

You’ll see DevOps being combined with Agile and Lean Manufacturing principles to create “DevOps” but for me this is just part of DevOps itself. If you try and implement the DevOps automation elements alone, without running in an agile style and using real process controls, you haven’t embraced DevOps – so that’s not a good argument for the distinction for me.

Figure 2 - DataKitchen.io's DataOps Heritage

What does resonate however, is the principle difference of data platforms to more classic solution architecture – constant change and experimentation.

In a traditional software platform, if a developer tries something new and it doesn’t work – it rarely makes it to live. Whereas on a data platform, experimentation is performed on live data; it has to be. Data Science teams need the freedom to bring in new datasets, to try five different modelling techniques and pick the one that works. Analytics teams need to quickly react to urgent business demands and deploy new reports as soon as they’re ready. Data Platforms need to combine the rigour and discipline of software development with the flexibility and agility of data work – and THAT is why DataOps is seen as a separate movement.

Figure 3 - Convergence of disciplines in DataOps

For me, this bringing together of disciplines, adoption of modern processes and general modernisation of data organisations is essential. Applying engineering principles to Data Science gives huge benefits and helps avoid the many pitfalls of starting a data science capability… but don’t even get me started on the people who call that “MLOps”…

If you’d like to know more about augmenting your warehouses with modern data engineering and agile data delivery, please get in touch at simon@advancinganalytics.co.uk or visit www.advancinganalytics.co.uk to learn more.

Till next time.

Simon Whiteley Director of Engineering

Simon Whiteley
About the author

Simon Whitely is the Director of Engineering for Advancing Analytics Ltd and a Microsoft Data Platform MVP. Simon is a seasoned solution architect and technical lead with well over a decade of Microsoft Analytics experience. A deep techie with a focus on emerging cloud technologies and applying "big data" thinking to traditional analytics problems, Simon also has a passion for bringing it back to the high level and making sense of the bigger picture. When not tinkering with tech, Simon is a death-dodging London cyclist, a sampler of craft beers, an avid chef, and a generally nerdy person.

Please login or register to post comments.

Back to Top
cage-aids
cage-aids
cage-aids
cage-aids