Carmine's Kernel

Oana Platon: Inside Service Fabric Health Manager

Oana Platon is a Principal Software Engineer - and employee number 10 - on the Service Fabric team. She wrote the first version of the replicator you learned about in Sumukh's awesome deep dive. She holds a degree in Computer Science from the Politehnica University of Bucharest (Universitatea Politehnica din București).

For the past few years, Oana has been working on the Health Manager, a critical system service inside Service Fabric's Management subsystem.

Health Manager enables health monitoring of applications, services, and cluster entities. Cluster entities (such as nodes, service partitions, and replicas) can report health information, which is then aggregated into the centralized health store. This health information provides an overall point-in-time health snapshot of the services and nodes distributed across multiple nodes in the cluster, enabling you to take any needed corrective actions. Health query APIs enable you to query the health events reported to the health subsystem. The health query APIs return the raw health data stored in the health store or the aggregated, interpreted health data for a specific cluster entity.

How does the Health Manager work? When a cluster is determined to be in an unhealthy state, what happens? How is health management exposed to developers? What are some of the areas that Oana and team are working on to evolve Service Fabric health management, analysis and evaluation?

 

Tune in. Meet Oana.

 

 

Health Manager source code

Learn more about Service Fabric health management

Service Fabric Subsystem Explorer

 

 

 

Sumukh Shivaprakash: Inside Service Fabric Data Replication

Sumukh Shivaprakash is a senior developer on the Service Fabric team. He works on multiple projects related to reliability and data replication. His is the low-level world of replicated state machines and distributed data reliability.

One of these areas is the replicator that works in concert with the Reliable State Manager (deep dive coming soon) and the TStore that you learned a lot about with Preetha in our last conversation introducing Reliable Collections ( https://youtu.be/wcfewGJ6Tu8 ). Specifically, Sumukh digs into the logging replicator and friends. This is a natural continuation from Preetha's excellent whiteboarding adventure.

Get some popcorn, put on your thinking cap, and enjoy. Meet Sumukh, another super talented developer on the Service Fabric team.

Tune in. Meet Sumukh.

 

Links:

Service Fabric Repo: http://aka.ms/SFRepo

Sumukh's code related to this conversation:

Logging Replicator

Transaction Replicator

Service Fabric Subsystem Explorer

 

 

 

Preetha Subbarayalu: Inside Reliable Collections

Preetha Subbarayalu is a Senior Software Engineer on the Service Fabric team. She works on Reliable Collections and, specifically, the backing store that makes Reliable Collections reliably reliable. Deep interviews her and time is spent at the whiteboard digging into the architecture and implementation of Reliable Collections.

Reliable Collections enable you to write highly-available, geo-scale, and low-latency cloud applications as though you were writing single computer applications. Remember the essence of Service Fabric articulated by our fearless leader, Gopal? The whole point of our blood, sweat, and tears is to make programming distributed systems easy, to make programming in the large feel like programming in the small. Tune in and see how this translates to what Preetha crafts.

Enjoy.

Links:

Reliable Collections Docs

 

 

Gopal Kakivaya: The Essence of Service Fabric

I recently got the opportunity to geek out with the founder and VP of Service Fabric. It was highly educational. It's amazing how much Gopal knows and it's equally impressive that he was not at all prepared for this interview - meaning he had no idea what we were going to talk about besides Service Fabric. Many executives have handlers who require questions in advance. Not Gopal. He's actually a VP who is also a hands-on engineer and architect. Of course, I filmed it (that was the point, really) :) Enjoy!

Gopal Kakivaya is the founder and Vice President of Microsoft Azure's Hyperscale Compute team, which is the home of Service Fabric, an open source, general purpose distributed systems infrastructure and microservices platform that powers several of Azure's core services. Of course, it's not just for internal use by the likes of Cosmos DB, SQL Azure, Intune, Skype for Business, Azure VM APIs, Event Grid, and more... It's tailored specifically for general purpose use for running cloud (micro)services of any size and complexity - authored by you.

The primary goal of Service Fabric, from a developer's perspective is, as Gopal says, "to make programming in the large (planetary scale) feel like programming in the small (your dev machine)". All of the distributed computing complexity of the cloud is abstracted away and you're left to focus on what really matters to you and your business: your (micro)service(s), your scenarios, your code.

So, how does Service Fabric solve some of the key hard problems of distributed computing, even abstracting them away completely from developers? Where did Service Fabric originate and where is it heading? What's the essence of Service Fabric?

Gopal gives an introductory tour of Service Fabric's subsystem architecture. We also discuss Service Fabric's built-in fault analysis services, the importance of verifying the behavior of cloud services under faulty conditions, and the future of Service Fabric's Chaos Engineering infrastructure.

Tune in. Enjoy. Learn.

 

More information:

Service Fabric Overview

Service Fabric Github Repo

Core Service Fabric Subsystem Explorer

Service Fabric Controlled Chaos

 

Service Fabric Chaos Engineering Service: We want to hear from you!

Greetings fellow chaos engineers and SREs!

I am currently working on adding capabilities to Service Fabric's Chaos engineering and experimentation service and I'm looking for feedback and insights from folks who are already using Service Fabric (but, of course, if you're not, please do check out our offering!).

Service Fabric is a general purpose distributed systems infrastructure and orchestrator (VMs, containers, etc...). It powers a lot of Microsoft Azure today including services like Cosmos DB, SQL Azure, etc...

Service Fabric was originally designed for internal (to Azure) workloads, but has been in the public domain for a few years now - and just recently we open-sourced it.

Service Fabric has a built-in (first class!) chaos experimentation subsystem as part of its Fault Analysis Service. SF Chaos is geared toward injecting faults that typically happen in clustered distributed system environments that are designed for extreme availability (node failure, quorum loss, replication failure, application level start/stops, etc...).

You can experiment with Service Fabric clusters in the cloud by using a free Party Clusters service if you want to deploy your SF application to a cloud environment without having an Azure account.

I'm looking to get more information on who's using our Chaos service today (we know many services are, but we don't know who they are) in order to get your feedback and insights, which will help in furthering the capabilities of the system. Also, if you have any questions, just ask. You can always get to me by emailing ctorre at microsoft... If you didn't know Service Fabric has chaos experimentation built-in and first class, now you do :slightly_smiling_face: Please let me know if you are using it and want to chat.

Cheers and happy chaos!

...Charles

We are the chaos monkeys of planet Earth

The video below is well worth 28 minutes of your time.

 

Dr. James Lovelock is as correct and as prescient as ever. Gaia has been unquestionably proven (thus it's a theory and no longer a hypothesis...) and, as he mentions, is just called another name by modern bio/eco/geological science ("Earth Systems Science", specifically)... Pay attention and hopefully you'll learn a few things that are independent of Climate Politics (even if you disagree with an environmentalist who supports nuclear energy and questions the impact of some currently popular alternative energy resources - or if you disagree with "Climate Change"...).

Open your mind. Listen.

The Earth is a system of systems and to ensure our species will continue as an inhabitant for as long as possible, we'll need to work on solutions to counter and prevent positive feedback loops - amplification of processes that negatively impact the overall stability and resiliency of the Earth System, Gaia, effectively "injecting" negative feedback versus positive.

We are the chaos monkeys of planet Earth. This is true. Yet we are also the conscious of the Earth - its eyes and ears - as Dr. Lovelock so clearly states. Through our intelligence, itself the result of billions of years of evolution, the Earth has for the first time achieved a level of self-awareness that it can recognize itself for what it is - a self-sustaining planet, where biological systems benefit from the planet's conditions which support life and effectively maintain these conditions over time via the actions associated with living (algae producing clouds over the oceans, the oxygen and carbon cycles, etc...)....

We are not a plague or a virus or a cancer..., but more so we are a biological system that tends to add chaos to an already highly resilient system of systems (in part because of our level of awareness and intelligence - an incredible irony indeed...), in which we are inextricably tied (after all, we aren't only "made of stars". We sprung from this planet like everything else that lives here...).

We can do the right thing for our future, but we will need to work out some critical mental flaws in addition to inventing technologies and promoting/regulating behaviors that will limit the global affect of the chaos we tend to create.

 

Developing Manycore Applications with Concurrency Auto-Tuners

Continuing on with our coverage of ICSE 2011, meet Dr. Victor Pankratius. Dr. Pankratius heads the Multicore Software Engineering investigator group at the Karlsruhe Institute of Technology, Germany. He also serves as the elected chairman of the Software Engineering for parallel Systems (SEPARS) international working group. Dr. Pankratius' current research concentrates on how to make parallel programming easier. His work on multicore software engineering covers a range of research topics including empirical studies, auto-tuning, language design, and debugging.

In this video, Wolfram Schulte joins Victor to discuss the challenges of making concurrency easier for developers. One of the really interesting approaches that Victor and team are investigating is concurrency auto-tuning, and the example discussed here involve adding OS kernel-level support for auto-tuning user mode applications for manycore processor architectures. This is very fascinating research with great potential. Concurrency auto-tuner in an OS kernel? Concurrency-enlightened operating systems? Why not? Always great to meet young innovators with no fear of failure. I hope to see this type of thing materialize. Very interesting research and real world problem. Go Victor. Go!

Thanks to Wolfram and Victor for another great conversation. Tune in.

C++ Renaissance

When you think about all the code executing in the world at any given time, there's a good chance you're thinking about a lot of code written in C/C++ (aka native code). As Mohsen Agsen mentions so astutely in this impromptu and candid conversation, C++ can be thought of as the dark matter of the developer universe: there is so much of it all around us, hiding in the computational shadows, powering so much of what we take for granted, technologically.

C++ is currently undergoing a renaissance. This means that, by definition, the language, compilers and compositional tooling are evolving and coalescing into a state that maximizes native developer efficiency, productivity, and creativity across hardware and software domains (PCs, mobile devices, embedded systems, operating systems, user applications, services, etc). C++ is a powerful "systems" programming language, but it's more than that. It's object oriented, but it's more than that. At Microsoft, most of our flagship products are written in C++ (and C, like the Windows kernel...). As somebody with a keen interest in programming languages and software engineering, generally, I wanted to get some answers to broad questions concerning the language that consistently ranks near the top of the most widely used general purpose programming languages in the world. Who better to talk to than some key technical leaders driving Microsoft's Visual C++ business?

Mohsen Agsen is a Microsoft Technical Fellow and veteran C++ developer who runs the VC++ engineering team. Craig Symonds is the Director of Program Management for VC++ and a long time Microsoft dev tools veteran. Both Mohsen and Craig have been at the company for many years and have a ton of industry experience. I paid them a visit to see what’s on their minds these days regarding the native developer community, C++, Visual Studio, and more. As you will learn, Microsoft and Visual Studio, specifically, are re-doubling efforts to take part in the native code renaissance. Accordingly, you may see advances in our native tooling that the team thinks of as “C++ first” -> VC++ will extend its capabilities on a faster pace than it has ever done so in the past, at times surpassing the other VS languages/runtimes, in specific scenarios. This is exciting and a long time coming, but of course it's more passion than promise at this point. There is no specific news here, just perspectives and insights among some very bright people driving Microsoft's C++ efforts. I really enjoyed the conversation with Craig and Mohsen and hope to chat with them again in the near future.

Links into specific conversation points (video skimming):

Introductions
State of the union for C++ from Microsoft's perspective
C++ as both a low level and high level programming language
C++ as object oriented, but OO is a capability, not a requirement
Garbage collection in C++. Why not?
C++ portability
Visual Studio C++ integration and parity with the other languages/runtimes in VS and C++ first

Here’s to the C++ Renaissance.

Tune in. Enjoy.

C

100th Episode of Going Deep!

Welcome to the 100th episode of Going Deep! It's been a great ride so far and I hope you've been enjoying the show over the years. For this momentous occasion, meet Bart de Smet, a software engineer extraordinaire on the WPF team who spends his free time blogging (what an incredible wealth of truly useful technical information to be found on Bart's blog!) and creating custom LINQ providers. In fact, Bart is probably the world's most prolific LINQ provider creator, from LINQ-to-MSI to LINQ-to-Simpsons! How does he do it???

Of course, in this special episode who better to have involved in this LINQ'ified conversation (with lots of whiteboarding) than LINQ co-creator, programming languages designer, fundamentalist functional programming high priest and Channel 9 star Erik Meijer? In fact, given that this is an Expert to Expert Going Deep, Erik co-conducts the interview (and he's great at it as usual!). Nothing better than watching and listening to two experts geeking out at a whiteboard!

What makes LINQ so readily "providable"? How does Bart concoct his LINQ provider magic? Here, we dig into the details that enable LINQ-to-Anything.

Tune in. There's a lot of very useful technical information here (which I hope is the case for all 100 Going Deep episodes).

Enjoy and thanks for watching!

Software Composability and the Future of Programming Languages

How will imperative programming langauges evolve to suit the needs of developers in the age of Concurrency and Composability? What role can programming languages play in enabling true composability? What are the implications of LINQ on the furture of managed (CLS-based) and unmanaged(C++) languages? How will our imperative languages (static) become more functional (dynamic) in nature while preserving their static "experience" for developers?

Answers to these questions and much more are to be found in this Channel 9 interview with some of Microsoft's leading language designers and programming thought leaders: Anders Hejlsberg, Technical Fellow and Chief Architect of C#, Herb Sutter, Architect in the C++ language design group, Erik Meijer, Architect in both VB.Net and C# language design and programming language guru, and Brian Beckman, physicist and programming language architect working on VB.Net.

This is a great conversation with some of the industry's most influential programming language designers. Tune in. You may be surprised by what you learn...