Protobuf ?

Hello friends this is a follow-up to my earlier post related to gRPC Vs Restconf and as promised below is a quick summary on Protobuf (the idea that made gRPC look so good). 

Before I summarize Protobuf, let's revisit what serialization is?  Serialization is taking a complex data structure and serializing it into a string that can be sent over a connection. The recipient will use the de-serialization process to recover the original data structure. Serialization is also required when you want to save data structure to a file. Some popular forms of serialization are JSON, YAML, XML, and protocol buffers(Protobuf).

Protobuf is a method of serializing data that can be transmitted over wire or be stored in a file. 

Two keywords to remember when we talk about Protobuf - Protobuf is platform-neutral and highly optimized for microservices architecture. 

As of 2019, Google has over 48000 Protobuf message types in 12000 .proto files.

Speed:

Protobuf is faster than XML/JSON serialization as it offloads the description of data in a proto file. 

20-100x times faster than XML 

Network bandwidth:

Protobuf utilizes binary transmission format over string used by JSON/XML thus less bandwidth is required during transmission. 

3-10X smaller than JSON/XML

Throughput:

Protobuf payload size is ~60% smaller than JSON. 

JSON is self-contained while Protobuf utilizes schema(.proto) files to keep a record of data types and schema.

Courtesy: Geoffery Hunter

Backward compatibility: 

Super easy, if the new change does not work move to your last known good proto file schema.

Polyglot support:

Protobuf support many popular programming languages like C#, Go, python, Java, Javascript.

Engineer productivity :

Protobuf support automatic client code generation using code generators, just specify the proto file with the target language.

However, there are times when JSON is a better fit:

 1) Data to be human readable(protobuf is binary)

 2) Data is consumed directly by a web browser(right now there is no support in web-browser for protobuf)

 3) When your server-side application is in javascript.

 4) JSON is self-contained (protobuf needs a proto file) 

Next post gRPC :)

References:

Serialization formats - Geoffrey Hunter

My earlier post on gRPC Vs Restconf

gRPC Vs RESTCONF

Hello friends this is the follow-up post of my previous post on Netconf Vs Restconf and in this post, I have tried to compare gRPC with RESTCONF implementation. I hope this shall provide an answer to "why gRPC". 

Microservices-based architecture is the contemporary software design and development practice and gRPC is the best option because of its unmatched performance and polyglot(many programming languages) support. 

Per google gRPC is 

"a modern, bandwidth and CPU efficient, low latency way to create massively distributed systems that span data centers, as well as power mobile apps, real-time communications, IoT devices and APIs"

Let's summarize the main difference between gRPC and RESTCONF

What could be the limitations of using gRPC?

1. Higher ramp-up time of development teams 

2. You cannot call a gRPC service from a web browser (because of HTTP/2) and need a proxy 

Next post let's deal with protobuf  :) Happy reading

References:

Google blog - gRPC

My earlier post on NETCONF Vs RESTCONF 

A short youtube video(Why gRPC ?)

NETCONF Vs RESTCONF

Hello friends, last week many of my colleagues asked me about Netconf, Restconf & gRPC, specifically what is the difference among them. 

At a high level, my colleagues understand that these protocols were developed to minimize "vendor-lockin" and build vendor-agnostic network management & monitoring applications for a specific technology. 

Let me try to summarize(as succinctly as possible):

1. Both NETCONF and RESTCONF have CONF in common, which means these interfaces can be used to retrieve the device configuration and manipulate the device configuration. 

2. Both NETCONF and RESTCONF use YANG data models which define the schema/template of the information that is pushed and retrieved from device/s.

3. Why do we need RESTCONF: RESTful APIs were very famous in the software industry, many processes, and integration were already done using RESTful APIs, keeping this in mind RESTCONF was promoted as another technique(apart from NETCONF) to work with YANG data models.

Let's compare the major difference/s:

A quick summary of major differences

Comparison of operations between Netconf and Restconf:

Netconf  layers(from RFC):

gRPC in another post and will link it to this post, Happy reading! 

References:

RFC Netconf(6241)

RFC RESTCONF(8040)

Edge Compute - Why ?

Edge computing is a logical evolution of cloud-based network architecture (hosting centralized compute, storage for business application/s). 

Cloud architecture resulted in the exponential growth of business applications providing pay as you grow business model, truly revolutionary. What happened next was every business application(there are exceptions and valid use cases for not moving to the cloud) moved to the cloud to save CAPEX, improved time to market, and operational flexibility( XaaS). 

Datacenter began to proliferate at an exponential speed and as per Statista's survey, there are 7.2 million data centers globally as of 2021. 

A logical business problem that became much more noticeable with cloud architecture was almost all the end-user data was routed to a datacenter for business analytics and return traffic to execute any business decision on the end device.

IoT exacerbated this problem with now millions of devices ready to send traffic to these data centers. You can guess, the Internet will become a bottleneck(bandwidth) apart from latency and any unpredictable network disruptions. 

How does Edge computing help? 

Edge compute moves some portion of the storage and compute resources from centralized data centers and closer to the data source itself. It can be considered as an evolution from centralized computing to distributed computing. Keep it in mind it is not like traditional computing with on-premise deployment, although you can consider it something similar to the branch office concept. 

What could be the challenge? 

Decentralization may lead to high levels of monitoring and control as compared to the centralized model, are we going back 😉 ?

Interesting use cases: 

1. Autonomous vehicles: Everyone's favorite, firstly you will have millions of cars, and sending everything to a centralized data center will choke the internet, and secondly you don't want to have latency or networking disruptions while the car is on it way 😉

2. Farming: Collect all the sensor data, compute it locally and ensure crops are harvested in peak season. 

3. User experience(latency): Measure performance for users across the internet and determine the most reliable low latency path 

What is the opportunity :

For product vendors, it provides a new market segment to build devices with limited compute and storage but enormous volume, pluggable/s to connect these edge mini data centers to end-users and upstream to mother data centers. 

I plan to follow with an additional post on the architecture for edge computing, till then 

Happy reading and stay safe ! 

Transponder(less) Architecture

Hello friends, today I will touch upon how some innovations in silicon photonics and miniaturization of the transponder functions catalyzing the evolution of new transport network architecture. 

You can also read Transponder(less) architectures as architectures with "less/limited" transponder/s.

Every device, connection, and software code could be a potential point of failure in a network and keeping the # of components minimal to realize services is always a good idea. The new architecture with reduced components enables this design philosophy. 

Now let's see what do we mean by transponder "less" architecture. 

**I am using Transponder and muxponder interchangeably, read Transponder as (Trans/Mux)ponders.

In general, Transponders provided gray-to-color translation(for long-distance transmission), and muxponder aggregate many smaller services into a bigger colored pipe to transport over the optical transport network.

Transponder less architecture is the act of collapsing the features provided by the transponder/muxponder to another point/layer in the network. 

How is this possible now?

The main reason this is possible now is the miniaturization of transponder features into small form factors like CFP2-DCO and ZR/+. 

These optics have consumed the DSP and other optical analog functions provided by transponders thus eliminating the need for a transponder( generally fulfilled by specific hardware). 

IPoDWDM was something similar done in past but the solution required having a Transponder(like) hardware on routing boxes consuming slots on routing box and also interlocking the refresh cycles of routers and transport hardware( both having considerably different refresh cycles)

How would it benefit 

With commercial versions of CFP2 and ZR available many CSPs can take the advantage of these innovations to reduce the network cost and more importantly minimize the # of components required to realize the same service. 

In many metro and DCI where distance is limited (~30-80kms) many network connections can now be made with these new optics and some passive(or active) mux/demux blades without a need of deploying an entire transport line system. 

Apart from reducing the CAPEX this should significantly reduce the OPEX( limited power requirement, employee training, hardware spare), and the most important benefit in my view is a simplified network. 

Happy reading! Stay safe!

References:

CFP2 DCO optics 

400G ZR Optics

Customer is WRONG !

Hello friends, an apology for deploying a provocative title to catch your attention😉

All organizations aspire to "delight" the customers, quality is implicit and any slack in user experience can turn into an existential crisis for the product and organization. 

No one can escape accountability when customers are not able to consume the product/service for the problem the product/service was intended to solve but, 

what happens when customers discover a new workflow or configure/use the product in a way it was never designed or built 😟

Customer fault  -  A BIG NO 

Now how can you find these "unusual", "out of the order" workflows before they are reported by customers? 

Validation teams within an organization have the accountability to ensure these problems are found and fixed before the feature/product/service is available to the customer. 

For products having limited complexity and combinations, validating these negative scenarios can work( especially with the automated test suite) but for a multilayer,multi-component complex product/service this approach(find & fix) will not scale. 

Secondly, does it even make sense to spend time and resources in validating something which is not a revenue (💲💲)generating activity? 

What is a better approach? - "Mistake proofing"

One such mistake-proofing technique is "Poka-Yoke".  Poka ➡ "mistake", Yoke ➡"prevent"

Poka-Yoke is a process introduced by Japanese engineer Shigeo Shingo. The purpose of Poka-Yoke is to develop processes to reduce defects by avoiding or correcting mistakes in the early design and development phases.

Building a product that takes care of this unusual workflow in design and development is the best phase to avoid a humungous effort later( with the find & fix approach). 

All the cross-functional team's product owners, software architects, development, validation, operations & support team should come together during the design phase and plug all the possible entry points preventing any inadvertent misuse of the product/service by the end-user. 

"Remember Proactive is better than Reactive"

Finding defects(on paper)

The shelf life of technology is decreasing and the contraction is accelerating every day. What does this mean? The time from ideation to production is shortening and new disruptions have quickly become a norm. 

High quality of the service and/or product is an implicit assumption for the organization/product to exist. 

How do we measure product quality? Run it through a test cycle, file the deviations, fix what is necessary, and document what can be managed.

The testing phase involves writing a test plan, building testbeds, environment, recording/reporting deviations, validating the changes, and finally handing it over to the customers. Testing is a costly but mandatory endeavor to gain confidence in the success of the product. 

So what is finding defects on paper mean? 

In the current system of product validation rewards/recognition is based on what is recorded 😀. 

Organizations tend to reward folks where NO(or almost) defects were reported for the Software/Hardware piece they delivered AND/OR folks who reported maximum high-quality defects. Seems to be a sensible way to differentiate among employees providing the maximum value in a specified time. 

What most organizations don't measure is how many of these defects could be found without testing literally(remember literal testing is a cost to the organization and profitability) and continuously encourage (and provide time, resources, and training) to find as many defects as possible on paper(without testing literally).  

Finding defects on paper means how many defects can be found during PRD review( yes as early as PRD), design review, code review, and test plan review. 

Now we can debate, that is how it is done today so how is it different?

There are two tactical actions that organizations can start to measure the effectiveness(current) and build a feedback loop for continuous improvement. 

First, for any review, feedback/comments/AI which are not clarifications but problems(defects) caught early,  start tracking them as early offline defects. Keep the overhead of tracking these offline defects as minimal as possible by using automation. Closing them automatically when the review owner confirms the compliance. 

2) During  PPA(post-project analysis)  review the defects which were found "literally" and identify if strengthening any upstream review process would have helped to find them "on paper". This will build a feedback loop that continuously strengthens the review processes. 

In summary, for us, it is difficult to appreciate and reward the actions that we cannot measure(and 👀) and compare. 

Finding defects through the review process helps to reduce the cost of defects(because they are found early), save cost (running labs, servers), enable any participants in the review process to report, and moves the focus of improvement(and investment to strengthen) on upstream processes.