Oracle has open sourced Graphpipe – a standard, high-performance protocol for transmitting tensor data over the network. The new tool should simplify and standardize the deployment and query of machine learning models for enterprises from any framework.

Graphpipe comprises a set of libraries and tools for following the standard. Vish Abrams, an architect focusing on cloud, virtualization, and container technologies, leads the project at Oracle.

Organizations face several unique challenges while deploying and managing machine learning technology on their own. First, there is no standard for model serving APIs, so they are likely stuck with whatever their framework gives them. Second, building their model server can be incredibly complicated. Finally, many of the existing solutions don’t focus on performance, so for certain use cases they fall short. With GraphPipe, Oracle aims to address these three challenges.

“Presently, no dominant standard exists for how tensor-like data should be transmitted between components in a deep learning architecture. As such it is common for developers to use protocols like JSON, which is extremely inefficient, or TensorFlow-serving’s protocol buffers, which carries with it the baggage of TensorFlow, a large and complex piece of software. GraphPipe is designed to bring the efficiency of a binary, memory-mapped format while remaining simple and light on dependencies,” Abrams says in a blog post.

GraphPipe includes:
• A set of flatbuffer definitions
• Guidelines for serving models consistently according to the flatbuffer definitions
• Examples for serving models from TensorFlow, ONNX, and caffe2
• Client libraries for querying models served via GraphPipe

In essence, a GraphPipe request behaves like a TensorFlow-serving predict request, but using flatbuffers as the message format. GraphPipe’s servers can serve models built in TensorFlow, PyTorch, mxnet, CNTK, or caffe2.

GraphPipe is now available on Oracle’s GitHub.