何涛-Vineyard开源分布式内存数据管理引擎GOTC深圳会场23页.pdf

综合技术专场

：开源分布式数据管理引擎

何涛2021年8月1日

阿里巴巴

Whybother

1.Sharingdataefficiently(with“0-copy”)betweenlibraries

iseasywithinasinglepythonprocess

2.Itisnotaseasytodosoacrossprocesses/runtimesona

singlemachinePyDataisthede-factostandardfordataanalysis

–PossiblewithplasmafromApacheArrow,alocalTherearelotsoflibrariesfordifferentworkloads

objectstoreusingshared-memory(imagecredit:https://coiled.io/blog/pydata-dask/)

3.Whataboutprocessingbigdatathatcannotfitintoa

singlemachine,andinvolvingdifferentworkloads?

–Usevineyard+K8s!

Bigdataanalyticalpipelines

Ananti-fraudpipeline

ML/Tensorflow

SQL

Fraud

LabelPostETL

Loaddata,ETLdetection

Propogationprocessing

withDNN

GraphComputation

（LPA）/GRAPESQL

Bigdataanalyticalpipelines

Ananti-fraudpipeline

DataExtractionGraphAnalyticalTrain/InferwithPostprocessing

andAlgorithmsaNNmodelandETL

Preprocessing

Tensorflow/

ETLGRAPESQL

（SQL）Pytorch

DataWarehouse(Hive)、DistributedFS(HDFS),ObjectStore(S3)