知识图谱论文资源1912.00147.pdfVIP

IntegratingGraphContextualizedKnowledgeintoPre-trainedLanguageModels

1111123

BinHe,DiZhou,JinghuiXiao,XinJiang,QunLiu,NicholasJingYuan,TongXu

1HuaweiNoahsArkLab

2HuaweiCloudAI

3SchoolofComputerScience,UniversityofScienceandTechnologyofChina

hebin.nlp,zhoudi7,xiaojinghui4,jiang.xin,qun.liu,nicholas.yuan@,tongxu@

9

1

0Abstract

2

Complexnodeinteractionsarecommoninknowledgegraphs,

candtheseinteractionsalsocontainrichknowledgeinforma-

e

tion.However,traditionalmethodsusuallytreatatripleas

Datrainingunitduringtheknowledgerepresentationlearn-

3ing(KRL)procedure,neglectingcontextualizedinformation

ofthenodesinknowledgegraphs(KGs).Wegeneralizethe

]modelingobjecttoaverygeneralform,whichtheoretically

Lsupportsanysubgraphextractedfromtheknowledgegraph,

Candthesesubgraphsarefedintoanoveltransformer-based

.modeltolearntheknowledgeembeddings.Tobroadenus-

sagescenariosofknowledge,pre-trainedlanguagemodelsareFigure1:Asubgraphextractedfromamedicalknowledge

cutilizedtobuildamodelthatincorporatesthelearnedknowl-graph.Therectanglesrepresentmedicalentitiesanddirected

[edgerepresentations.Experimentalresultsdemonstratethatarrowsdenotetherelationsbetweentheseentities.Onlypart

2