{"id":33591,"date":"2022-04-03T22:12:16","date_gmt":"2022-04-03T14:12:16","guid":{"rendered":"https:\/\/swarma.org\/?p=33591"},"modified":"2022-04-03T22:12:16","modified_gmt":"2022-04-03T14:12:16","slug":"%e9%a2%84%e8%ae%ad%e7%bb%83%e8%af%ad%e8%a8%80%e6%a8%a1%e5%9e%8b%e5%a6%82%e4%bd%95%e5%ad%a6%e4%b9%a0%e4%ba%8b%e5%ae%9e%e6%80%a7%e7%9f%a5%e8%af%86%ef%bc%9f%e5%85%bc%e8%ae%ba%e4%b8%80%e7%a7%8d%e5%9b%a0","status":"publish","type":"post","link":"https:\/\/swarma.org\/?p=33591","title":{"rendered":"\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\u5982\u4f55\u5b66\u4e60\u4e8b\u5b9e\u6027\u77e5\u8bc6\uff1f\u517c\u8bba\u4e00\u79cd\u56e0\u679c\u5e72\u9884\u7684\u8bc4\u4f30\u65b9\u6cd5"},"content":{"rendered":"
\n
\n
\n
\n
\n

<\/p>\n<\/section>\n

\n
\n
\n
\n
\n
\n

<\/section>\n<\/section>\n
\n
\n

\u5bfc\u8bed<\/strong><\/p>\n<\/section>\n<\/section>\n

\n

<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
\n
\n
\n

<\/strong>\u8fd1\u5e74\u6765\uff0c\u5bf9\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\uff08Pre-trained Language Model, PLM\uff09\u6355\u83b7\u7684\u4e8b\u5b9e\u6027\u77e5\u8bc6\u8fdb\u884c\u7814\u7a76\u6210\u4e3a\u4e00\u79cd\u8d8b\u52bf\u3002\u8bb8\u591a\u7814\u7a76\u663e\u793a\uff0cPLM \u80fd\u591f\u5728\u5b8c\u5f62\u586b\u7a7a\u5f62\u5f0f\u7684\u4e8b\u5b9e\u76f8\u5173\u95ee\u9898\uff0c\u5982\u201c\u4f46\u4e01\u51fa\u751f\u5728[MASK]\u201d\u4e2d\uff0c\u586b\u8865\u7f3a\u5931\u7684\u4e8b\u5b9e\u8bcd\u3002\u7136\u800c\uff0cPLM \u5982\u4f55\u6b63\u786e\u5730\u751f\u6210\u7ed3\u679c\u4ecd\u7136\u662f\u4e00\u4e2a\u8c1c\uff1a\u4f9d\u9760\u6709\u6548\u7684\u7ebf\u7d22\u8fd8\u662f\u6377\u5f84\u6a21\u5f0f\uff1f\u534e\u4e3a\u8bfa\u4e9a\u65b9\u821f\u5b9e\u9a8c\u5ba4\u8bed\u97f3\u8bed\u4e49\u56e2\u961f\u4e0e\u63a8\u8350\u641c\u7d22\u56e2\u961f\u7684\u4e00\u9879\u6700\u65b0\u7814\u7a76\u6210\u679c\uff0c\u8bd5\u56fe\u901a\u8fc7\u4e00\u79cd\u56e0\u679c\u542f\u53d1\u7684\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\u5206\u6790\u65b9\u6cd5\u6765\u56de\u7b54\u8fd9\u4e2a\u95ee\u9898\u3002<\/span><\/strong><\/p>\n


<\/span><\/strong><\/p>\n

\u7814\u7a76\u9886\u57df\uff1a\u56e0\u679c\u63a8\u65ad\uff0c\u81ea\u7136\u8bed\u8a00\u5904\u7406<\/strong><\/strong><\/span><\/strong><\/span><\/strong><\/span><\/strong><\/p>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n

\n
\n
\n
<\/svg><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
\n
<\/strong><\/section>\n<\/section>\n
\n

\u674e\u5c11\u535a\u3001\u674e\u5c0f\u5149\u3001\u8463\u632f\u534e<\/span> <\/span>| \u4f5c\u8005<\/strong><\/span><\/p>\n

\u9093\u4e00\u96ea<\/span> | \u7f16\u8f91<\/strong><\/span><\/strong><\/span><\/p>\n


<\/p>\n


<\/p>\n

<\/p>\n<\/section>\n

\n
\n
\n

\u8bba\u6587\u9898\u76ee\uff1a<\/span><\/p>\n

How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis<\/span>
<\/p>\n

\u8bba\u6587\u94fe\u63a5\uff1a<\/span><\/section>\n
https:\/\/arxiv.org\/abs\/2203.16747<\/span><\/section>\n<\/section>\n<\/section>\n<\/blockquote>\n

<\/section>\n

\u534e\u4e3a\u8bfa\u4e9a\u65b9\u821f\u5b9e\u9a8c\u5ba4\u8bed\u97f3\u8bed\u4e49\u56e2\u961f\u4e0e\u63a8\u8350\u641c\u7d22\u56e2\u961f\u7684\u6700\u65b0\u7814\u7a76\u6210\u679c\uff1a\u56e0\u679c\u542f\u53d1\u7684\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\u5206\u6790\u65b9\u6cd5\uff0c<\/span>\u88ab2022\u5e74ACL Finding\u5f55\u7528\u63a5\u6536<\/span>\u3002\u8be5\u7814\u7a76\u901a\u8fc7\u56e0\u679c\u5e72\u9884\u7684\u65b9\u6cd5\uff0c\u5206\u6790\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\u80fd\u5426\u5b66\u4e60\u5230\u4e8b\u5b9e\u6027\u77e5\u8bc6\uff0c\u4ee5\u53ca\u5b66\u4e60\u8fc7\u7a0b\u7684\u5185\u90e8\u673a\u7406\uff0c\u8fd9\u662f\u534e\u4e3a\u7b2c\u4e00\u6b21\u5c06\u56e0\u679c\u63a8\u65ad\u6280\u672f\u5e94\u7528\u4e8e\u81ea\u7136\u8bed\u8a00\u5904\u7406\u9886\u57df\u7684\u7814\u7a76\uff0c\u4e5f\u662f\u6211\u4eec\u7814\u7a76\u5982\u4f55\u6784\u5efa\u9762\u5411\u56e0\u679c\u5173\u7cfb\u5efa\u6a21\u7684\u9884\u8bad\u7ec3\u8bed\u8a00\u6a21\u578b\u7684\u7b2c\u4e00\u6b65\u3002<\/span><\/span><\/p>\n


<\/span><\/span><\/p>\n

Mask Language Modeling<\/span>\uff08MLM\uff09<\/span>\u5728\u88ab\u5e7f\u6cdb\u5e94\u7528\u4e8ePre-trained Language Model<\/span>\uff08PLM\uff09<\/span>\u4e2d\u540e\uff0c\u5927\u5bb6\u4e00\u76f4\u5728\u79ef\u6781\u5f00\u5c55\u5bf9MLM\u8fd9\u4e00\u4efb\u52a1\u7684\u5206\u6790\u3002\u5176\u4e2dLAMA\u63a2\u6d4b\u4e86\u5728\u7ecf\u7531MLM\u65b9\u5f0f\u7684\u8bad\u7ec3\u540e\uff0cPLM\u80fd\u5426\u6355\u83b7\u4e00\u5b9a\u91cf\u7684\u4e8b\u5b9e\u6027\u77e5\u8bc6\u3002LAMA\u8981\u6c42PLM\u56de\u7b54\u5b8c\u5f62\u586b\u7a7a\u5f62\u5f0f\u7684\u4e8b\u5b9e\u76f8\u5173\u95ee\u9898\uff0c\u4f8b\u5982\u201c\u7535\u5f71\u300a\u666e\u7f57\u7c73\u4fee\u65af\u300b\u7684\u5bfc\u6f14\u662f[MASK]\u201d\uff0c\u82e5PLM\u80fd\u591f\u6210\u529f\u5730\u5c06[MASK]\u66ff\u6362\u4e3a\u4e0e\u4e8b\u5b9e\u76f8\u7b26\u7684\u5185\u5bb9\u201c\u96f7\u5fb7\u5229\u00b7\u65af\u79d1\u7279\u201d\uff0c\u5219\u8ba4\u4e3aPLM\u6210\u529f\u5730\u6355\u83b7\u4e86\u8fd9\u4e00\u6761\u4e8b\u5b9e\u6027\u77e5\u8bc6\u3002\u901a\u8fc7\u6784\u9020\u5927\u91cf\u7684\u5b8c\u5f62\u586b\u7a7a\u5f0f\u95ee<\/span><\/span><\/span>\u9898\uff0cLAMA\u53ef\u4ee5\u6839\u636ePLM\u56de\u7b54\u7684\u6b63\u786e\u7387\uff0c\u5b9a\u91cf\u5730\u4f30\u8ba1PLM\u6240\u6355\u83b7\u4e8b\u5b9e\u6027\u77e5\u8bc6\u6570\u91cf\u3002<\/span><\/p>\n


<\/span><\/p>\n

\u901a\u8fc7LAMA\u5f62\u5f0f\u7684\u63a2\u6d4b\uff0c\u4eba\u4eec\u53d1\u73b0PLM\u53ef\u4ee5\u901a\u8fc7\u5728MLM\u6837\u672c\u4e0a\u7684\u9884\u8bad\u7ec3\uff0c\u6765\u6355\u83b7\u4e00\u5b9a\u6570\u91cf\u7684\u4e8b\u5b9e\u6027\u77e5\u8bc6\u3002\u8fd9\u8fdb\u4e00\u6b65\u5f15\u53d1\u4e86\u4eba\u4eec\u5bf9\u201cPLM\u662f\u5982\u4f55\u8bb0\u4f4f\u8fd9\u4e9b\u4e8b\u5b9e\u6027\u77e5\u8bc6\u7684\uff1f\u201d\u8fd9\u4e00\u95ee\u9898\u7684\u597d\u5947\u3002\u5bf9\u6b64\uff0c\u672c\u6587\u4ece\u4ee5\u4e0b\u4e24\u4e2a\u65b9\u9762\u5bf9\u8fd9\u4e00\u95ee\u9898\u8fdb\u884c\u4e86\u63a2\u7a76\uff1a<\/span><\/p>\n