{"id":63835,"date":"2025-08-25T17:16:10","date_gmt":"2025-08-25T09:16:10","guid":{"rendered":"https:\/\/swarma.org\/?p=63835"},"modified":"2025-08-25T17:16:10","modified_gmt":"2025-08-25T09:16:10","slug":"%e4%bb%8eq%e5%ad%a6%e4%b9%a0%e5%88%b0ai%e6%b8%b8%e6%88%8f%e5%a4%a7%e5%b8%88%ef%bc%9adqn%e5%a6%82%e4%bd%95%e7%a0%b4%e8%a7%a3%e5%a4%8d%e6%9d%82%e4%bb%bb%e5%8a%a1%ef%bc%9f%e4%b8%a8%e9%9b%86%e6%99%ba","status":"publish","type":"post","link":"https:\/\/swarma.org\/?p=63835","title":{"rendered":"\u4eceQ\u5b66\u4e60\u5230AI\u6e38\u620f\u5927\u5e08\uff1aDQN\u5982\u4f55\u7834\u89e3\u590d\u6742\u4efb\u52a1\uff1f\u4e28\u96c6\u667a\u767e\u79d1"},"content":{"rendered":"
\n
\n
\n
\n
<\/section>\n
\n
\n
\n
\n
\n

<\/span><\/section>\n<\/section>\n
\n
\n

\u5bfc\u8bed<\/span><\/strong><\/p>\n<\/section>\n<\/section>\n

\n

<\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
\n
\n
\n

\u6df1\u5ea6Q\u7f51\u7edc(Deep Q-Network, DQN)\u662f\u4e00\u79cd\u57fa\u4e8e\u6df1\u5ea6\u5b66\u4e60\u7684\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\uff0c\u5b83\u901a\u8fc7\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u6765\u8fd1\u4f3c\u6700\u4f18\u52a8\u4f5c\u4ef7\u503c\u51fd\u6570\uff08Q\u51fd\u6570\uff09\uff0c\u80fd\u591f\u76f4\u63a5\u4ece\u9ad8\u7ef4\u611f\u77e5\u8f93\u5165\uff08\u5982\u6e38\u620f\u753b\u9762\u50cf\u7d20\uff09\u4e2d\u5b66\u4e60\u6700\u4f18\u51b3\u7b56\u7b56\u7565\u3002\u8be5\u7b97\u6cd5\u7684\u6838\u5fc3\u521b\u65b0\u5728\u4e8e\u7ed3\u5408\u4e86\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u548c\u76ee\u6807\u7f51\u7edc\u6280\u672f\uff0c\u6709\u6548\u89e3\u51b3\u4e86\u4f20\u7edfQ\u5b66\u4e60\u5728\u975e\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u4e2d\u7684\u7a33\u5b9a\u6027\u95ee\u9898\uff0c\u9996\u6b21\u5728Atari\u6e38\u620f\u4e0a\u5b9e\u73b0\u4e86\u4ece\u539f\u59cb\u50cf\u7d20\u8f93\u5165\u5230\u63a7\u5236\u6307\u4ee4\u7684\u7aef\u5230\u7aef\u5b66\u4e60\uff0c\u8fbe\u5230\u4e86\u4eba\u7c7b\u6c34\u5e73\u7684\u6e38\u620f\u8868\u73b0\u3002<\/span><\/span>
<\/span><\/span><\/strong><\/span><\/strong><\/p>\n


<\/span><\/span><\/strong><\/p>\n

\u201c\u96c6\u667a\u767e\u79d1\u7cbe\u9009\u201d\u662f\u4e00\u4e2a\u957f\u671f\u4e13\u680f\uff0c\u6301\u7eed\u4e3a\u5927\u5bb6\u63a8\u9001\u590d\u6742\u6027\u79d1\u5b66\u76f8\u5173\u7684\u57fa\u672c\u6982\u5ff5\u548c\u8d44\u6e90\u4fe1\u606f\u3002\u4f5c\u4e3a\u96c6\u667a\u4ff1\u4e50\u90e8\u7684\u5f00\u6e90\u79d1\u5b66\u9879\u76ee\uff0c\u96c6\u667a\u767e\u79d1\u5e0c\u671b\u6253\u9020\u590d\u6742\u6027\u79d1\u5b66\u9886\u57df\u6700\u5168\u9762\u7684\u767e\u79d1\u5168\u4e66\uff0c\u6b22\u8fce\u5bf9\u590d\u6742\u6027\u79d1\u5b66\u611f\u5174\u8da3\u3001\u70ed\u7231\u77e5\u8bc6\u6574\u7406\u548c\u5206\u4eab\u7684\u670b\u53cb\u52a0\u5165\uff0c\u6587\u672b\u53ef\u4ee5\u626b\u7801\u62a5\u540d\u52a0\u5165\u767e\u79d1\u5fd7\u613f\u8005\uff01<\/span><\/strong><\/span><\/strong><\/p>\n


<\/span><\/strong><\/span><\/strong><\/p>\n

<\/section>\n

\u2191\u2191\u2191\u626b\u7801\u76f4\u8fbe\u767e\u79d1\u8bcd\u6761<\/span><\/p>\n

<\/span><\/strong><\/p>\n


<\/span><\/p>\n<\/section>\n

\u5173\u952e\u8bcd\uff1a<\/span>\u6df1\u5ea6\u5b66\u4e60\uff0cDeep Q Network\uff0c\u65e0\u6a21\u578b\uff08Model free\uff09\u7b97\u6cd5\uff0c\u52a8\u4f5c\u4ef7\u503c\u51fd\u6570\uff0c\u7ecf\u9a8c\u56de\u653e\uff08Experience Replay)<\/span><\/strong><\/strong><\/strong><\/strong><\/span><\/section>\n
<\/strong><\/strong><\/section>\n
<\/strong><\/strong><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
\n
\n
\n
<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
\n
<\/section>\n<\/section>\n
\n

\u90b9\u4f73\u73ae\u3001\u7f57\u798f\u6d77<\/span> <\/span><\/span><\/span>| \u7f16\u5199<\/span><\/span><\/p>\n

\n

\u5f20\u6c5f<\/span><\/span> <\/span><\/span><\/span>| \u5ba1\u6821<\/span><\/span><\/p>\n<\/section>\n


<\/span><\/span><\/p>\n<\/section>\n

\n
\n
\n
\n
\n

\u76ee\u5f55<\/span><\/strong><\/p>\n

1. \u80cc\u666f<\/span><\/span><\/span><\/span><\/section>\n

1.1\t\u5f3a\u5316\u5b66\u4e60\u7684\u57fa\u672c\u6982\u5ff5\u4e0e\u6311\u6218<\/span><\/span><\/p>\n

1.1.1\t\u57fa\u672c\u6982\u5ff5<\/span><\/span><\/p>\n

1.1.2\t\u9762\u4e34\u7684\u6311\u6218<\/span><\/span><\/p>\n

1.2\tQ-Learning \u7b80\u4ecb<\/span><\/span><\/p>\n

1.3\t\u6df1\u5ea6\u5b66\u4e60\u4e0e\u5f3a\u5316\u5b66\u4e60\u7684\u878d\u5408\u80cc\u666f<\/span><\/span><\/p>\n

1.4\tDQN \u7684\u63d0\u51fa\u53ca\u5176\u5728 Atari \u6e38\u620f\u4e2d\u7684\u7a81\u7834\u6027\u6210\u679c<\/span><\/span><\/p>\n

1.4.1\t\u63d0\u51fa\u7684\u6a21\u578b<\/span><\/span><\/p>\n

1.4.2\t\u6a21\u578b\u7684\u7ec6\u8282<\/span><\/span><\/p>\n

1.4.3\t\u5177\u4f53\u7b97\u6cd5\u53ca\u5176\u8bad\u7ec3\u6d41\u7a0b<\/span><\/span><\/p>\n

2. Q-Learning \u57fa\u7840<\/span><\/span><\/p>\n

2.1\t\u9a6c\u5c14\u53ef\u592b\u51b3\u7b56\u8fc7\u7a0b\uff08MDP\uff09\u7b80\u4ecb<\/span><\/span><\/p>\n

2.1.1\tMDP \u7684\u6838\u5fc3\u601d\u60f3<\/span><\/span><\/p>\n

2.1.2\t\u7b56\u7565\u4e0e\u4ef7\u503c\u51fd\u6570<\/span><\/span><\/p>\n

2.2\tQ \u503c\u51fd\u6570\u7684\u5b9a\u4e49\u4e0e\u66f4\u65b0\u516c\u5f0f<\/span><\/span><\/p>\n

2.2.1\t\u5b9a\u4e49<\/span><\/span><\/p>\n

2.2.2\t\u66f4\u65b0\u516c\u5f0f<\/span><\/span><\/p>\n

2.3\t\u03b5-greedy \u7b56\u7565\u4e0e\u63a2\u7d22-\u5229\u7528\u6743\u8861<\/span><\/span><\/p>\n

2.4\tQ-Learning \u7684\u6536\u655b\u6027<\/span><\/span><\/p>\n

2.5\tQ-Learning \u7684\u5c40\u9650\u6027<\/span><\/span><\/p>\n

3. DQN \u7684\u6838\u5fc3\u601d\u60f3\u4e0e\u7ed3\u6784<\/span><\/span><\/p>\n

3.1\t\u4f7f\u7528\u795e\u7ecf\u7f51\u7edc\u903c\u8fd1 Q \u503c\u51fd\u6570<\/span><\/span><\/p>\n

3.2\t\u7ecf\u9a8c\u56de\u653e\uff08Experience Replay\uff09\u673a\u5236<\/span><\/span><\/p>\n

3.3\t\u76ee\u6807\u7f51\u7edc\uff08Target Network\uff09\u7684\u5f15\u5165\u4e0e\u53c2\u6570\u66f4\u65b0\u7b56\u7565<\/span><\/span><\/p>\n

4. DQN\u5728pytorch\u4e2d\u7684\u7b80\u5355\u5b8c\u6574\u5b9e\u73b0<\/span><\/span><\/p>\n

5. DQN \u7684\u4e3b\u8981\u6539\u8fdb\u7b97\u6cd5<\/span><\/span><\/p>\n

5.1\tDouble DQN\uff1a\u89e3\u51b3 Q \u503c\u8fc7\u9ad8\u4f30\u8ba1\u95ee\u9898<\/span><\/span><\/p>\n

5.1.1\tDouble DQN\u7684\u601d\u60f3<\/span><\/span><\/p>\n

5.1.2\tDouble DQN\u7684\u4f18\u52bf<\/span><\/span><\/p>\n

5.2\tDueling DQN\uff1a\u5206\u79bb\u72b6\u6001\u4ef7\u503c\u4e0e\u4f18\u52bf\u51fd\u6570<\/span><\/span><\/p>\n

5.2.1\tDueling DQN\u7684\u601d\u60f3<\/span><\/span><\/p>\n

5.2.2\tDueling DQN\u7684\u4f18\u52bf<\/span><\/span><\/p>\n

5.3\tDRQN<\/span><\/span><\/p>\n

5.3.1\tDRQN\u7684\u601d\u60f3<\/span><\/span><\/p>\n

5.3.2\tDRQN\u7684\u4f18\u52bf\u4e0e\u5c40\u9650<\/span><\/span><\/p>\n

6. DQN \u7684\u5e94\u7528\u9886\u57df<\/span><\/span><\/p>\n

6.1\tAtari \u6e38\u620f\u4e2d\u7684\u5e94\u7528\u4e0e\u8868\u73b0<\/span><\/span><\/p>\n

6.2\t\u673a\u5668\u4eba\u63a7\u5236\u4e0e\u8def\u5f84\u89c4\u5212<\/span><\/span><\/p>\n

6.3\t\u63a8\u8350\u7cfb\u7edf\u4e2d\u7684\u7b56\u7565\u4f18\u5316<\/span><\/span><\/p>\n

7. DQN \u7684\u5c40\u9650\u6027\u4e0e\u6311\u6218<\/span><\/span><\/p>\n

7.1\t\u5bf9\u8fde\u7eed\u52a8\u4f5c\u7a7a\u95f4\u7684\u5904\u7406\u56f0\u96be<\/span><\/span><\/p>\n

7.2\t\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u6837\u672c\u6548\u7387\u95ee\u9898<\/span><\/span><\/p>\n

7.3\t\u7b56\u7565\u6536\u655b\u901f\u5ea6\u6162\u4e0e\u7a33\u5b9a\u6027\u95ee\u9898<\/span><\/span><\/p>\n

7.4\t\u6cdb\u5316\u80fd\u529b\u4e0e\u8fc7\u62df\u5408\u98ce\u9669<\/span><\/span><\/p>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n


<\/span><\/span><\/h2>\n

\n
\n
\n
\n
\n

<\/span><\/section>\n

<\/span><\/section>\n
\n
<\/section>\n<\/section>\n<\/section>\n
\n

1. \u80cc\u666f<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n

\n

<\/span><\/section>\n

<\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n


<\/span><\/p>\n

1.1 \u5f3a\u5316\u5b66\u4e60\u7684\u57fa\u672c\u6982\u5ff5\u4e0e\u6311\u6218<\/span><\/span><\/span><\/h3>\n


<\/span><\/span><\/h3>\n

1.1.1 \u57fa\u672c\u6982\u5ff5<\/span><\/span><\/span><\/h4>\n


<\/span><\/span><\/h4>\n

\u5f3a\u5316\u5b66\u4e60\uff08Reinforcement Learning, RL\uff09\u662f\u4e00\u79cd\u673a\u5668\u5b66\u4e60\u65b9\u6cd5\uff0c\u5f3a\u8c03\u667a\u80fd\u4f53\uff08agent\uff09\u901a\u8fc7\u4e0e\u73af\u5883\u7684\u4ea4\u4e92\uff0c\u5728\u8bd5\u9519\u8fc7\u7a0b\u4e2d\u5b66\u4e60\u7b56\u7565\uff0c\u4ee5\u6700\u5927\u5316\u7d2f\u79ef\u5956\u52b1\u3002\u5176\u6838\u5fc3\u5728\u4e8e\u89e3\u51b3\u201c\u63a2\u7d22-\u5229\u7528\u201d\u4e4b\u95f4\u7684\u6743\u8861\uff1a\u65e2\u8981\u5c1d\u8bd5\u65b0\u7684\u52a8\u4f5c\u4ee5\u53d1\u73b0\u6f5c\u5728\u7684\u9ad8\u5956\u52b1\u7b56\u7565\uff08\u63a2\u7d22\uff09\uff0c\u53c8\u8981\u5229\u7528\u5df2\u77e5\u7684\u6700\u4f18\u7b56\u7565\u4ee5\u83b7\u5f97\u5373\u65f6\u6536\u76ca\uff08\u5229\u7528\uff09 \u3002<\/span><\/span><\/p>\n


<\/span><\/span><\/h4>\n

1.1.2 \u9762\u4e34\u7684\u6311\u6218<\/span><\/span><\/span><\/h4>\n


<\/span><\/span><\/h4>\n

\u5c3d\u7ba1\u5f3a\u5316\u5b66\u4e60\u5728\u6e38\u620fAI\u3001\u673a\u5668\u4eba\u63a7\u5236\u3001\u63a8\u8350\u7cfb\u7edf\u7b49\u9886\u57df\u53d6\u5f97\u4e86\u663e\u8457\u6210\u679c\uff0c\u4f46\u5728\u5b9e\u9645\u5e94\u7528\u4e2d\u4ecd\u9762\u4e34\u8bf8\u591a\u6311\u6218\uff1a<\/span><\/span><\/p>\n

\uff081\uff09\u6837\u672c\u6548\u7387\u4f4e\uff1a\u5f3a\u5316\u5b66\u4e60\u901a\u5e38\u9700\u8981\u5927\u91cf\u7684\u4ea4\u4e92\u6570\u636e\u624d\u80fd\u5b66\u4e60\u6709\u6548\u7b56\u7565\uff0c\u8fd9\u5728\u6570\u636e\u83b7\u53d6\u6210\u672c\u9ad8\u6216\u4ea4\u4e92\u53d7\u9650\u7684\u573a\u666f\u4e2d\u5c24\u4e3a\u56f0\u96be\u3002<\/span><\/span><\/p>\n

\uff082\uff09\u6cdb\u5316\u80fd\u529b\u5f31\uff1a\u5728\u7279\u5b9a\u73af\u5883\u4e2d\u8bad\u7ec3\u51fa\u7684\u7b56\u7565\uff0c\u5f80\u5f80\u96be\u4ee5\u9002\u5e94\u7a0d\u6709\u53d8\u5316\u7684\u65b0\u73af\u5883\uff0c\u7f3a\u4e4f\u8fc1\u79fb\u5b66\u4e60\u80fd\u529b\u3002<\/span><\/span><\/p>\n

\uff083\uff09\u5956\u52b1\u8bbe\u8ba1\u56f0\u96be\uff1a\u8bbe\u8ba1\u5408\u9002\u7684\u5956\u52b1\u51fd\u6570\u81f3\u5173\u91cd\u8981\uff0c\u4e0d\u5f53\u7684\u5956\u52b1\u8bbe\u8ba1\u53ef\u80fd\u5bfc\u81f4\u667a\u80fd\u4f53\u5b66\u4e60\u5230\u4e0e\u9884\u671f\u4e0d\u7b26\u7684\u884c\u4e3a\u3002<\/span><\/span><\/p>\n

\uff084\uff09\u8bad\u7ec3\u7a33\u5b9a\u6027\u4e0e\u6536\u655b\u6027\u95ee\u9898\uff1a\u7279\u522b\u662f\u5728\u57fa\u4e8e\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u7684\u6a21\u578b\u4e2d\uff0c\u8bad\u7ec3\u8fc7\u7a0b\u53ef\u80fd\u4e0d\u7a33\u5b9a\uff0c\u5bb9\u6613\u53d1\u751f\u53d1\u6563\u3002\u7b56\u7565\u6216\u73af\u5883\u7684\u5fae\u5c0f\u53d8\u5316\u53ef\u80fd\u5bfc\u81f4\u6027\u80fd\u7684\u6781\u5927\u6ce2\u52a8\uff0c\u4f7f\u5f97\u96be\u4ee5\u83b7\u5f97\u4e00\u81f4\u7684\u7ed3\u679c\u3002\u5728\u8fde\u7eed\u6216\u9ad8\u7ef4\u52a8\u4f5c\u7a7a\u95f4\u7684\u60c5\u51b5\u4e0b\uff0c\u8fd9\u79cd\u4e0d\u7a33\u5b9a\u6027\u66f4\u52a0\u660e\u663e\uff0c\u5b66\u4e60\u8fc7\u7a0b\u53d8\u5f97\u66f4\u590d\u6742\u4e14\u96be\u4ee5\u9884\u6d4b\u3002<\/span><\/span><\/p>\n


<\/span><\/p>\n

1.2 Q-Learning \u7b80\u4ecb<\/span><\/span><\/span><\/h3>\n


<\/span><\/p>\n

Q-Learning \u662f\u5f3a\u5316\u5b66\u4e60\u4e2d\u4e00\u79cd\u7ecf\u5178\u4e14\u5e7f\u6cdb\u5e94\u7528\u7684\u65e0\u6a21\u578b\uff08model-free\uff09\u7b97\u6cd5\uff0c\u5c5e\u4e8e\u57fa\u4e8e\u503c\uff08value-based\uff09\u7684\u65b9\u6cd5\u3002\u5176\u6838\u5fc3\u76ee\u6807\u662f\u901a\u8fc7\u5b66\u4e60\u4e00\u4e2a\u52a8\u4f5c\u4ef7\u503c\u51fd\u6570\uff08Q\u51fd\u6570\uff09\uff0c\u6307\u5bfc\u667a\u80fd\u4f53\u5728\u4e0d\u540c\u72b6\u6001\u4e0b\u9009\u62e9\u6700\u4f18\u52a8\u4f5c\uff0c\u4ee5\u6700\u5927\u5316\u957f\u671f\u7d2f\u79ef\u5956\u52b1\u3002<\/span><\/span><\/p>\n

\u4e3e\u4f8b\u8001\u9f20\u5728\u8ff7\u5bab\u4e2d\u5bfb\u627e\u5976\u916a\u6765\u7406\u89e3Q-Learning \uff1a<\/span><\/span><\/p>\n


<\/span>\uff081\uff09\u73af\u5883\u8bbe\u5b9a<\/span><\/span><\/p>\n

    \n
  • \u667a\u80fd\u4f53\uff08Agent\uff09\uff1a\u4e00\u53ea\u8001\u9f20\uff0c\u8d77\u59cb\u4f4d\u7f6e\u56fa\u5b9a\u3002<\/span><\/span><\/span><\/li>\n
  • \u76ee\u6807\uff08Goal\uff09\uff1a\u5728\u8ff7\u5bab\u7684\u53f3\u4e0b\u89d2\u6709\u4e00\u5757\u5976\u916a\uff0c\u8001\u9f20\u7684\u4efb\u52a1\u662f\u627e\u5230\u5e76\u5403\u5230\u5976\u916a\u3002<\/span><\/span><\/span><\/li>\n
  • \u969c\u788d\uff08Obstacle\uff09\uff1a\u8ff7\u5bab\u4e2d\u67d0\u4e9b\u4f4d\u7f6e\u6709\u6bd2\u836f\uff0c\u8001\u9f20\u9700\u8981\u907f\u514d\u8fd9\u4e9b\u4f4d\u7f6e\u3002<\/span><\/span><\/span><\/li>\n
  • \u52a8\u4f5c\uff08Actions\uff09\uff1a\u8001\u9f20\u53ef\u4ee5\u9009\u62e9\u5411\u4e0a\u3001\u4e0b\u3001\u5de6\u3001\u53f3\u79fb\u52a8\u3002<\/span><\/span><\/span><\/li>\n
  • \u5956\u52b1\uff08Rewards\uff09\uff1a\u5403\u5230\u5976\u916a\uff1a+10\uff1b\u8bef\u5165\u6bd2\u836f\uff1a-10\uff1b\u5176\u4ed6\u79fb\u52a8\uff1a0<\/span><\/span><\/span><\/li>\n<\/ul>\n


    <\/span>\uff082\uff09Q-Learning \u5b66\u4e60\u8fc7\u7a0b<\/span><\/span><\/p>\n

    \u8001\u9f20\u901a\u8fc7\u4e0d\u65ad\u5c1d\u8bd5\u4e0d\u540c\u7684\u8def\u5f84\uff0c\u5b66\u4e60\u5982\u4f55\u4ece\u8d77\u70b9\u79fb\u52a8\u5230\u5976\u916a\u6240\u5728\u7684\u4f4d\u7f6e\uff0c\u540c\u65f6\u907f\u514d\u6bd2\u836f\u3002\u6bcf\u6b21\u79fb\u52a8\u540e\uff0c\u8001\u9f20\u6839\u636e\u83b7\u5f97\u7684\u5956\u52b1\u66f4\u65b0\u5176\u5bf9\u5f53\u524d\u72b6\u6001\u4e0b\u5404\u4e2a\u52a8\u4f5c\u7684\u4ef7\u503c\u4f30\u8ba1\uff08\u5373 Q \u503c\uff09\u3002\u968f\u7740\u5b66\u4e60\u7684\u8fdb\u884c\uff0c\u8001\u9f20\u9010\u6e10\u5f62\u6210\u4e00\u5957\u7b56\u7565\uff0c\u4f7f\u5176\u80fd\u591f\u66f4\u5feb\u3001\u66f4\u5b89\u5168\u5730\u627e\u5230\u5976\u916a\u3002<\/span><\/span><\/p>\n


    <\/span>\uff083\uff09Q \u503c\u66f4\u65b0\u516c\u5f0f<\/span><\/span><\/p>\n

    Q-Learning \u7684\u6838\u5fc3\u5728\u4e8e\u901a\u8fc7\u4ee5\u4e0b\u516c\u5f0f\u66f4\u65b0 Q \u503c\uff1a <\/span><\/span><\/span><\/p>\n

    <\/section>\n

    \u5176\u4e2d\uff1a<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5f53\u524d\u72b6\u6001\uff1b<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5f53\u524d\u52a8\u4f5c\uff1b<\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5373\u65f6\u5956\u52b1\uff1b<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\uff1a\u6267\u884c\u52a8\u4f5c\u540e\u7684\u65b0\u72b6\u6001\uff1b<\/span><\/span>\u03b1<\/span><\/span>\u2208<\/span><\/span>(<\/span><\/span>0<\/span><\/span>,<\/span><\/span>1<\/span><\/span>]<\/span><\/span>\uff1a\u5b66\u4e60\u7387\uff1b<\/span><\/span>\u03b3<\/span><\/span>\u2208<\/span><\/span>[<\/span><\/span>0<\/span><\/span>,<\/span><\/span>1<\/span><\/span>)<\/span><\/span>\uff1a\u6298\u6263\u56e0\u5b50<\/span><\/span><\/span><\/p>\n


    <\/span>\uff084\uff09\u7b56\u7565\u9009\u62e9\uff1a\u03b5-\u8d2a\u5a6a\u7b56\u7565<\/span><\/span><\/p>\n

    \u4e3a\u4e86\u5728\u63a2\u7d22\u65b0\u8def\u5f84\u548c\u5229\u7528\u5df2\u77e5\u6700\u4f73\u8def\u5f84\u4e4b\u95f4\u53d6\u5f97\u5e73\u8861\uff0c\u91c7\u7528 \u03b5-\u8d2a\u5a6a\u7b56\u7565\uff1a<\/span><\/span><\/p>\n

      \n
    • \n
      \u4ee5\u6982\u7387 \u03b5 \u968f\u673a\u9009\u62e9\u4e00\u4e2a\u52a8\u4f5c\uff08\u63a2\u7d22\uff09<\/span><\/span><\/section>\n<\/li>\n
    • \n
      \u4ee5\u6982\u7387 1 – \u03b5 \u9009\u62e9\u5f53\u524d Q \u503c\u6700\u5927\u7684\u52a8\u4f5c\uff08\u5229\u7528\uff09<\/span><\/span><\/section>\n<\/li>\n<\/ul>\n

      \u968f\u7740\u8bad\u7ec3\u7684\u8fdb\u884c\uff0c\u9010\u6e10\u51cf\u5c0f \u03b5 \u7684\u503c\uff0c\u4ee5\u51cf\u5c11\u63a2\u7d22\uff0c\u589e\u52a0\u5229\u7528\u3002<\/span><\/span><\/p>\n


      <\/span>\uff085\uff09\u5b66\u4e60\u8fc7\u7a0b<\/span><\/span><\/p>\n

      \u4ee5\u4e0b\u662f\u8001\u9f20\u5728\u8ff7\u5bab\u4e2d\u5b66\u4e60\u7684\u7b80\u8981\u8fc7\u7a0b\uff1a<\/span><\/span><\/p>\n

      a.\u521d\u59cb\u5316<\/span><\/span><\/b>\uff1a\u5c06\u6240\u6709\u72b6\u6001-\u52a8\u4f5c\u5bf9\u7684 Q \u503c\u521d\u59cb\u5316\u4e3a 0\u3002<\/span><\/span><\/span><\/p>\n

      b.\u9009\u62e9\u52a8\u4f5c<\/span><\/span><\/b>\uff1a\u6839\u636e \u03b5-\u8d2a\u5a6a\u7b56\u7565\u9009\u62e9\u4e00\u4e2a\u52a8\u4f5c\u3002<\/span><\/span><\/span><\/p>\n

      c.\u6267\u884c\u52a8\u4f5c<\/span><\/span><\/b>\uff1a\u8001\u9f20\u79fb\u52a8\u5230\u65b0\u7684\u4f4d\u7f6e\uff0c\u83b7\u5f97\u76f8\u5e94\u7684\u5956\u52b1\u3002<\/span><\/span><\/span><\/p>\n

      d.\u66f4\u65b0 Q \u503c<\/span><\/span><\/b>\uff1a\u6839\u636e Q \u503c\u66f4\u65b0\u516c\u5f0f\uff0c\u66f4\u65b0\u5f53\u524d\u72b6\u6001\u4e0b\u8be5\u52a8\u4f5c\u7684 Q \u503c\u3002<\/span><\/span><\/span><\/p>\n

      e.\u91cd\u590d<\/span><\/span><\/b>\uff1a\u5c06\u65b0\u4f4d\u7f6e\u4f5c\u4e3a\u5f53\u524d\u72b6\u6001\uff0c\u91cd\u590d\u6b65\u9aa4 2 \u81f3 4\uff0c\u76f4\u5230\u8fbe\u5230\u7ec8\u6b62\u6761\u4ef6\uff08\u5982\u627e\u5230\u5976\u916a\u6216\u8fbe\u5230\u6700\u5927\u6b65\u6570\uff09\u3002<\/span><\/span><\/span><\/p>\n


      <\/span><\/p>\n

      \u901a\u8fc7\u591a\u6b21\u8fed\u4ee3\uff0c\u8001\u9f20\u80fd\u591f\u5b66\u4e60\u5230\u4ece\u8d77\u70b9\u5230\u5976\u916a\u7684\u6700\u4f18\u8def\u5f84\uff0c\u540c\u65f6\u907f\u514d\u6bd2\u836f\u3002\u8be5\u793a\u4f8b\u5c55\u793a\u4e86 Q-Learning \u5982\u4f55\u901a\u8fc7\u4e0e\u73af\u5883\u7684\u4ea4\u4e92\uff0c\u5b66\u4e60\u6700\u4f18\u7b56\u7565\u4ee5\u6700\u5927\u5316\u7d2f\u79ef\u5956\u52b1\u3002\u901a\u8fc7\u4e0d\u65ad\u66f4\u65b0 Q \u503c\uff0c\u667a\u80fd\u4f53\u80fd\u591f\u5728\u590d\u6742\u7684\u73af\u5883\u4e2d\u627e\u5230\u6700\u4f18\u8def\u5f84\u3002<\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1.3 \u6df1\u5ea6\u5b66\u4e60\u4e0e\u5f3a\u5316\u5b66\u4e60\u7684\u878d\u5408\u80cc\u666f<\/span><\/span><\/span><\/h3>\n


      <\/span><\/span><\/h3>\n

      \u6df1\u5ea6\u5b66\u4e60\u4e0e\u5f3a\u5316\u5b66\u4e60\u7684\u878d\u5408\u8d77\u6e90\u4e8e\u4e24\u8005\u4e92\u8865\u7684\u4f18\u52bf\uff1a\u5f3a\u5316\u5b66\u4e60\u64c5\u957f\u51b3\u7b56\u4e0e\u7b56\u7565\u4f18\u5316\uff0c\u800c\u6df1\u5ea6\u5b66\u4e60\u64c5\u957f\u5904\u7406\u9ad8\u7ef4\u611f\u77e5\u6570\u636e\uff08\u5982\u56fe\u50cf\u3001\u8bed\u97f3\u3001\u6587\u672c\u7b49\uff09\u3002\u4f20\u7edf\u5f3a\u5316\u5b66\u4e60\u65b9\u6cd5\u5728\u9762\u5bf9\u5927\u89c4\u6a21\u72b6\u6001\u7a7a\u95f4\uff08\u4f8b\u5982\u56fe\u50cf\u8f93\u5165\uff09\u65f6\uff0c\u56e0\u65e0\u6cd5\u4f7f\u7528\u67e5\u8868\u7684\u65b9\u5f0f\u5b58\u50a8 Q \u503c\u800c\u8868\u73b0\u4e0d\u4f73\u3002\u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e00\u74f6\u9888\uff0c\u7814\u7a76\u8005\u5f15\u5165\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u4f5c\u4e3a\u51fd\u6570\u903c\u8fd1\u5668\uff0c\u7528\u4ee5\u4f30\u8ba1\u72b6\u6001-\u52a8\u4f5c\u503c\u51fd\u6570\uff08Q \u503c\uff09\u6216\u7b56\u7565\u51fd\u6570\uff0c\u4ece\u800c\u5927\u5e45\u63d0\u5347\u4e86\u5f3a\u5316\u5b66\u4e60\u5728\u590d\u6742\u73af\u5883\u4e2d\u7684\u6cdb\u5316\u80fd\u529b\u548c\u5b9e\u7528\u6027\u3002<\/span><\/span><\/p>\n


      <\/span><\/p>\n

      \u8fd9\u4e00\u878d\u5408\u7684\u4ee3\u8868\u6027\u6210\u679c\u662f 2015 \u5e74 DeepMind \u63d0\u51fa\u7684\u6df1\u5ea6 Q \u7f51\u7edc\uff08DQN\uff09<\/span><\/span>[1]<\/span><\/span>\uff0c\u9996\u6b21\u5728\u65e0\u9700\u4eba\u5de5\u7279\u5f81\u8bbe\u8ba1\u7684\u60c5\u51b5\u4e0b\uff0c\u4ec5\u51ed\u50cf\u7d20\u8f93\u5165\u4fbf\u5728\u591a\u6b3e Atari \u6e38\u620f\u4e2d\u5b9e\u73b0\u4e86\u8d85\u8d8a\u4eba\u7c7b\u6c34\u5e73\u7684\u8868\u73b0\u3002<\/span><\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1.4 DQN \u7684\u63d0\u51fa\u53ca\u5176\u5728 Atari \u6e38\u620f\u4e2d\u7684\u7a81\u7834\u6027\u6210\u679c<\/span><\/span><\/span><\/h3>\n


      <\/span><\/p>\n

      <\/section>\n

      \u4e3e\u4f8b\uff1aAtari\u6e38\u620f\u4e4bbreakout\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n


      <\/span><\/span><\/p>\n

      Deep Q-Network\uff08DQN\uff09\u662f\u7531DeepMind\u56e2\u961f\u57282015\u5e74\u53d1\u8868\u7684\u8bba\u6587<\/span><\/span>[1]<\/span><\/span>\u4e2d\u63d0\u51fa\u7684\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\u3002\u8be5\u7814\u7a76\u9996\u6b21\u5c06\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u4e0eQ-learning\u7ed3\u5408\uff0c\u76f4\u63a5\u4ece\u9ad8\u7ef4\u611f\u5b98\u8f93\u5165\uff08\u5982Atari\u6e38\u620f\u7684\u50cf\u7d20\uff09\u5b66\u4e60\u63a7\u5236\u7b56\u7565\uff0c\u800c\u65e0\u9700\u624b\u5de5\u8bbe\u8ba1\u7279\u5f81\u3002DQN\u5728Atari 2600\u6e38\u620f\u4e0a\u7684\u8868\u73b0\u8fbe\u5230\u6216\u8d85\u8d8a\u4eba\u7c7b\u6c34\u5e73\uff0c\u6807\u5fd7\u7740\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u5728\u590d\u6742\u4efb\u52a1\u4e2d\u7684\u91cd\u5927\u7a81\u7834\u3002<\/span><\/span><\/span><\/p>\n


      <\/span>DQN\u7684\u6838\u5fc3\u521b\u65b0\u5305\u62ec\u4f7f\u7528\u6df1\u5ea6\u5377\u79ef\u795e\u7ecf\u7f51\u7edc(CNN)\u5904\u7406\u539f\u59cb\u50cf\u7d20\u8f93\u5165\uff0c\u6a21\u4eff\u751f\u7269\u89c6\u89c9\u7cfb\u7edf\u7684\u5c42\u6b21\u5316\u7279\u5f81\u63d0\u53d6\u80fd\u529b\u3002\u6b64\u5916\uff0c\u4e3a\u4e86\u89e3\u51b3\u4f20\u7edf\u5f3a\u5316\u5b66\u4e60\u5728\u975e\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u4e0b\u7684\u4e0d\u7a33\u5b9a\u6027\uff0cDQN\u5f15\u5165\u4e86<\/span><\/span>\u7ecf\u9a8c\u56de\u653e\uff08Experience Replay)<\/span><\/span><\/b>\u673a\u5236\uff0c\u901a\u8fc7\u5b58\u50a8\u5e76\u968f\u673a\u91c7\u6837\u5386\u53f2\u7ecf\u9a8c\u6765\u6253\u7834\u6570\u636e\u76f8\u5173\u6027\uff0c\u63d0\u9ad8\u8bad\u7ec3\u7a33\u5b9a\u6027\u3002\u540c\u65f6\uff0c\u91c7\u7528\u76ee\u6807\u7f51\u7edc\u5206\u79bb\u6280\u672f\uff0c\u5b9a\u671f\u51bb\u7ed3\u76ee\u6807Q\u7f51\u7edc\u7684\u53c2\u6570\uff0c\u51cf\u5c11\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u6ce2\u52a8\u3002<\/span><\/span><\/span><\/p>\n


      <\/span>\u5728\u5b9e\u9a8c\u4e2d\uff0cDQN\u572849\u6b3eAtari 2600\u6e38\u620f\u4e0a\u8fdb\u884c\u6d4b\u8bd5\uff0c\u4ec5\u4ee5\u6e38\u620f\u5c4f\u5e55\u50cf\u7d20\u548c\u5206\u6570\u4f5c\u4e3a\u8f93\u5165\uff0c\u4f7f\u7528\u76f8\u540c\u7684\u7f51\u7edc\u67b6\u6784\u548c\u8d85\u53c2\u6570\u3002\u7ed3\u679c\u663e\u793a\uff0cDQN\u572843\u6b3e\u6e38\u620f\u4e2d\u8d85\u8d8a\u4e86\u6b64\u524d\u6240\u6709\u5f3a\u5316\u5b66\u4e60\u65b9\u6cd5\uff0c\u5e76\u572829\u6b3e\u6e38\u620f\u4e2d\u8fbe\u5230\u4eba\u7c7b\u4e13\u4e1a\u6d4b\u8bd5\u5458\u6c34\u5e73\u768475%\u4ee5\u4e0a\u3002\u4f8b\u5982\uff0c\u5728\u201cBreakout\u201d\u4e2d\uff0cDQN\u81ea\u4e3b\u53d1\u73b0\u201c\u4fa7\u8fb9\u6316\u96a7\u9053\u201d\u7684\u9ad8\u5206\u7b56\u7565\uff0c\u6700\u7ec8\u5f97\u5206\u8fdc\u8d85\u4eba\u7c7b\u73a9\u5bb6\uff1b\u5728\u201cSpace Invaders\u201d\u4e2d\uff0cDQN\u5b66\u4f1a\u5ffd\u7565\u65e0\u5173\u89c6\u89c9\u7279\u5f81\uff0c\u4e13\u6ce8\u4e8e\u5173\u952e\u72b6\u6001\u4ef7\u503c\u9884\u6d4b\u3002<\/span><\/span><\/p>\n


      <\/span>DQN\u7684\u6210\u529f\u4e0d\u4ec5\u9a8c\u8bc1\u4e86\u7aef\u5230\u7aef\u5f3a\u5316\u5b66\u4e60\u7684\u53ef\u884c\u6027\uff0c\u8fd8\u63ed\u793a\u4e86\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u4e0e\u751f\u7269\u5b66\u4e60\u673a\u5236\u7684\u76f8\u4f3c\u6027\uff0c\u5982\u7ecf\u9a8c\u56de\u653e\u4e0e\u6d77\u9a6c\u4f53\u8bb0\u5fc6\u91cd\u653e\u7684\u7c7b\u6bd4\u3002\u7136\u800c\uff0cDQN\u4ecd\u5b58\u5728\u8bad\u7ec3\u6570\u636e\u9700\u6c42\u5927\u3001\u5bf9\u8d85\u957f\u7a0b\u89c4\u5212\u4efb\u52a1\uff08\u5982\u201cMontezuma\u2019s Revenge\u201d\u8868\u73b0\u6709\u9650\u7b49\u6311\u6218\u3002\u5c3d\u7ba1\u5982\u6b64\uff0cDQN\u4e3a\u540e\u7eed\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u7814\u7a76\u5960\u5b9a\u4e86\u57fa\u7840\uff0c\u63a8\u52a8\u4e86\u4ece\u6e38\u620fAI\u5230\u673a\u5668\u4eba\u63a7\u5236\u7b49\u9886\u57df\u7684\u5e7f\u6cdb\u5e94\u7528\u3002<\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1.4.1 \u63d0\u51fa\u7684\u6a21\u578b<\/span><\/span><\/span><\/h4>\n


      <\/span><\/p>\n

      <\/section>\n

      \u6a21\u578b\u793a\u610f\u56fe\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n


      <\/span><\/span><\/p>\n

      \u8be5\u793a\u610f\u56fe\u5c55\u793a\u4e86\u6df1\u5ea6Q\u7f51\u7edc\uff08DQN\uff09\u7684\u5178\u578b\u67b6\u6784\u8bbe\u8ba1\uff0c\u76f4\u89c2\u5448\u73b0\u4e86\u4ece\u539f\u59cb\u6e38\u620f\u753b\u9762\u5230\u63a7\u5236\u6307\u4ee4\u7684\u7aef\u5230\u7aef\u5904\u7406\u6d41\u7a0b\u3002\u7cfb\u7edf\u91c7\u7528\u591a\u7ea7\u7279\u5f81\u63d0\u53d6\u67b6\u6784\uff0c\u5de6\u4fa7\u8f93\u5165\u5c42\u63a5\u6536\u9884\u5904\u7406\u540e\u7684\u6e38\u620f\u753b\u9762\uff0884\u00d784\u50cf\u7d20\uff0c4\u5e27\u5806\u53e0\uff09\uff0c\u901a\u8fc7\u4e24\u4e2a\u5377\u79ef\u5c42\uff08Convolution\uff09\u8fdb\u884c\u7a7a\u95f4\u7279\u5f81\u63d0\u53d6\uff0c\u5176\u4e2d\u7f51\u683c\u72b6\u7ed3\u6784\u8868\u793a\u5377\u79ef\u6838\u5728\u7a7a\u95f4\u7ef4\u5ea6\u4e0a\u7684\u7279\u5f81\u6620\u5c04\u8fc7\u7a0b\u3002\u968f\u540e\u7ecf\u7531\u4e24\u4e2a\u5168\u8fde\u63a5\u5c42\uff08Fully connected\uff09\u5b9e\u73b0\u9ad8\u7ea7\u7279\u5f81\u7684\u62bd\u8c61\u4e0e\u7ec4\u5408\uff0c\u6700\u7ec8\u8f93\u51fa\u5c42\u4e0e\u6e38\u620f\u624b\u67c4\u63a7\u5236\u6307\u4ee4\u76f4\u63a5\u5173\u8054\uff0c\u4f5c\u4e3a\u6839\u636e\u5b66\u5230\u7684\u52a8\u4f5c\u4ef7\u503cQ\u6765\u9009\u62e9\u52a8\u4f5c\u3002\u8be5\u67b6\u6784\u8bbe\u8ba1\u5145\u5206\u4f53\u73b0\u4e86DQN\u76f4\u63a5\u4ece\u539f\u59cb\u50cf\u7d20\u5b66\u4e60\u63a7\u5236\u7b56\u7565\u7684\u6838\u5fc3\u601d\u60f3\uff0c\u901a\u8fc7\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\u7684\u7279\u5f81\u63d0\u53d6\u80fd\u529b\u4e0e\u5f3a\u5316\u5b66\u4e60\u7684\u51b3\u7b56\u673a\u5236\u76f8\u7ed3\u5408\uff0c\u5b9e\u73b0\u4e86\u4ece\u611f\u77e5\u5230\u884c\u52a8\u7684\u5b8c\u6574\u6620\u5c04\u3002<\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1.4.2 \u6a21\u578b\u7684\u7ec6\u8282<\/span><\/span><\/span><\/h4>\n


      <\/span><\/span><\/p>\n

      \u8be5\u7814\u7a76\u5728\u6a21\u578b\u8bbe\u8ba1\u4e0a\u4f53\u73b0\u4e86\u663e\u8457\u7684\u666e\u9002\u6027\u7279\u5f81\uff0c\u91c7\u7528\u5355\u4e00\u7f51\u7edc\u67b6\u6784\u5904\u7406\u6240\u6709Atari\u6e38\u620f\u4efb\u52a1\uff0c\u4ec5\u9700\u8f93\u5165\u539f\u59cb\u50cf\u7d20\u3001\u52a8\u4f5c\u7a7a\u95f4\u548c\u751f\u547d\u503c\u7b49\u57fa\u672c\u5148\u9a8c\u77e5\u8bc6\u3002\u5728\u7b97\u6cd5\u5b9e\u73b0\u65b9\u9762\uff0c\u521b\u65b0\u6027\u5730\u5f15\u5165\u4e86\u7ecf\u9a8c\u56de\u653e\u673a\u5236\uff08experience replay\uff09\uff0c\u901a\u8fc7\u5faa\u73af\u7f13\u51b2\u533a\u5b58\u50a8\u5386\u53f2\u8f6c\u79fb\u6837\u672c<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>)<\/span><\/span>\uff0c\u5e76\u91c7\u7528\u5747\u5300\u968f\u673a\u91c7\u6837\u65b9\u5f0f\u62bd\u53d6\u5c0f\u6279\u91cf\uff08minibatch size=32\uff09\u8fdb\u884c\u8bad\u7ec3\u3002\u8fd9\u79cd\u8bbe\u8ba1\u6709\u6548\u6253\u7834\u4e86\u6570\u636e\u95f4\u7684\u65f6\u95f4\u76f8\u5173\u6027\uff0c\u663e\u8457\u63d0\u9ad8\u4e86\u6837\u672c\u5229\u7528\u7387\u548c\u8bad\u7ec3\u7a33\u5b9a\u6027\u3002<\/span><\/span><\/span><\/p>\n


      <\/span>\u5728\u5956\u52b1\u51fd\u6570\u8bbe\u8ba1\u65b9\u9762\uff0c\u7814\u7a76\u8005\u91c7\u7528\u4e86\u5f52\u4e00\u5316\u5904\u7406\u7b56\u7565\uff0c\u5c06\u6b63\u5411\u5956\u52b1\u8bbe\u4e3a+1\uff0c\u8d1f\u5411\u5956\u52b1\u8bbe\u4e3a-1\uff0c\u5176\u4f59\u60c5\u51b5\u4e3a0\uff0c\u8fd9\u79cd\u6807\u51c6\u5316\u65b9\u6cd5\u6709\u6548\u63a7\u5236\u4e86\u8bef\u5dee\u8303\u56f4\u5e76\u5b9e\u73b0\u4e86\u8de8\u6e38\u620f\u7684\u7edf\u4e00\u8bad\u7ec3\u901f\u5ea6\u3002\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u4e0e\u5956\u52b1\u5f52\u4e00\u5316\u7684\u534f\u540c\u4f5c\u7528\uff0c\u4f7f\u5f97\u7b97\u6cd5\u80fd\u591f\u5728\u4e0d\u540c\u6e38\u620f\u95f4\u4fdd\u6301\u4e00\u81f4\u7684\u6027\u80fd\u8868\u73b0\u3002<\/span><\/span><\/p>\n


      <\/span>\u9488\u5bf9\u5927\u89c4\u6a21\u8bad\u7ec3\u6570\u636e\u7684\u9ad8\u5197\u4f59\u7279\u6027\uff0c\u7814\u7a76\u91c7\u7528\u4e86<\/span><\/span>RMSProp\u4f18\u5316\u7b97\u6cd5<\/span><\/span><\/b>\uff0c\u5176\u6838\u5fc3\u673a\u5236\u662f\u901a\u8fc7\u5c06\u68af\u5ea6\u9664\u4ee5\u5176\u8fd1\u671f\u5e45\u5ea6\u7684\u6ed1\u52a8\u5e73\u5747\u503c\u6765\u8c03\u6574\u5b66\u4e60\u7387\u3002\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u5f15\u5165\u8870\u51cf\u7cfb\u6570<\/span><\/span>\u03f5<\/span><\/span>\uff0c\u91c7\u7528\u03f5-greedy\u7b56\u7565\u8fdb\u884c\u63a2\u7d22-\u5229\u7528\u6743\u8861\uff1a\u5728\u524d1\u767e\u4e07\u8bad\u7ec3\u6b65\u4e2d\uff0c<\/span><\/span>\u03f5<\/span><\/span>\u503c\u4ece1\u7ebf\u6027\u8870\u51cf\u81f30.1\uff0c\u4e4b\u540e\u4fdd\u6301\u6052\u5b9a\u3002\u8fd9\u79cd\u52a8\u6001\u8c03\u6574\u7b56\u7565\u4e0e\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u76f8\u7ed3\u5408\uff0c\u6709\u6548\u907f\u514d\u4e86\u8fc7\u65e9\u6536\u655b\u95ee\u9898\uff0c\u7279\u522b\u9002\u5408\u5904\u7406\u975e\u5e73\u7a33\u76ee\u6807\u51fd\u6570\u3002\u503c\u5f97\u6ce8\u610f\u7684\u662f\uff0c\u7ecf\u9a8c\u56de\u653e\u7f13\u51b2\u533a\u7684\u5bb9\u91cf\u8bbe\u7f6e\u4e3a1,000,000\u4e2a\u6837\u672c\uff0c\u5e76\u5728\u8bad\u7ec3\u524d\u9884\u586b\u514550,000\u5e27\u6570\u636e\u4ee5\u786e\u4fdd\u521d\u59cb\u91c7\u6837\u8d28\u91cf\u3002<\/span><\/span><\/span><\/p>\n


      <\/span>\u4e3a\u63d0\u9ad8\u8ba1\u7b97\u6548\u7387\uff0c\u7814\u7a76\u8005\u5bf9\u6240\u6709\u6e38\u620f\u91c7\u7528\u4e86\u8df3\u5e27\u6280\u672f\uff08\u6bcf4\u5e27\u5904\u74061\u5e27\uff09\uff0c\u5728\u4fdd\u8bc1\u6027\u80fd\u65e0\u660e\u663e\u4e0b\u964d\u7684\u524d\u63d0\u4e0b\u663e\u8457\u63d0\u5347\u4e86\u8bad\u7ec3\u901f\u5ea6\u3002\u8fd9\u79cd\u65f6\u95f4\u62bd\u8c61\u65b9\u6cd5\u4e0e\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u534f\u540c\u5de5\u4f5c\uff0c\u4f7f\u5f97\u6837\u672c\u590d\u6742\u5ea6\u76f8\u6bd4\u5728\u7ebf\u7b56\u7565\u964d\u4f4e\u4e86\u7ea610\u500d\u3002\u5b9e\u9a8c\u7ed3\u679c\u8868\u660e\uff0c\u79fb\u9664\u7ecf\u9a8c\u56de\u653e\u4f1a\u5bfc\u81f4\u6027\u80fd\u4e0b\u964d58.7%\uff0c\u5145\u5206\u9a8c\u8bc1\u4e86\u8be5\u673a\u5236\u7684\u5fc5\u8981\u6027\u3002\u8fd9\u4e9b\u521b\u65b0\u8bbe\u8ba1\u5171\u540c\u6784\u6210\u4e86DQN\u7b97\u6cd5\u7684\u6838\u5fc3\u6846\u67b6\uff0c\u4e3a\u540e\u7eed\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u7814\u7a76\u5960\u5b9a\u4e86\u91cd\u8981\u57fa\u7840\u3002<\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1.4.3 \u5177\u4f53\u7b97\u6cd5\u53ca\u5176\u8bad\u7ec3\u6d41\u7a0b<\/span><\/span><\/span><\/h4>\n


      <\/span><\/p>\n

      <\/section>\n

      \u7b97\u6cd5\u6d41\u7a0b\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n


      <\/span><\/p>\n

      <\/section>\n

      \u7b97\u6cd5\u6d41\u7a0b\u793a\u610f\u56fe\uff0c\u5f15\u81ea<\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n


      <\/span><\/p>\n

      1. \u521d\u59cb\u5316\u9636\u6bb5<\/span><\/span><\/p>\n

        \n
      • \n
        \u56de\u653e\u8bb0\u5fc6\u5e93(Replay Memory)\uff1a\u521b\u5efa\u5bb9\u91cf\u4e3a<\/span><\/span>N<\/span><\/span>\u7684\u7ecf\u9a8c\u5b58\u50a8\u5e93<\/span><\/span>D<\/span><\/span>\uff0c\u7528\u4e8e\u7f13\u5b58\u5386\u53f2\u8f6c\u79fb\u6570\u636e\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
      • \n
        \u52a8\u4f5c-\u4ef7\u503c\u51fd\u6570\u7f51\u7edc\uff1a<\/span><\/span><\/section>\n<\/li>\n
          \n
        • \n
          \u4e3b\u7f51\u7edc<\/span><\/span>Q<\/span><\/span>\uff1a\u968f\u673a\u521d\u59cb\u5316\u7f51\u7edc\u53c2\u6570<\/span><\/span>\u03b8<\/span><\/span>\uff0c\u8d1f\u8d23\u5b9e\u65f6\u52a8\u4f5c\u9009\u62e9\u548c\u4ef7\u503c\u4f30\u8ba1\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
        • \n
          \u76ee\u6807\u7f51\u7edc<\/span><\/span>Q<\/span><\/span>\u2212<\/span><\/span><\/sup>\uff1a\u521d\u59cb\u53c2\u6570<\/span><\/span>\u03b8<\/span><\/span>\u2212<\/span><\/span><\/sup>\uff0c\u7528\u4e8e\u7a33\u5b9a\u5316\u8ba1\u7b97\u76ee\u6807Q\u503c\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
        • \n
          \u8d85\u53c2\u6570\u8bbe\u5b9a\uff1a\u8bbe\u7f6e\u8bad\u7ec3\u603b\u56de\u5408\u6570M\u3001\u5355\u56de\u5408\u6700\u5927\u6b65\u6570T\u3001\u76ee\u6807\u7f51\u7edc\u540c\u6b65\u9891\u7387<\/span><\/span>C<\/span><\/span>\u3001\u6298\u6263\u56e0\u5b50<\/span><\/span>\u03b3<\/span><\/span>\u3001\u63a2\u7d22\u7387<\/span><\/span>\u03f5<\/span><\/span>\u7b49\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n<\/ul>\n<\/ul>\n


          <\/span>2. \u56de\u5408\uff08Episode\uff09\u5faa\u73af \u5bf9\u6bcf\u4e2a\u56de\u5408episode=1,2,…,<\/span><\/span>M<\/span><\/span>\u6267\u884c\uff1a \u72b6\u6001\u9884\u5904\u7406\uff1a\u521d\u59cb\u5316\u73af\u5883\u89c2\u6d4b<\/span><\/span>x<\/span><\/span>1<\/span><\/span><\/sub>,\u901a\u8fc7\u9884\u5904\u7406\u51fd\u6570<\/span><\/span>\u03d5<\/span><\/span>\u751f\u6210\u521d\u59cb\u72b6\u6001\u8868\u5f81<\/span><\/span>\u03d5<\/span><\/span>1<\/span><\/span><\/sub>=<\/span><\/span>\u03d5<\/span><\/span>(<\/span><\/span>s<\/span><\/span>1<\/span><\/span><\/sub>)<\/span><\/span>\u3002<\/span><\/span><\/span><\/p>\n


          <\/span>3. \u65f6\u95f4\u6b65\uff08Timestep\uff09\u5faa\u73af \u5bf9\u6bcf\u4e2a\u65f6\u95f4\u6b65t=1,2,…,T\u6267\u884c\uff1a<\/span><\/span><\/p>\n

            \n
          • \n
            \u52a8\u4f5c\u9009\u62e9\uff1a<\/span><\/span><\/section>\n<\/li>\n
              \n
            • \n
              \u63a2\u7d22\u7b56\u7565\uff1a\u4ee5\u6982\u7387\u03f5\u968f\u673a\u9009\u62e9\u52a8\u4f5c<\/span><\/span>a<\/span><\/span>t<\/span><\/span>,\u4fc3\u8fdb\u73af\u5883\u63a2\u7d22\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
            • \n
              \u5229\u7528\u7b56\u7565\uff1a\u5426\u5219\uff0c\u6839\u636e\u4e3b\u7f51\u7edc\u9009\u62e9\u6700\u4f18\u52a8\u4f5c\uff1a<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>=<\/span><\/span>a<\/span><\/span>r<\/span><\/span>g<\/span><\/span>m<\/span><\/span>a<\/span><\/span>x<\/span><\/span>a<\/span><\/span><\/sub>Q<\/span><\/span>(<\/span><\/span>\u03d5<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>)<\/span><\/span>,<\/span><\/span>a<\/span><\/span>;<\/span><\/span>\u03b8<\/span><\/span>)<\/span><\/span><\/span><\/section>\n<\/li>\n<\/ul>\n
            • \n
              \u6267\u884c\u4e0e\u73af\u5883\u4ea4\u4e92\uff1a<\/span><\/span><\/section>\n<\/li>\n
                \n
              • \n
                \u6267\u884c\u52a8\u4f5ca<\/span><\/span>t<\/span><\/span><\/sub>\uff0c\u83b7\u53d6\u5956\u52b1r<\/span><\/span>t<\/span><\/span><\/sub>\u548c\u4e0b\u4e00\u5e27\u56fe\u50cf<\/span><\/span>x<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
              • \n
                \u9884\u5904\u7406\u4e0b\u4e00\u72b6\u6001<\/span><\/span>\u03d5<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>=<\/span><\/span>\u03d5<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>)<\/span><\/span>\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n<\/ul>\n
              • \n
                \u5b58\u50a8\u8f6c\u79fb\u6570\u636e\uff1a\u5c06\u8f6c\u79fb\u5143\u7ec4<\/span><\/span>(<\/span><\/span>\u03d5<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>\u03d5<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>)<\/span><\/span>\u5b58\u5165\u56de\u653e\u8bb0\u5fc6\u5e93<\/span><\/span>D<\/span><\/span>\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
              • \n
                \u7ecf\u9a8c\u56de\u653e\u4e0e\u7f51\u7edc\u66f4\u65b0\uff1a<\/span><\/span><\/section>\n<\/li>\n
                  \n
                • \n
                  \u968f\u673a\u91c7\u6837\uff1a\u4ece<\/span><\/span>D<\/span><\/span>\u4e2d\u5747\u5300\u62bd\u53d6\u5c0f\u6279\u91cf\u8f6c\u79fb<\/span><\/span>(<\/span><\/span>\u03d5<\/span><\/span>j<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>j<\/span><\/span><\/sub>,<\/span><\/span>r<\/span><\/span>j<\/span><\/span><\/sub>,<\/span><\/span>\u03d5<\/span><\/span>j<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>)<\/span><\/span>\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n
                • \n
                  \u76ee\u6807<\/span><\/span>Q<\/span><\/span>\u503c\u8ba1\u7b97\uff1a <\/span><\/span>y<\/span><\/span>j<\/span><\/span><\/sub>=<\/span><\/span>r<\/span><\/span>j<\/span><\/span><\/sub>+<\/span><\/span>\u03b3<\/span><\/span>m<\/span><\/span>a<\/span><\/span>x<\/span><\/span>a<\/span><\/span><\/sub>Q<\/span><\/span>\u2212<\/span><\/span><\/sup>(<\/span><\/span>\u03d5<\/span><\/span>j<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span><\/span><\/section>\n<\/li>\n
                • \n
                  \u68af\u5ea6\u4e0b\u964d: <\/span><\/span>\u2207<\/span><\/span>\u03b8<\/span><\/span><\/sub>1<\/span><\/span>\/<\/span><\/span>|<\/span><\/span>B<\/span><\/span>|<\/span><\/span>\u2211<\/span><\/span>(<\/span><\/span>y<\/span><\/span>j<\/span><\/span><\/sub>\u2212<\/span><\/span>Q<\/span><\/span>(<\/span><\/span>\u03d5<\/span><\/span>j<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>j<\/span><\/span><\/sub>;<\/span><\/span>\u03b8<\/span><\/span>)<\/span><\/span>)<\/span><\/span>2<\/span><\/span><\/sup> \uff0cB\u4e3abatch\u6570\u636e<\/span><\/span><\/span><\/section>\n<\/li>\n<\/ul>\n
                • \n
                  \u76ee\u6807\u7f51\u7edc\u540c\u6b65\uff1a\u6bcf\u9694<\/span><\/span>C<\/span><\/span>\u6b65\u5c06\u4e3b\u7f51\u7edc\u53c2\u6570<\/span><\/span>\u03b8<\/span><\/span>\u590d\u5236\u81f3\u76ee\u6807\u7f51\u7edc<\/span><\/span>Q<\/span><\/span>\u2212<\/span><\/span><\/sup>\uff0c\u5373<\/span><\/span>\u03b8<\/span><\/span>\u2212<\/span><\/span><\/sup>\u2190<\/span><\/span>\u03b8<\/span><\/span>\u3002<\/span><\/span><\/span><\/section>\n<\/li>\n<\/ul>\n


                  <\/span><\/p>\n


                  <\/span><\/span><\/h2>\n

                  \n
                  \n
                  \n
                  \n
                  \n

                  <\/span><\/section>\n

                  <\/span><\/section>\n
                  \n
                  <\/section>\n<\/section>\n<\/section>\n
                  \n

                  2. Q-Learning \u57fa\u7840<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n

                  \n

                  <\/span><\/section>\n

                  <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n


                  <\/span><\/p>\n

                  2.1 \u9a6c\u5c14\u53ef\u592b\u51b3\u7b56\u8fc7\u7a0b\uff08MDP\uff09\u7b80\u4ecb<\/span><\/span><\/span><\/h3>\n


                  <\/span><\/span><\/h3>\n

                  \u9a6c\u5c14\u53ef\u592b\u51b3\u7b56\u8fc7\u7a0b\uff08Markov Decision Process\uff0c\u7b80\u79f0 MDP\uff09\u662f\u5f3a\u5316\u5b66\u4e60\u7684\u6838\u5fc3\u6570\u5b66\u6846\u67b6\uff0c\u7528\u4e8e\u5efa\u6a21\u667a\u80fd\u4f53\u5728\u4e0d\u786e\u5b9a\u73af\u5883\u4e2d\u901a\u8fc7\u8bd5\u9519\u5b66\u4e60\u6700\u4f18\u7b56\u7565\u7684\u8fc7\u7a0b\u3002\u5b83\u5728\u9a6c\u5c14\u53ef\u592b\u8fc7\u7a0b\u7684\u57fa\u7840\u4e0a\u5f15\u5165\u4e86\u201c\u52a8\u4f5c\u201d\u548c\u201c\u5956\u52b1\u201d\u673a\u5236\uff0c\u5e7f\u6cdb\u5e94\u7528\u4e8e\u4eba\u5de5\u667a\u80fd\u3001\u63a7\u5236\u7cfb\u7edf\u3001\u7ecf\u6d4e\u5b66\u7b49\u9886\u57df\u3002<\/span><\/span><\/p>\n


                  <\/span><\/p>\n

                  <\/section>\n

                  \u9a6c\u5c14\u53ef\u592b\u51b3\u7b56\u8fc7\u7a0b<\/span><\/span><\/p>\n


                  <\/span><\/p>\n

                  2.1.1 MDP \u7684\u6838\u5fc3\u601d\u60f3<\/span><\/span><\/span><\/h4>\n

                  \u5728 MDP \u4e2d\uff0c\u667a\u80fd\u4f53\u4e0e\u73af\u5883\u7684\u4ea4\u4e92\u8fc7\u7a0b\u5982\u53f3\u56fe\u6240\u793a\uff1a<\/span><\/span><\/p>\n

                  (1)\u667a\u80fd\u4f53\u89c2\u5bdf\u5f53\u524d\u72b6\u6001<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\u3002<\/span><\/span><\/span><\/p>\n

                  (2)\u6839\u636e\u7b56\u7565 <\/span><\/span>\u03c0<\/span><\/span>\uff0c\u9009\u62e9\u4e00\u4e2a\u52a8\u4f5c<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>\u3002<\/span><\/span><\/span><\/p>\n

                  (3)\u73af\u5883\u6839\u636e\u72b6\u6001\u8f6c\u79fb\u6982\u7387<\/span><\/span>P<\/span><\/span> \uff0c\u8f6c\u79fb\u5230\u4e0b\u4e00\u4e2a\u72b6\u6001 <\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\uff0c\u5e76\u7ed9\u4e88\u667a\u80fd\u4f53\u5956\u52b1 <\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>\u3002<\/span><\/span><\/span><\/p>\n

                  (4)\u667a\u80fd\u4f53\u6839\u636e\u83b7\u5f97\u7684\u5956\u52b1\u548c\u65b0\u7684\u72b6\u6001\uff0c\u66f4\u65b0\u7b56\u7565 <\/span><\/span>\u03c0<\/span><\/span>\uff0c\u4ee5\u671f\u83b7\u5f97\u66f4\u9ad8\u7684\u7d2f\u8ba1\u5956\u52b1\u3002<\/span><\/span><\/span><\/p>\n

                  \u8fd9\u4e00\u8fc7\u7a0b\u6301\u7eed\u8fdb\u884c\uff0c\u667a\u80fd\u4f53\u901a\u8fc7\u4e0d\u65ad\u4e0e\u73af\u5883\u4ea4\u4e92\uff0c\u5b66\u4e60\u6700\u4f18\u7b56\u7565\u3002<\/span><\/span><\/p>\n


                  <\/span><\/p>\n

                  2.1.2 \u7b56\u7565\u4e0e\u4ef7\u503c\u51fd\u6570<\/span><\/span><\/span><\/h4>\n

                  \u5728MDP\u4e2d\uff0c\u7b56\u7565\u4e0e\u4ef7\u503c\u51fd\u6570\u662f\u5f3a\u5316\u5b66\u4e60\u7684\u6838\u5fc3\u6982\u5ff5\uff0c\u5b83\u4eec\u5171\u540c\u6784\u6210\u4e86\u667a\u80fd\u4f53\u5b66\u4e60\u4e0e\u51b3\u7b56\u7684\u7406\u8bba\u57fa\u7840\u3002\u4ee5\u4e0b\u662f\u5173\u4e8e\u7b56\u7565\u4e0e\u4ef7\u503c\u51fd\u6570\u7684\u5b9a\u4e49\uff1a<\/span><\/span><\/p>\n

                   \u2022 <\/span><\/span>\u7b56\u7565 <\/span><\/span>\u03c0<\/span><\/span>(<\/span><\/span>a<\/span><\/span>|<\/span><\/span>s<\/span><\/span>)\uff1a\u5728\u72b6\u6001 s\u4e0b\u9009\u62e9\u52a8\u4f5c a \u7684\u6982\u7387\u5206\u5e03\u3002<\/span><\/span><\/span><\/p>\n

                   \u2022 <\/span><\/span>\u72b6\u6001\u503c\u51fd\u6570<\/span><\/span>V<\/span><\/span>\u03c0<\/span><\/span><\/sub>(<\/span><\/span>s<\/span><\/span>)\uff1a\u5728\u7b56\u7565 \u03c0 \u4e0b\uff0c\u4ece\u72b6\u6001 s \u5f00\u59cb\uff0c\u671f\u671b\u83b7\u5f97\u7684\u7d2f\u8ba1\u5956\u52b1\u3002\u5176\u5b9a\u4e49\u4e3a\uff1a<\/span><\/span><\/span><\/p>\n

                  <\/span><\/span><\/p>\n

                  \u5176\u4e2d<\/span><\/span>\u03b3<\/span><\/span>\u2208<\/span><\/span>[<\/span><\/span>0<\/span><\/span>,<\/span><\/span>1<\/span><\/span>)<\/span><\/span>\u4e3a\u6298\u6263\u56e0\u5b50\uff0c\u63a7\u5236\u672a\u6765\u5956\u52b1\u7684\u6743\u91cd\uff1b<\/span><\/span>r<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\u4e3a\u4ece<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\u5230<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\u83b7\u5f97\u7684\u5373\u65f6\u5956\u52b1\u3002<\/span><\/span><\/span><\/p>\n

                   \u2022 <\/span><\/span>\u52a8\u4f5c\u503c\u51fd\u6570 <\/span><\/span>Q<\/span><\/span>\u03c0<\/span><\/span><\/sub>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>)\uff1a\u5728\u7b56\u7565 \u03c0 \u4e0b\uff0c\u4ece\u72b6\u6001 s \u5f00\u59cb\uff0c\u91c7\u53d6\u52a8\u4f5c a \u540e\uff0c\u671f\u671b\u83b7\u5f97\u7684\u7d2f\u8ba1\u5956\u52b1\u3002\u5176\u5b9a\u4e49\u4e3a\uff1a<\/span><\/span><\/span><\/p>\n

                  <\/span><\/span><\/p>\n

                  \u8fd9\u4e9b\u51fd\u6570\u7528\u4e8e\u8bc4\u4f30\u7b56\u7565\u7684\u4f18\u52a3\uff0c\u6307\u5bfc\u7b56\u7565\u7684\u6539\u8fdb\u3002<\/span><\/span><\/span><\/p>\n


                  <\/span><\/p>\n

                  2.2 Q \u503c\u51fd\u6570\u7684\u5b9a\u4e49\u4e0e\u66f4\u65b0\u516c\u5f0f<\/span><\/span><\/span><\/h3>\n


                  <\/span><\/span><\/h4>\n

                  2.2.1 \u5b9a\u4e49<\/span><\/span><\/span><\/h4>\n

                  Q \u503c\u51fd\u6570 <\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span><\/b> \u8868\u793a\u667a\u80fd\u4f53\u5728\u72b6\u6001 <\/span><\/span>s<\/span><\/span>\u4e0b\u91c7\u53d6\u52a8\u4f5c <\/span><\/span>a<\/span><\/span> \u540e\uff0c\u9075\u5faa\u67d0\u4e00\u7b56\u7565\u6240\u80fd\u83b7\u5f97\u7684\u671f\u671b\u7d2f\u79ef\u5956\u52b1\u3002<\/span><\/span><\/span><\/p>\n


                  <\/span><\/span><\/p>\n

                  2.2.2 \u66f4\u65b0\u516c\u5f0f<\/span><\/span><\/span><\/h4>\n

                  Q-Learning \u7b97\u6cd5\u4f7f\u7528\u4ee5\u4e0b\u66f4\u65b0\u516c\u5f0f\u6765\u8fed\u4ee3 Q \u503c\uff1a<\/span><\/span><\/p>\n

                  <\/section>\n

                  \u5176\u4e2d\uff0c\u7b26\u53f7\u542b\u4e49\u5982\u4e0b\uff1a<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5f53\u524d\u72b6\u6001\uff1b<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5f53\u524d\u52a8\u4f5c\uff1b<\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>\uff1a\u5373\u65f6\u5956\u52b1\uff1b<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>\uff1a\u6267\u884c\u52a8\u4f5c\u540e\u7684\u65b0\u72b6\u6001\uff1b<\/span><\/span>\u03b1<\/span><\/span>\u2208<\/span><\/span>(<\/span><\/span>0<\/span><\/span>,<\/span><\/span>1<\/span><\/span>]<\/span><\/span>\uff1a\u5b66\u4e60\u7387\uff1b<\/span><\/span>\u03b3<\/span><\/span>\u2208<\/span><\/span>[<\/span><\/span>0<\/span><\/span>,<\/span><\/span>1<\/span><\/span>)<\/span><\/span>\uff1a\u6298\u6263\u56e0\u5b50\u3002<\/span><\/span><\/span><\/p>\n


                  <\/span><\/span><\/p>\n

                  \u6b64\u516c\u5f0f\u57fa\u4e8e\u8d1d\u5c14\u66fc\u65b9\u7a0b\u7684\u65f6\u5e8f\u5dee\u5206\uff08TD\uff09\u66f4\u65b0\u601d\u60f3\uff0c\u9010\u6b65\u8c03\u6574 <\/span><\/span>Q<\/span><\/span> \u503c\u51fd\u6570\u4ee5\u903c\u8fd1\u6700\u4f18\u7b56\u7565\u3002<\/span><\/span><\/span><\/p>\n


                  <\/span><\/p>\n

                  2.3 \u03b5-greedy \u7b56\u7565\u4e0e\u63a2\u7d22-\u5229\u7528\u6743\u8861<\/span><\/span><\/span><\/h3>\n


                  <\/span><\/span><\/h3>\n

                  Q-learning \u901a\u5e38\u9762\u4e34\u201c\u63a2\u7d22\uff08exploration\uff09\u201d\u4e0e\u201c\u5229\u7528\uff08exploitation\uff09\u201d\u4e4b\u95f4\u7684\u6743\u8861\u95ee\u9898\u3002\u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e00\u95ee\u9898\uff0c\u901a\u5e38\u91c7\u7528 \u03b5-greedy \u7b56\u7565\u3002\u03b5-greedy \u7b56\u7565\u662f\u4e00\u79cd\u7b80\u5355\u800c\u6709\u6548\u7684\u52a8\u4f5c\u9009\u62e9\u7b56\u7565\uff0c\u65e8\u5728\u5e73\u8861\u63a2\u7d22\u4e0e\u5229\u7528\u3002<\/span><\/span><\/p>\n

                    \n
                  • \u5229\u7528\uff08Exploitation\uff09\uff1a\u4ee5 1\u2212\u03f5 \u7684\u6982\u7387\u9009\u62e9\u5f53\u524d\u72b6\u6001\u4e0b Q \u503c\u6700\u5927\u7684\u52a8\u4f5c\uff0c\u5373\u9009\u62e9\u5df2\u77e5\u7684\u6700\u4f73\u52a8\u4f5c\u3002<\/span><\/span><\/span><\/li>\n
                  • \u63a2\u7d22\uff08Exploration\uff09\uff1a\u4ee5 \u03f5 \u7684\u6982\u7387\u968f\u673a\u9009\u62e9\u4e00\u4e2a\u52a8\u4f5c\uff0c\u4ece\u800c\u63a2\u7d22\u65b0\u7684\u53ef\u80fd\u6027\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n


                    <\/span><\/p>\n

                    \u5176\u4e2d\uff0c\u03f5 \u662f\u4e00\u4e2a\u4ecb\u4e8e 0 \u548c 1 \u4e4b\u95f4\u7684\u53c2\u6570\uff0c\u63a7\u5236\u63a2\u7d22\u7684\u9891\u7387\u3002\u8f83\u5927\u7684 \u03f5 \u503c\u9f13\u52b1\u66f4\u591a\u7684\u63a2\u7d22\uff0c\u800c\u8f83\u5c0f\u7684 \u03f5 \u503c\u5219\u503e\u5411\u4e8e\u66f4\u591a\u7684\u5229\u7528\u3002<\/span><\/span><\/p>\n


                    <\/span>\u5728\u5b9e\u9645\u5e94\u7528\u4e2d\uff0c\u901a\u5e38\u4f1a\u968f\u7740\u8bad\u7ec3\u7684\u8fdb\u884c\u9010\u6b65\u51cf\u5c0f \u03f5 \u503c\uff0c\u4ece\u800c\u5b9e\u73b0\u4ee5\u4e0b\u76ee\u6807\uff1a<\/span><\/span><\/p>\n

                      \n
                    • \u521d\u671f\u9636\u6bb5\uff1a\u8bbe\u7f6e\u8f83\u5927\u7684 \u03f5 \u503c\uff0c\u9f13\u52b1\u667a\u80fd\u4f53\u8fdb\u884c\u5145\u5206\u7684\u63a2\u7d22\u3002<\/span><\/span><\/span><\/li>\n
                    • \u540e\u671f\u9636\u6bb5\uff1a\u9010\u6b65\u51cf\u5c0f \u03f5 \u503c\uff0c\u589e\u52a0\u5229\u7528\u5df2\u77e5\u6700\u4f73\u7b56\u7565\u7684\u9891\u7387\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n


                      <\/span><\/p>\n

                      \u8fd9\u79cd\u7b56\u7565\u6709\u52a9\u4e8e\u667a\u80fd\u4f53\u5728\u5b66\u4e60\u521d\u671f\u83b7\u53d6\u8db3\u591f\u7684\u7ecf\u9a8c\uff0c\u5728\u540e\u671f\u5219\u80fd\u591f\u7a33\u5b9a\u5730\u6267\u884c\u6700\u4f18\u7b56\u7565\u3002<\/span><\/span><\/p>\n


                      <\/span><\/span><\/h3>\n

                      2.4 Q-Learning \u7684\u6536\u655b\u6027<\/span><\/span><\/span><\/h3>\n


                      <\/span><\/p>\n

                      Q-Learning \u6536\u655b\u6027\u7684\u8bc1\u660e\u4e3b\u8981\u57fa\u4e8e\u4ee5\u4e0b\u51e0\u70b9\uff1a<\/span><\/span><\/p>\n


                      <\/span><\/span><\/p>\n

                      \uff081\uff09Bellman \u6700\u4f18\u7b97\u5b50\u662f\u538b\u7f29\u6620\u5c04<\/span><\/span>[2]<\/span><\/span><\/span><\/p>\n

                      Bellman \u6700\u4f18\u7b97\u5b50 B \u5b9a\u4e49\u4e3a\uff1a<\/span><\/span>\"image.png\"<\/span><\/sub>\u5b83\u6ee1\u8db3\uff1a <\/span><\/span>\u2016<\/span><\/span>B<\/span><\/span>Q<\/span><\/span>1<\/span><\/span><\/sub>\u2212<\/span><\/span>B<\/span><\/span>Q<\/span><\/span>2<\/span><\/span><\/sub>\u2016<\/span><\/span>\u221e<\/span><\/span>\u2264<\/span><\/span>\u03b3<\/span><\/span>\u2016<\/span><\/span>Q<\/span><\/span>1<\/span><\/span><\/sub>\u2212<\/span><\/span>Q<\/span><\/span>2<\/span><\/span><\/sub>\u2016<\/span><\/span>\u221e<\/span><\/span> \u5373B\u662f\u4ee5\u6298\u6263\u56e0\u5b50\u03b3\u4e3a\u538b\u7f29\u7387\u7684\u538b\u7f29\u6620\u5c04\u3002<\/span><\/span><\/span><\/p>\n


                      <\/span>\uff082\uff09Banach \u4e0d\u52a8\u70b9\u5b9a\u7406\u4fdd\u8bc1\u552f\u4e00\u56fa\u5b9a\u70b9<\/span><\/span><\/p>\n

                      \u56e0\u4e3aB\u662f\u538b\u7f29\u6620\u5c04\uff0c\u5b58\u5728\u552f\u4e00\u7684\u56fa\u5b9a\u70b9<\/span><\/span>Q<\/span><\/span>\u2217<\/span><\/span><\/sup>\u6ee1\u8db3\uff1a <\/span><\/span>Q<\/span><\/span>\u2217<\/span><\/span><\/sup>=<\/span><\/span>B<\/span><\/span>Q<\/span><\/span>\u2217<\/span><\/span><\/sup> \u8fd9\u5c31\u662f\u6700\u4f18\u72b6\u6001-\u52a8\u4f5c\u503c\u51fd\u6570\u3002<\/span><\/span><\/span><\/p>\n


                      <\/span>\uff083\uff09\u968f\u673a\u903c\u8fd1\u6846\u67b6\u4e2d\u7684\u6536\u655b\u6761\u4ef6<\/span><\/span>[3]<\/span><\/span><\/span><\/p>\n

                      Q-Learning \u901a\u8fc7\u968f\u673a\u91c7\u6837\u548c\u8fed\u4ee3\u903c\u8fd1<\/span><\/span>Q<\/span><\/span>\u2217<\/span><\/span><\/sup>\uff0c\u5728\u5b66\u4e60\u7387\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\u65f6\u6536\u655b\uff1a<\/span><\/span>\"image.png\"<\/span><\/sub>\u540c\u65f6\uff0c\u7b97\u6cd5\u9700\u4fdd\u8bc1\u6bcf\u4e2a\u72b6\u6001-\u52a8\u4f5c\u5bf9 (s,a) \u88ab\u65e0\u9650\u6b21\u8bbf\u95ee\u3002<\/span><\/span><\/span><\/p>\n


                      <\/span>\u6536\u655b\u7684\u5173\u952e\u6761\u4ef6\uff1a<\/span><\/span><\/p>\n

                      \u5145\u5206\u63a2\u7d22\uff1a\u667a\u80fd\u4f53\u5fc5\u987b\u4fdd\u8bc1\u6240\u6709\u5bf9\u65e0\u9650\u6b21\u8bbf\u95ee\uff0c\u901a\u5e38\u901a\u8fc7 \u03b5-greedy \u7b49\u7b56\u7565\u5b9e\u73b0\u3002<\/span><\/span><\/p>\n

                      \u5b66\u4e60\u7387\u8870\u51cf\uff1a\u5b66\u4e60\u7387<\/span><\/span>\u03b1<\/span><\/span>t<\/span><\/span>\u968f\u65f6\u95f4\u8870\u51cf\uff0c\u4f46\u4e0d\u5b9c\u8fc7\u5feb\u3002\u6298\u6263\u56e0\u5b50\u9650\u5236\uff1a\u03b3\u2208[0,1)\uff0c\u4fdd\u8bc1\u672a\u6765\u5956\u52b1\u7684\u5f71\u54cd\u9012\u51cf\u3002<\/span><\/span><\/span><\/p>\n

                      \u5728\u6ee1\u8db3\u9002\u5f53\u7684\u5b66\u4e60\u7387\u8c03\u8282\u3001\u5145\u5206\u63a2\u7d22\u3001\u4ee5\u53ca\u6298\u6263\u56e0\u5b50\u7ea6\u675f\u7684\u60c5\u51b5\u4e0b\uff0cQ-Learning \u7b97\u6cd5\u80fd\u591f\u4ee5\u6982\u7387 1 \u6536\u655b\u5230\u6700\u4f18\u503c\u51fd\u6570<\/span><\/span>Q<\/span><\/span>\u2217<\/span><\/span>\uff0c\u5176\u6570\u5b66\u8868\u8fbe\u4e3a\uff1a<\/span><\/span>\"image.png\"<\/span><\/sub>\u8fd9\u4e3a\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\u7684\u7406\u8bba\u53ef\u9760\u6027\u63d0\u4f9b\u4e86\u575a\u5b9e\u57fa\u7840\u3002<\/span><\/span><\/span><\/p>\n


                      <\/span><\/span><\/p>\n

                      2.5 Q-Learning \u7684\u5c40\u9650\u6027<\/span><\/span><\/span><\/h3>\n


                      <\/span><\/span><\/h3>\n

                      Q-Learning \u5e7f\u6cdb\u5e94\u7528\u4e8e\u591a\u79cd\u51b3\u7b56\u4efb\u52a1\u3002\u5728\u5b9e\u9645\u5e94\u7528\u4e2d\uff0c\u5b83\u7684\u5c40\u9650\u6027\u4e3b\u8981\u4f53\u73b0\u5728\u4ee5\u4e0b\u51e0\u4e2a\u65b9\u9762\uff1a<\/span><\/span><\/p>\n


                      <\/span><\/p>\n

                      \uff081\uff09\u72b6\u6001-\u52a8\u4f5c\u7a7a\u95f4\u7ef4\u5ea6\u707e\u96be\uff08Curse of Dimensionality\uff09\uff1aQ-Learning \u4f7f\u7528 Q \u8868\u6765\u5b58\u50a8\u6bcf\u4e2a\u72b6\u6001-\u52a8\u4f5c\u5bf9\u7684\u4ef7\u503c\u3002\u5f53\u72b6\u6001\u6216\u52a8\u4f5c\u7a7a\u95f4\u975e\u5e38\u5927\u65f6\uff0cQ \u8868\u7684\u5c3a\u5bf8\u4f1a\u6025\u5267\u81a8\u80c0\uff0c\u5bfc\u81f4\u5b58\u50a8\u548c\u66f4\u65b0\u53d8\u5f97\u4e0d\u5207\u5b9e\u9645\u3002<\/span><\/span><\/p>\n


                      <\/span><\/p>\n

                      \u4f8b\u5982\uff0c\u5728\u5177\u6709\u6570\u767e\u4e07\u72b6\u6001\u548c\u52a8\u4f5c\u7684\u73af\u5883\u4e2d\uff0c\u4f20\u7edf\u7684 Q \u8868\u65b9\u6cd5\u53ef\u80fd\u65e0\u6cd5\u6709\u6548\u5904\u7406\u3002\u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\uff0c\u901a\u5e38\u91c7\u7528\u51fd\u6570\u903c\u8fd1\u65b9\u6cd5\uff0c\u5982\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u6216\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\uff0c\u4ee5\u4f30\u8ba1 Q \u503c\uff0c\u4ece\u800c\u907f\u514d\u76f4\u63a5\u4f7f\u7528\u5e9e\u5927\u7684 Q \u8868\u3002<\/span><\/span><\/p>\n


                      <\/span>\uff082\uff09\u5728\u72b6\u6001\u548c\u52a8\u4f5c\u7a7a\u95f4\u8f83\u5927\u7684\u73af\u5883\u4e2d\uff0cQ-Learning \u7684\u5b66\u4e60\u8fc7\u7a0b\u53ef\u80fd\u975e\u5e38\u7f13\u6162\uff0c\u901a\u5e38\u9700\u8981\u5927\u91cf\u7684\u8bad\u7ec3\u56de\u5408\u624d\u80fd\u6536\u655b\u5230\u6700\u4f18\u7b56\u7565\u3002\u8fd9\u79cd\u6162\u6536\u655b\u6027\u4f7f\u5f97 Q-Learning \u5728\u9700\u8981\u5feb\u901f\u5b66\u4e60\u7684\u5e94\u7528\u573a\u666f\u4e2d\u53ef\u80fd\u4e0d\u592a\u9002\u7528\u3002[1]<\/span><\/span><\/p>\n


                      <\/span>\uff083\uff09\u63a2\u7d22\u4e0e\u5229\u7528\u7684\u5e73\u8861\u95ee\u9898\uff1aQ-Learning \u4f9d\u8d56\u4e8e\u63a2\u7d22\uff08exploration\uff09\u548c\u5229\u7528\uff08exploitation\uff09\u4e4b\u95f4\u7684\u5e73\u8861\u3002\u8fc7\u5ea6\u63a2\u7d22\u53ef\u80fd\u5bfc\u81f4\u5b66\u4e60\u6548\u7387\u4f4e\u4e0b\uff0c\u800c\u8fc7\u5ea6\u5229\u7528\u53ef\u80fd\u5bfc\u81f4\u9677\u5165\u5c40\u90e8\u6700\u4f18\u89e3\u3002\u4f8b\u5982\uff0c\u03b5-greedy \u7b56\u7565\u901a\u8fc7\u5f15\u5165\u968f\u673a\u6027\u6765\u5e73\u8861\u63a2\u7d22\u4e0e\u5229\u7528\uff0c\u4f46\u9009\u62e9\u5408\u9002\u7684 \u03b5 \u503c\u4ecd\u7136\u662f\u4e00\u4e2a\u6311\u6218\u3002<\/span><\/span><\/p>\n


                      <\/span>\uff084\uff09\u5bf9\u8d85\u53c2\u6570\u654f\u611f\uff1aQ-Learning \u7684\u6027\u80fd\u5bf9\u5b66\u4e60\u7387\uff08\u03b1\uff09\u3001\u6298\u6263\u56e0\u5b50\uff08\u03b3\uff09\u548c\u63a2\u7d22\u7387\uff08\u03b5\uff09\u7b49\u8d85\u53c2\u6570\u975e\u5e38\u654f\u611f\u3002\u9009\u62e9\u4e0d\u5f53\u7684\u8d85\u53c2\u6570\u53ef\u80fd\u5bfc\u81f4\u5b66\u4e60\u8fc7\u7a0b\u4e0d\u7a33\u5b9a\u6216\u6536\u655b\u5230\u6b21\u4f18\u89e3\u3002\u4f8b\u5982\uff0c\u8fc7\u9ad8\u7684\u5b66\u4e60\u7387\u53ef\u80fd\u5bfc\u81f4 Q \u503c\u66f4\u65b0\u8fc7\u5feb\uff0c\u5bfc\u81f4\u9707\u8361\uff1b\u800c\u8fc7\u4f4e\u7684\u5b66\u4e60\u7387\u53ef\u80fd\u5bfc\u81f4\u5b66\u4e60\u8fc7\u7a0b\u8fc7\u6162\u3002<\/span><\/span><\/p>\n


                      <\/span>\uff085\uff09\u5bf9\u51fd\u6570\u903c\u8fd1\u7684\u7a33\u5b9a\u6027\u95ee\u9898\uff1a\u5f53\u4f7f\u7528\u51fd\u6570\u903c\u8fd1\uff08\u5982\u795e\u7ecf\u7f51\u7edc\uff09\u6765\u4f30\u8ba1 Q \u503c\u65f6\uff0cQ-Learning \u53ef\u80fd\u51fa\u73b0\u4e0d\u7a33\u5b9a\u6216\u53d1\u6563\u7684\u60c5\u51b5\uff0c\u5c24\u5176\u662f\u5728\u975e\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u5668\u7684\u60c5\u51b5\u4e0b\u3002<\/span><\/span><\/p>\n

                      \u8fd9\u79cd\u4e0d\u7a33\u5b9a\u6027\u4e3b\u8981\u6e90\u4e8e\u4f30\u8ba1\u76ee\u6807\u503c\u4e0e\u5f53\u524d Q \u503c\u4e4b\u95f4\u7684\u76f8\u4e92\u4f9d\u8d56\u5173\u7cfb\u3002\u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\uff0c\u63d0\u51fa\u4e86 Double Q-Learning \u7b49\u65b9\u6cd5\uff0c\u901a\u8fc7\u4f7f\u7528\u4e24\u4e2a\u72ec\u7acb\u7684 Q \u503c\u4f30\u8ba1\u5668\u6765\u51cf\u5c11\u8fc7\u4f30\u8ba1\u504f\u5dee\uff0c\u4ece\u800c\u63d0\u9ad8\u7a33\u5b9a\u6027\u3002[2]<\/span><\/span><\/p>\n


                      <\/span><\/p>\n


                      <\/span><\/span><\/h2>\n

                      \n
                      \n
                      \n
                      \n
                      \n

                      <\/span><\/section>\n

                      <\/span><\/section>\n
                      \n
                      <\/section>\n<\/section>\n<\/section>\n
                      \n

                      4. DQN \u7684\u6838\u5fc3\u601d\u60f3\u4e0e\u7ed3\u6784<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n

                      \n

                      <\/span><\/section>\n

                      <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n


                      <\/span><\/p>\n

                      3.1 \u4f7f\u7528\u795e\u7ecf\u7f51\u7edc\u903c\u8fd1 Q \u503c\u51fd\u6570<\/span><\/span><\/span><\/h3>\n


                      <\/span><\/span><\/p>\n

                      DQN\u7684<\/span><\/span>\u2f8f\u4e3a\u503c\u51fd\u6570<\/span><\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>;<\/span><\/span>\u03b8<\/span><\/span>)<\/span><\/span> \u5229\u2f64\u795e\u7ecf\u2f79\u7edc\u903c\u8fd1\uff0c\u5c5e\u4e8e\u2fae\u7ebf\u6027\u903c\u8fd1\u3002\u6b64\u5904\u7684\u503c\u51fd\u6570\u5bf9\u5e94\u7740\u2f00\u7ec4\u53c2\u6570\uff0c\u5728\u795e\u7ecf\u2f79\u7edc\u2fa5\u53c2\u6570\u662f\u6bcf\u5c42\u2f79\u7edc\u7684\u6743\u91cd\uff0c\u7528<\/span><\/span><\/span>\u03b8\u8868\u793a\u3002 \u7528\u516c\u5f0f\u8868\u793a\u7684\u8bdd\uff0c\u5982\u4e0b\uff1a<\/span><\/span><\/span><\/p>\n


                      <\/span><\/span><\/span><\/p>\n

                      Q<\/span><\/span><\/span><\/span>\u2217<\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span>\u2248<\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>;<\/span><\/span>\u03b8<\/span><\/span>)<\/span><\/span><\/span><\/p>\n


                      <\/span><\/p>\n

                      \u5176\u4e2d\uff1a<\/span><\/span><\/p>\n

                        \n
                      • \u8f93\u5165\uff1a\u539f\u59cb\u72b6\u6001 s\uff08\u5982\u6e38\u620f\u753b\u9762\u50cf\u7d20\u77e9\u9635\uff09<\/span><\/span><\/span><\/li>\n
                      • \u8f93\u51fa\uff1a\u6bcf\u4e2a\u52a8\u4f5c a \u5bf9\u5e94\u7684Q\u503c\uff08\u79bb\u6563\u52a8\u4f5c\u7a7a\u95f4\uff09<\/span><\/span><\/span><\/li>\n
                      • \u53c2\u6570\uff1a\u03b8 \u4e3a\u795e\u7ecf\u7f51\u7edc\u7684\u6743\u91cd\uff0c\u901a\u8fc7\u68af\u5ea6\u4e0b\u964d\u4f18\u5316<\/span><\/span><\/span><\/li>\n<\/ul>\n


                        <\/span><\/span><\/p>\n

                        <\/span><\/span><\/p>\n

                        \u7f51\u7edc\u5177\u4f53\u7ed3\u6784<\/span><\/span><\/p>\n


                        <\/span><\/span><\/p>\n

                        \u5728\u8bba\u6587<\/span><\/span>[1]<\/span><\/span>\u4e2dDQN\u6240\u7528\u7684\u7f51\u7edc\u7ed3\u6784\u662f\u4e09\u4e2a\u5377\u79ef\u5c42\u52a0\u4e24\u4e2a\u5168\u8fde\u63a5\u5c42\uff0c\u7f51\u7edc\u7684\u8f93\u5165\u662f\u7ecf\u8fc7\u9884\u5904\u7406\u540e\u7684\u6570\u636e\uff0c\u9884\u5904\u7406\u6307\u7684\u662f\u5c06210*160\u7684\u6e38\u620f\u56fe\u50cf\u8f6c\u5316\u4e3a84*84\u7684\u56fe\u50cf\u7b49\u64cd\u4f5c\uff0c\u5f97\u5230\u7b97\u6cd5\u7ed3\u6784\u4e2d\u7684\u5177\u4f53\u4ee3\u7801\u5b9e\u73b0\u548c\u7f51\u7edc\u7ed3\u6784\u89c1\u56fe\u548c\u4ee5\u4e0b\u4ee3\u7801\uff1a<\/span><\/span><\/span><\/p>\n


                        <\/span><\/span><\/p>\n

                        \n
                          \n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n
                        • <\/li>\n<\/ul>\n
                          # \u9884\u5904\u7406\u56fe\u50cf<\/span><\/span><\/code>def<\/span> preprocess(frame):<\/span><\/code>    # \u8f6c\u7070\u5ea6\u56fe<\/span><\/span><\/code>    frame<\/span> = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)<\/span><\/code>    # \u8c03\u6574\u5927\u5c0f<\/span><\/span><\/code>    frame<\/span> = cv2.resize(frame, (84<\/span>, 84<\/span>))<\/span><\/code>    # \u5f52\u4e00\u5316<\/span><\/span><\/code>    frame<\/span> = frame.astype(np.float32) \/ 255<\/span>.0<\/span><\/span><\/code>    return<\/span> frame<\/span><\/code># Q-Network (CNN)<\/span><\/span><\/code>class<\/span> DQN(nn.Module):<\/span><\/code>    def<\/span> __init__(self, action_dim):<\/span><\/code>        super<\/span>(DQN, self).__init__()<\/span><\/code>        self<\/span>.conv1 = nn.Conv2d(4<\/span>, 32<\/span>, kernel_size=8<\/span>, stride=4<\/span>)<\/span><\/code>        self<\/span>.conv2 = nn.Conv2d(32<\/span>, 64<\/span>, kernel_size=4<\/span>, stride=2<\/span>)<\/span><\/code>        self<\/span>.conv3 = nn.Conv2d(64<\/span>, 64<\/span>, kernel_size=3<\/span>, stride=1<\/span>)<\/span><\/code>        self<\/span>.fc1 = nn.Linear(64<\/span> * 7<\/span> * 7<\/span>, 512<\/span>)<\/span><\/code>        self<\/span>.fc2 = nn.Linear(512<\/span>, action_dim)<\/span><\/code>    def<\/span> forward(self, x):<\/span><\/code>        x<\/span> = F.relu(self.conv1(x))<\/span><\/code>        x<\/span> = F.relu(self.conv2(x))<\/span><\/code>        x<\/span> = F.relu(self.conv3(x))<\/span><\/code>        x<\/span> = torch.flatten(x, start_dim=1<\/span>)<\/span><\/code>        x<\/span> = F.relu(self.fc1(x))<\/span><\/code><\/pre>\n<\/section>\n

                          <\/span><\/span><\/p>\n
                          \u6838\u5fc3\u610f\u4e49<\/span><\/span><\/b>\uff1a<\/span><\/span><\/span><\/p>\n
                            \n
                          • \u7aef\u5230\u7aef\u5b66\u4e60\uff1a\u76f4\u63a5\u4ece\u539f\u59cb\u72b6\u6001\uff08\u5982\u50cf\u7d20\uff09\u6620\u5c04\u5230\u52a8\u4f5c\u4ef7\u503c\uff0c\u65e0\u9700\u624b\u5de5\u8bbe\u8ba1\u7279\u5f81\u3002<\/span><\/span><\/span><\/li>\n
                          • \u6cdb\u5316\u80fd\u529b\uff1a\u76f8\u4f3c\u72b6\u6001\u5171\u4eab\u795e\u7ecf\u7f51\u7edc\u7684\u7279\u5f81\u8868\u793a\uff0c\u63d0\u5347\u6837\u672c\u5229\u7528\u7387\u3002<\/span><\/span><\/span><\/li>\n
                          • \u9ad8\u7ef4\u5904\u7406\uff1a\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff08CNN\uff09\u53ef\u9ad8\u6548\u63d0\u53d6\u56fe\u50cf\u4e2d\u7684\u7a7a\u95f4\u7279\u5f81\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n

                            <\/span><\/span><\/h3>\n
                            3.2 \u7ecf\u9a8c\u56de\u653e\uff08Experience Replay\uff09\u673a\u5236<\/span><\/span><\/span><\/h3>\n

                            <\/span><\/p>\n
                            \u901a\u8fc7\u7ecf\u9a8c\u56de\u653e\u53ef\u4ee5\u4ee4\u795e\u7ecf\u7f51\u7edc\u7684\u8bad\u7ec3\u6536\u655b\u4e14\u7a33\u5b9a\u3002\u8fd9\u662f\u56e0\u4e3a\u8bad\u7ec3\u795e\u7ecf\u2f79\u7edc\u65f6\uff0c\u5b58\u5728\u7684\u5047\u8bbe\u662f\u8bad\u7ec3\u6570\u636e\u662f\u72ec\u2f74\u540c\u5206\u5e03\u7684\uff0c\u4f46\u662f\u901a\u8fc7\u5f3a\u5316\u5b66\u4e60\u91c7\u96c6\u7684\u6570\u636e\u4e4b\u95f4\u5b58\u5728\u7740\u5173\u8054\u6027\uff0c\u5229\u2f64\u8fd9\u4e9b\u6570\u636e\u8fdb\u2f8f\u987a\u5e8f\u8bad\u7ec3\uff0c\u795e\u7ecf\u2f79\u7edc\u5f53\u7136\u4e0d\u7a33\u5b9a\u3002\u7ecf\u9a8c\u56de\u653e\u53ef\u4ee5\u6253\u7834\u6570\u636e\u95f4\u7684\u5173\u8054\u3002\u5728\u5f3a\u5316\u5b66\u4e60\u8fc7\u7a0b\u4e2d\uff0c\u667a\u80fd\u4f53\u5c06\u6570\u636e\u5b58\u50a8\u5230\u2f00\u4e2a\u6570\u636e\u5e93\u4e2d\uff0c\u518d\u5229\u2f64\u5747\u5300\u968f\u673a\u91c7\u6837\u7684\u2f45\u6cd5\u4ece\u6570\u636e\u5e93\u4e2d\u62bd\u53d6\u6570\u636e\uff0c\u7136\u540e\u5229\u2f64\u62bd\u53d6\u7684\u6570\u636e\u8bad\u7ec3\u795e\u7ecf\u2f79\u7edc\u3002 \u8fd9\u79cd\u7ecf\u9a8c\u56de\u653e\u7684\u6280\u5de7\u53ef\u4ee5\u6253\u7834\u6570\u636e\u4e4b\u95f4\u7684\u5173\u8054\u6027\u3002<\/span><\/span><\/p>\n

                            <\/span><\/p>\n
                            <\/section>\n
                            \u7ecf\u9a8c\u56de\u6536<\/span><\/span><\/p>\n

                            <\/span>\u4ee3\u7801\u5b9e\u73b0\u53ef\u53c2\u8003\uff1a<\/span><\/span><\/p>\n

                            <\/span><\/p>\n
                            \n
                              \n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n
                            • <\/li>\n<\/ul>\n
                              # \u7ecf\u9a8c\u56de\u653e<\/span><\/span><\/code>class<\/span> ReplayMemory<\/span>:<\/span><\/code>    def<\/span> __init__<\/span>(self<\/span><\/span>, capacity<\/span>):<\/span><\/code>        self<\/span>.memory = deque(maxlen=capacity)<\/span><\/code>    def<\/span> push<\/span>(self<\/span><\/span>, state, action, reward, next_state, done<\/span>):<\/span><\/code>        self<\/span>.memory.append((state, action, reward, next_state, done))<\/span><\/code>    def<\/span> sample<\/span>(self<\/span><\/span>, batch_size<\/span>):<\/span><\/code>        return<\/span> random.sample(self<\/span>.memory, batch_size)<\/span><\/code>    def<\/span> __len__<\/span>(self<\/span><\/span>):<\/span><\/code>        return<\/span> len(self<\/span>.memory)<\/span><\/code><\/pre>\n<\/section>\n

                              <\/span><\/span><\/pre>\n
                              3.3 \u76ee\u6807\u7f51\u7edc\uff08Target Network\uff09\u7684\u5f15\u5165\u4e0e\u53c2\u6570\u66f4\u65b0\u7b56\u7565<\/span><\/span><\/span><\/h3>\n

                              <\/span><\/span><\/h3>\n
                              \u4e3a\u4e86\u66f4\u957f\u8fdc\u7684\u63d0\u9ad8\u7a33\u5b9a\u6027\uff0c\u4f7f\u7528\u4e00\u4e2a\u72ec\u7acb\u7684\u795e\u7ecfQ\u7f51\u7edc\u53bb\u5f97\u5230target y\uff0c\u5373\u5efa\u7acb\u53e6\u4e00\u4e2a\u72ec\u7acb\u7684\u795e\u7ecf\u7f51\u7edctarget network\u3002\u6bcf\u7ecf\u8fc7c\u6b21\u66f4\u65b0\u540e\uff0c\u590d\u5236 network Q\u5f97\u5230target network Q\u2019\uff0c\u7136\u540e\u4f7f\u7528Q\u2019\u53bb\u751f\u6210target y\u5e94\u7528\u4e0e\u63a5\u4e0b\u6765\u7684c\u6b21\u66f4\u65b0\u3002\u597d\u5904\u5c31\u662f\u6bd4\u8d77online Q-learning\u7b97\u6cd5\u66f4\u7a33\u5b9a\u3002\u56e0\u4e3aonline Q-learning\u6bcf\u6b21\u66f4\u65b0\u5728\u589e\u52a0<\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span>\u7684\u540c\u65f6\u4e5f\u589e\u52a0\u4e86\u9488\u5bf9\u6bcf\u4e2aa\u7684<\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span>\uff0c\u4ece\u800c\u589e\u52a0\u4e86 target y\uff0c\u8fd9\u53c8\u53ef\u80fd\u4f1a\u5f15\u8d77\u7b56\u7565\u7684\u9707\u8361\u751a\u81f3\u53d1\u6563\u3002\u6240\u4ee5\u76ee\u6807\u7f51\u7edc\uff08Target Network\uff09\u7684\u5f15\u5165\u53ef\u4ee5\u89e3\u51b3\u8fd9\u4e00\u95ee\u9898\u3002<\/span><\/span><\/span><\/p>\n

                              <\/span>\u4e8e\u662f\uff0c<\/span><\/span><\/p>\n
                               \u2022 <\/span><\/span>\u635f\u5931\u51fd\u6570\u8868\u793a\u4e3a\uff08\u5ffd\u7565\u65e0\u5173\u7684\u65b9\u5dee\u9879\uff09<\/span><\/span>
                              <\/span><\/span><\/p>\n

                              <\/span><\/span><\/p>\n
                              <\/span><\/span><\/p>\n

                              <\/span><\/span><\/p>\n
                               \u2022 <\/span><\/span>\u68af\u5ea6\u4e3a<\/span><\/span><\/span><\/section>\n

                              <\/span><\/span><\/section>\n
                              <\/span><\/span><\/section>\n

                              <\/span><\/span><\/section>\n
                              \u5de5\u7a0b\u4e2d\uff0c\u4f7f\u7528\u968f\u673a\u68af\u5ea6\u4e0b\u964d\u65b9\u6cd5\u6765\u5b8c\u6210\u4f18\u5316\uff0c\u53e6\u5916\u5f53\u4ee4<\/span><\/span>\"image.png\"<\/span><\/sub>\u65f6\uff0c\u9000\u5316\u4e3a\u539f\u59cb\u7684 Q-Learning \u66f4\u65b0\u5f62\u5f0f\u3002<\/span><\/span><\/span><\/section>\n

                              <\/span>DQN\u7684\u8bad\u7ec3\u4e0e\u53c2\u6570\u66f4\u65b0\u7b56\u7565\u53ef\u90e8\u5206\u53c2\u8003\u4ee5\u4e0b\u4ee3\u7801\u5b9e\u73b0\uff1a<\/span><\/span><\/p>\n

                              <\/span><\/p>\n
                              \n
                                \n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n
                              • <\/li>\n<\/ul>\n
                                # DQN Agent<\/span><\/span><\/code>class<\/span> DQNAgent<\/span>:<\/span><\/code>    def<\/span> __init__<\/span>(self, action_dim<\/span>):<\/span><\/code>        self.action_dim = action_dim<\/span><\/code>        self.memory = ReplayMemory(10000<\/span>)<\/span><\/code>        self.policy_net = DQN(action_dim).to(device)<\/span><\/code>        self.target_net = DQN(action_dim).to(device)<\/span><\/code>        self.target_net.load_state_dict(self.policy_net.state_dict())<\/span><\/code>        self.optimizer = optim.Adam(self.policy_net.parameters(), lr=0.0001<\/span>)<\/span><\/code>        self.exploration_rate = 1.0<\/span><\/span><\/code>        self.steps_done = 0<\/span><\/span><\/code>    def<\/span> select_action<\/span>(self, state<\/span>):<\/span><\/code>        # \u03f5-\u8d2a\u5fc3\u7b56\u7565<\/span><\/span><\/code>        if<\/span> random.random() < self.exploration_rate:<\/span><\/code>            return<\/span> random.randrange(self.action_dim)<\/span><\/code>        else<\/span>:<\/span><\/code>            with<\/span> torch.no_grad():<\/span><\/code>                return<\/span> self.policy_net(state).argmax().item()<\/span><\/code>    def<\/span> train<\/span>(self<\/span>):<\/span><\/code>        # \u6837\u672c\u4e0d\u8db3\u65f6\u76f4\u63a5\u8fd4\u56de<\/span><\/span><\/code>        if<\/span> len<\/span>(self.memory) < 64<\/span>:<\/span><\/code>            return<\/span><\/span><\/code>        # \u4ece\u7ecf\u9a8c\u56de\u653e\u4e2d\u91c7\u6837<\/span><\/span><\/code>        batch = self.memory.sample(64<\/span>)<\/span><\/code>        states, actions, rewards, next_states, dones = zip<\/span>(*batch)<\/span><\/code>        # \u8f6c\u4e3a\u5f20\u91cf\uff0c\u5e76\u53bb\u9664\u591a\u4f59\u7ef4\u5ea6<\/span><\/span><\/code>        states = torch.tensor(np.array(states), dtype=torch.float32).to(device)  # shape: (64, 1, 4, 84, 84)<\/span><\/span><\/code>        states = states.squeeze(1<\/span>)  # shape: (64, 4, 84, 84)<\/span><\/span><\/code>        actions = torch.tensor(actions, dtype=torch.int64).unsqueeze(1<\/span>).to(device)  # shape: (64, 1)<\/span><\/span><\/code>        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)  # shape: (64,)<\/span><\/span><\/code>        next_states = torch.tensor(np.array(next_states), dtype=torch.float32).to(device)  # shape: (64, 1, 4, 84, 84)<\/span><\/span><\/code>        next_states = next_states.squeeze(1<\/span>)  # shape: (64, 4, 84, 84)<\/span><\/span><\/code>        dones = torch.tensor(dones, dtype=torch.bool<\/span>).to(device)  # shape: (64,)<\/span><\/span><\/code>        # \u8ba1\u7b97\u5f53\u524d Q \u503c<\/span><\/span><\/code>        q_values = self.policy_net(states).gather(1<\/span>, actions).squeeze()<\/span><\/code>        # \u8ba1\u7b97\u4e0b\u4e00\u65f6\u523b\u6700\u5927 Q \u503c<\/span><\/span><\/code>        next_q_values = self.target_net(next_states).max<\/span>(1<\/span>)[0<\/span>].detach()<\/span><\/code>        # \u8ba1\u7b97\u76ee\u6807 Q \u503c<\/span><\/span><\/code>        target_q_values = rewards + (0.99<\/span> * next_q_values * ~dones)<\/span><\/code>        # \u4f18\u5316<\/span><\/span><\/code>        loss = F.mse_loss(q_values, target_q_values)<\/span><\/code>        self.optimizer.zero_grad()<\/span><\/code>        loss.backward()<\/span><\/code>        self.optimizer.step()<\/span><\/code>        # \u66f4\u65b0\u63a2\u7d22\u7387<\/span><\/span><\/code>        if<\/span> self.exploration_rate > 0.01<\/span>:<\/span><\/code>            self.exploration_rate *= 0.995<\/span><\/span><\/code>    def<\/span> update_target_net<\/span>(self<\/span>):<\/span><\/code>        self.target_net.load_state_dict(self.policy_net.state_dict())<\/span><\/code># \u8bad\u7ec3\u5faa\u73af<\/span><\/span><\/code>   ...<\/span><\/code>    # \u5b9a\u671f\u66f4\u65b0\u76ee\u6807\u7f51\u7edc<\/span><\/span><\/code>    if<\/span> episode % 10<\/span> == 0<\/span>:<\/span><\/code>        agent.update_target_net()<\/span><\/code>   ...<\/span><\/code><\/pre>\n<\/section>\n

                                <\/span><\/p>\n

                                <\/span><\/span><\/h2>\n
                                \n
                                \n
                                \n
                                \n
                                \n

                                <\/span><\/section>\n

                                <\/span><\/section>\n
                                \n
                                <\/section>\n<\/section>\n<\/section>\n
                                \n
                                4. DQN\u5728pytorch\u4e2d\u7684\u7b80\u5355\u5b8c\u6574\u5b9e\u73b0<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n
                                \n

                                <\/span><\/section>\n

                                <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n

                                <\/span><\/p>\n
                                \u8be5\u4ee3\u7801\u57fa\u4e8e\u6df1\u5ea6Q\u7f51\u7edc\uff08DQN\uff09\u7b97\u6cd5\uff0c\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u4f7f\u7528DQN\u89e3\u51b3OpenAI Gym\u4e2d\u7684\u7ecf\u5178\u63a7\u5236\u4efb\u52a1CartPole-v1\u7684\u5b8c\u6574\u4f8b\u5b50\u3002\u4efb\u52a1\u76ee\u6807\u662f\u63a7\u5236\u5c0f\u8f66\u5de6\u53f3\u79fb\u52a8\uff0c\u4f7f\u7ad6\u76f4\u6746\u5b50\u5c3d\u53ef\u80fd\u957f\u65f6\u95f4\u4fdd\u6301\u76f4\u7acb\u3002\u667a\u80fd\u4f53\u901a\u8fc74\u7ef4\u72b6\u6001\uff08\u5c0f\u8f66\u4f4d\u7f6e\u3001\u901f\u5ea6\u3001\u6746\u5b50\u89d2\u5ea6\u53ca\u89d2\u901f\u5ea6\uff09\u611f\u77e5\u73af\u5883\uff0c\u8f93\u51fa\u79bb\u6563\u52a8\u4f5c\uff08\u5de6\u79fb\/\u53f3\u79fb\uff09\uff0c\u6700\u7ec8\u5b9e\u73b0\u7a33\u5b9a\u5e73\u8861\u8d85\u8fc7500\u6b65\u7684\u6ee1\u5206\u8868\u73b0\u3002<\/span><\/span><\/p>\n

                                <\/span><\/p>\n
                                <\/section>\n
                                CartPole-v1<\/span><\/span><\/p>\n

                                <\/span>\u7b97\u6cd5\u5b9e\u73b0Q\u503c\u7f51\u7edc\uff1a \u4f7f\u7528\u4e09\u5c42\u5168\u8fde\u63a5\u795e\u7ecf\u7f51\u7edc\uff08\u8f93\u51654\u7ef4\u2192128\u2192128\u2192\u8f93\u51fa2\u7ef4\uff09\u8fd1\u4f3cQ\u503c\u51fd\u6570\uff0c\u8f93\u51fa\u5de6\u53f3\u79fb\u52a8\u7684\u52a8\u4f5c\u4ef7\u503c\u3002 \u76ee\u6807\u7f51\u7edc\u4e0e\u4e3b\u7f51\u7edc\u5206\u79bb\uff0c\u5b9a\u671f\u540c\u6b65\u53c2\u6570\u4ee5\u7a33\u5b9a\u8bad\u7ec3\u3002\u7ecf\u9a8c\u56de\u653e\uff1a \u5b58\u50a810,000\u6761\u5386\u53f2\u8f6c\u79fb\u6837\u672c\uff08\u72b6\u6001\u3001\u52a8\u4f5c\u3001\u5956\u52b1\u3001\u4e0b\u4e00\u72b6\u6001\u3001\u7ec8\u6b62\u6807\u5fd7\uff09\uff0c\u8bad\u7ec3\u65f6\u968f\u673a\u91c7\u683764\u6761\u6570\u636e\u6253\u7834\u65f6\u5e8f\u76f8\u5173\u6027\u3002\u8bad\u7ec3\u673a\u5236\uff1a \u03b5-\u8d2a\u5a6a\u7b56\u7565\u52a8\u6001\u8870\u51cf\uff08\u521d\u59cb\u03b5=1.0\uff0c\u8870\u51cf\u73870.995\uff0c\u6700\u4f4e\u03b5=0.01\uff09\uff0c\u5e73\u8861\u63a2\u7d22\u4e0e\u5229\u7528\u3002 \u57fa\u4e8e\u8d1d\u5c14\u66fc\u65b9\u7a0b\u8ba1\u7b97\u76ee\u6807Q\u503c\uff0c\u901a\u8fc7\u5747\u65b9\u8bef\u5dee\u635f\u5931\u66f4\u65b0\u7f51\u7edc\uff0c\u6298\u6263\u56e0\u5b50\u03b3=0.99\u3002<\/span><\/span><\/p>\n

                                <\/span><\/p>\n
                                \n
                                  \n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n
                                • <\/li>\n<\/ul>\n
                                  import<\/span> torch<\/span><\/code>import<\/span> torch.nn as<\/span> nn<\/span><\/code>import<\/span> torch.optim as<\/span> optim<\/span><\/code>import<\/span> random<\/span><\/code>from<\/span> collections import<\/span> deque<\/span><\/code>import<\/span> gym<\/span><\/code># \u5b9a\u4e49Q\u7f51\u7edc<\/span><\/span><\/code>class<\/span> QNetwork<\/span>(nn.Module):<\/span><\/code>    def<\/span> __init__<\/span>(self, input_dim, output_dim<\/span>):<\/span><\/code>        super<\/span>(QNetwork, self).__init__()<\/span><\/code>        self.fc = nn.Sequential(<\/span><\/code>            nn.Linear(input_dim, 128<\/span>),<\/span><\/code>            nn.ReLU(),<\/span><\/code>            nn.Linear(128<\/span>, 128<\/span>),<\/span><\/code>            nn.ReLU(),<\/span><\/code>            nn.Linear(128<\/span>, output_dim)<\/span><\/code>        )<\/span><\/code>    def<\/span> forward<\/span>(self, x<\/span>):<\/span><\/code>        return<\/span> self.fc(x)<\/span><\/code># \u5b9a\u4e49DQN\u4ee3\u7406<\/span><\/span><\/code>class<\/span> DQNAgent<\/span>:<\/span><\/code>    def<\/span> __init__<\/span>(self, state_dim, action_dim, lr=<\/span>0.001<\/span><\/span>, gamma=<\/span>0.99<\/span><\/span>, epsilon=<\/span>1.0<\/span><\/span>, epsilon_decay=<\/span>0.995<\/span><\/span>, min_epsilon=<\/span>0.01<\/span><\/span>, buffer_size=<\/span>10000<\/span><\/span>, batch_size=<\/span>64<\/span><\/span>):<\/span><\/code>        self.state_dim = state_dim<\/span><\/code>        self.action_dim = action_dim<\/span><\/code>        self.gamma = gamma<\/span><\/code>        self.epsilon = epsilon<\/span><\/code>        self.epsilon_decay = epsilon_decay<\/span><\/code>        self.min_epsilon = min_epsilon<\/span><\/code>        self.batch_size = batch_size<\/span><\/code>        self.q_network = QNetwork(state_dim, action_dim)<\/span><\/code>        self.target_network = QNetwork(state_dim, action_dim)<\/span><\/code>        self.target_network.load_state_dict(self.q_network.state_dict())<\/span><\/code>        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)<\/span><\/code>        self.replay_buffer = deque(maxlen=buffer_size)<\/span><\/code>        self.criterion = nn.MSELoss()<\/span><\/code>    def<\/span> choose_action<\/span>(self, state<\/span>):<\/span><\/code>        if<\/span> random.random() < self.epsilon:<\/span><\/code>            return<\/span> random.randint(0<\/span>, self.action_dim - 1<\/span>)<\/span><\/code>        else<\/span>:<\/span><\/code>            state = torch.tensor([state], dtype=torch.float32)<\/span><\/code>            q_values = self.q_network(state)<\/span><\/code>            return<\/span> torch.argmax(q_values).item()<\/span><\/code>    def<\/span> store_experience<\/span>(self, state, action, reward, next_state, done<\/span>):<\/span><\/code>        self.replay_buffer.append((state, action, reward, next_state, done))<\/span><\/code>    def<\/span> update_network<\/span>(self<\/span>):<\/span><\/code>        if<\/span> len<\/span>(self.replay_buffer) < self.batch_size:<\/span><\/code>            return<\/span><\/span><\/code>        batch = random.sample(self.replay_buffer, self.batch_size)<\/span><\/code>        states, actions, rewards, next_states, dones = zip<\/span>(*batch)<\/span><\/code>        states = torch.tensor(states, dtype=torch.float32)<\/span><\/code>        actions = torch.tensor(actions, dtype=torch.int64)<\/span><\/code>        rewards = torch.tensor(rewards, dtype=torch.float32)<\/span><\/code>        next_states = torch.tensor(next_states, dtype=torch.float32)<\/span><\/code>        dones = torch.tensor(dones, dtype=torch.float32)<\/span><\/code>        q_values = self.q_network(states).gather(1<\/span>, actions.unsqueeze(1<\/span>)).squeeze()<\/span><\/code>        next_q_values = self.target_network(next_states).max<\/span>(1<\/span>)[0<\/span>]<\/span><\/code>        target_q_values = rewards + self.gamma * next_q_values * (1<\/span> - dones)<\/span><\/code>        loss = self.criterion(q_values, target_q_values)<\/span><\/code>        self.optimizer.zero_grad()<\/span><\/code>        loss.backward()<\/span><\/code>        self.optimizer.step()<\/span><\/code>    def<\/span> update_target_network<\/span>(self<\/span>):<\/span><\/code>        self.target_network.load_state_dict(self.q_network.state_dict())<\/span><\/code>    def<\/span> decay_epsilon<\/span>(self<\/span>):<\/span><\/code>        self.epsilon = max<\/span>(self.min_epsilon, self.epsilon * self.epsilon_decay)<\/span><\/code># \u73af\u5883<\/span><\/span><\/code>env = gym.make('CartPole-v1'<\/span>)<\/span><\/code>state_dim = env.observation_space.shape[0<\/span>]<\/span><\/code>action_dim = env.action_space.n<\/span><\/code># \u521d\u59cb\u5316\u4ee3\u7406<\/span><\/span><\/code>agent = DQNAgent(state_dim, action_dim)<\/span><\/code># \u8bad\u7ec3\u5faa\u73af<\/span><\/span><\/code>episodes = 1000<\/span><\/span><\/code>for<\/span> episode in<\/span> range<\/span>(episodes):<\/span><\/code>    state = env.reset()<\/span><\/code>    total_reward = 0<\/span><\/span><\/code>    done = False<\/span><\/span><\/code>    while<\/span> not<\/span> done:<\/span><\/code>        action = agent.choose_action(state)<\/span><\/code>        next_state, reward, done, _ = env.step(action)<\/span><\/code>        agent.store_experience(state, action, reward, next_state, done)<\/span><\/code>        agent.update_network()<\/span><\/code>        state = next_state<\/span><\/code>        total_reward += reward<\/span><\/code>    agent.update_target_network()<\/span><\/code>    agent.decay_epsilon()<\/span><\/code>    print<\/span>(f'Episode <\/span>{episode + <\/span><\/span>1<\/span><\/span><\/span>}<\/span><\/span>\/<\/span>{episodes}<\/span><\/span>, Total Reward: <\/span>{total_reward}<\/span><\/span>, Epsilon: <\/span>{agent.epsilon:<\/span><\/span>.3<\/span><\/span><\/span>f}<\/span><\/span>'<\/span>)<\/span><\/code># \u8bc4\u4f30<\/span><\/span><\/code>state = env.reset()<\/span><\/code>done = False<\/span><\/span><\/code>while<\/span> not<\/span> done:<\/span><\/code>    action = agent.choose_action(state)<\/span><\/code>    state, _, done, _ = env.step(action)<\/span><\/code>    env.render()<\/span><\/code>env.close()<\/span><\/code><\/pre>\n<\/section>\n

                                  <\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \n
                                  \n
                                  \n
                                  \n
                                  \n

                                  <\/span><\/section>\n

                                  <\/span><\/section>\n
                                  \n
                                  <\/section>\n<\/section>\n<\/section>\n
                                  \n
                                  5. DQN \u7684\u4e3b\u8981\u6539\u8fdb\u7b97\u6cd5<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n
                                  \n

                                  <\/span><\/section>\n

                                  <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n

                                  <\/span><\/p>\n
                                  5.1 Double DQN\uff1a\u89e3\u51b3 Q \u503c\u8fc7\u9ad8\u4f30\u8ba1\u95ee\u9898<\/span><\/span><\/span><\/h3>\n

                                  <\/span><\/span><\/h3>\n
                                  5.1.1 Double DQN\u7684\u601d\u60f3<\/span><\/span><\/span><\/h4>\n

                                  <\/span><\/span><\/h4>\n
                                  <\/section>\n
                                  \u5728DQN\u4e0eDouble DQN\u4e2d\u4f7f\u7528\u76ee\u6807\u7f51\u7edc\u7684\u5b66\u4e60\u7ed3\u6784<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \u5728\u7ecf\u5178\u7684 Q-L<\/span>earning \u53ca\u5176\u6df1\u5ea6\u7248\u672c DQN\uff08Deep Q-Network\uff09\u4e2d\uff0c\u52a8\u4f5c\u503c\u51fd\u6570\u7684\u4f30\u8ba1\u5b58\u5728\u7cfb\u7edf\u6027\u504f\u5dee\uff0c\u5373\u8fc7\u9ad8\u4f30\u8ba1\uff08overestimation bias\uff09\u3002\u8fd9\u79cd\u504f\u5dee\u6e90\u81ea\u4e8e\u6700\u5927\u5316\u64cd\u4f5c\uff1a<\/span><\/p>\n
                                  <\/section>\n
                                  \u5f53\u4f30\u8ba1\u7684 Q \u503c\u5b58\u5728\u566a\u58f0\u65f6\uff0c\u76f4\u63a5\u9009\u53d6\u6700\u5927 Q \u503c\u52a8\u4f5c\u5bb9\u6613\u5bfc\u81f4\u671f\u671b\u7684\u8fc7\u9ad8\u4f30\u8ba1\uff0c\u4ece\u800c\u5f71\u54cd\u7b56\u7565\u7684\u7a33\u5b9a\u6027\u548c\u6700\u7ec8\u6027\u80fd\u3002\u4e3a\u4e86\u51cf\u5c11 DQN \u4e2d\u7684 Q \u503c\u8fc7\u9ad8\u4f30\u8ba1\uff0cDouble DQN \u5f15\u5165\u4e86\u4e24\u4e2a Q \u7f51\u7edc\uff1a\u5f53\u524d Q \u7f51\u7edc\u548c\u76ee\u6807 Q \u7f51\u7edc<\/span><\/span>[4]<\/span><\/span>\u3002<\/span><\/span><\/span><\/p>\n

                                  <\/span>\u7ecf\u5178 DQN \u7684\u76ee\u6807 Q \u503c\u8ba1\u7b97\u516c\u5f0f\u4e3a\uff1a<\/span><\/span><\/p>\n
                                  <\/section>\n
                                  \u5176\u4e2d\uff1a <\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub> \u662f\u5728\u65f6\u95f4\u6b65 <\/span><\/span>t<\/span><\/span> \u83b7\u53d6\u7684\u5956\u52b1\uff0c <\/span><\/span>\u03b3<\/span><\/span> \u662f\u6298\u6263\u56e0\u5b50\uff0c<\/span><\/span>\"image.png\"<\/span><\/sub>\u662f\u5728\u65f6\u95f4\u6b65 <\/span><\/span>t<\/span><\/span>+<\/span><\/span>1<\/span><\/span> \u4e0b\uff0c\u901a\u8fc7\u76ee\u6807\u7f51\u7edc <\/span><\/span>\u03b8<\/span><\/span>\u2212<\/span><\/span><\/sup> \u8ba1\u7b97\u7684\u6700\u5927 Q \u503c\uff0c\u9009\u62e9\u4e0b\u4e00\u4e2a\u72b6\u6001 <\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span> \u4e2d\u7684\u6700\u4f73\u52a8\u4f5c\u3002<\/span><\/span><\/span><\/p>\n

                                  <\/span>Double DQN\u76ee\u6807 Q \u503c\u8ba1\u7b97\u516c\u5f0f\u6539\u4e3a\uff1a<\/span><\/span><\/p>\n
                                  <\/section>\n
                                  \u52a8\u4f5c\u9009\u62e9\uff1a\u4f9d\u7136\u901a\u8fc7\u5f53\u524d Q \u7f51\u7edc <\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>a<\/span><\/span>;<\/span><\/span>\u03b8<\/span><\/span>)<\/span><\/span> \u6765\u9009\u62e9\u4e0b\u4e00\u4e2a\u72b6\u6001 <\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span> \u4e2d\u7684\u6700\u4f18\u52a8\u4f5c\uff0c<\/span><\/span>\"image.png\"<\/span><\/sub>\uff0c\u8fd9\u4e2a\u6b65\u9aa4\u4f7f\u7528\u7684\u662f\u5f53\u524d\u7f51\u7edc <\/span><\/span>\u03b8<\/span><\/span> \u6765\u9009\u62e9\u52a8\u4f5c\u3002<\/span><\/span><\/span><\/p>\n
                                  \u52a8\u4f5c\u8bc4\u4f30\uff1a\u8bc4\u4f30\u8fd9\u4e2a\u52a8\u4f5c\u7684 Q \u503c\u65f6\uff0c\u4f7f\u7528\u7684\u662f\u76ee\u6807\u7f51\u7edc <\/span><\/span>\u03b8<\/span><\/span>\u2212<\/span><\/span><\/sup>\uff0c\u5373\u8bc4\u4f30\u9009\u4e2d\u7684\u52a8\u4f5c <\/span><\/span>a<\/span><\/span>\u2217<\/span><\/span><\/sup> \u5bf9\u5e94\u7684 Q \u503c\uff1a<\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/sub><\/span>+<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>a<\/span><\/span>\u2217<\/span><\/span><\/sup>;<\/span><\/span>\u03b8<\/span><\/span>\u2212<\/span><\/span><\/sup>)<\/span><\/span>\uff0c\u901a\u8fc7\u8fd9\u79cd\u65b9\u5f0f\uff0cDouble DQN \u901a\u8fc7\u5206\u5f00\u52a8\u4f5c\u9009\u62e9\u548c\u52a8\u4f5c\u8bc4\u4f30\uff0c\u907f\u514d\u4e86\u7ecf\u5178 DQN \u4e2d\u540c\u4e00\u7f51\u7edc\u540c\u65f6\u6267\u884c\u9009\u62e9\u548c\u8bc4\u4f30\u7684\u64cd\u4f5c\uff0c\u4ece\u800c\u51cf\u5c11\u4e86\u56e0\u9009\u62e9\u6700\u5927 Q \u503c\u800c\u5bfc\u81f4\u7684\u8fc7\u9ad8\u4f30\u8ba1\u504f\u5dee\u3002<\/span><\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  5.1.2 Double DQN\u7684\u4f18\u52bf<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  <\/span><\/h4>\n
                                  \u4f20\u7edf DQN \u5728\u66f4\u65b0\u65f6\uff0c\u76f4\u63a5\u7528\u76ee\u6807\u7f51\u7edc\u4f30\u8ba1\u7684\u6700\u5927 Q \u503c\u4f5c\u4e3a\u76ee\u6807\uff0c\u8fd9\u79cd\u201c\u6700\u5927\u5316\u201d\u64cd\u4f5c\u5bb9\u6613\u4ece\u566a\u58f0\u4e2d\u6311\u9009\u51fa\u504f\u5927\u503c\uff0c\u5bfc\u81f4\u6574\u4f53\u504f\u9ad8\u3002Double DQN \u5219\u5148\u7528\u5f53\u524d\u7f51\u7edc\u6311\u9009\u52a8\u4f5c\uff0c\u518d\u7528\u76ee\u6807\u7f51\u7edc\u8ba1\u7b97\u4ef7\u503c\uff0c\u907f\u514d\u4e86\u5355\u4e00\u7f51\u7edc\u540c\u65f6\u8d1f\u8d23\u9009\u52a8\u4f5c\u548c\u8bc4\u4f30\u52a8\u4f5c\u4ef7\u503c\u6240\u5e26\u6765\u7684\u504f\u5dee\u3002Double DQN \u662f\u4e00\u79cd\u7b80\u5355\u6709\u6548\u7684\u6539\u8fdb\u7b56\u7565\uff0c\u901a\u8fc7\u201c\u52a8\u4f5c\u9009\u62e9\u201d\u548c\u201c\u52a8\u4f5c\u8bc4\u4f30\u201d\u5206\u5f00\u4f30\u8ba1\uff0c\u663e\u8457\u7f13\u89e3\u4e86 Q \u503c\u8fc7\u9ad8\u4f30\u8ba1\u95ee\u9898\uff0c\u63d0\u9ad8\u4e86\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\u7684\u7a33\u5b9a\u6027\u548c\u6027\u80fd\uff0c\u5df2\u6210\u4e3a\u6df1\u5ea6\u5f3a\u5316\u5b66\u4e60\u9886\u57df\u7684\u91cd\u8981\u65b9\u6cd5\u4e4b\u4e00\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  5.2 Dueling DQN\uff1a\u5206\u79bb\u72b6\u6001\u4ef7\u503c\u4e0e\u4f18\u52bf\u51fd\u6570<\/span><\/span><\/span><\/h3>\n

                                  <\/span><\/span><\/h3>\n
                                  5.2.1 Dueling DQN\u7684\u601d\u60f3<\/span><\/span><\/span><\/h4>\n

                                  <\/span><\/span><\/h4>\n
                                  <\/span><\/p>\n
                                  Dueling DQN \u7f51\u7edc\u7ed3\u6784\u793a\u610f\u56fe<\/span><\/span><\/p>\n
                                  <\/span><\/h4>\n

                                  <\/span><\/span><\/p>\n
                                  Dueling DQN\uff08Dueling Deep Q-Network\uff09\u662f\u5bf9\u7ecf\u5178 DQN \u7684\u4e00\u79cd\u7ed3\u6784\u6027\u6539\u8fdb\uff0c\u65e8\u5728\u63d0\u5347\u667a\u80fd\u4f53\u5728\u590d\u6742\u73af\u5883\u4e2d\u7684\u5b66\u4e60\u6548\u7387\u548c\u7a33\u5b9a\u6027<\/span><\/span>[5]<\/span><\/span>\u3002<\/span><\/span><\/span><\/p>\n

                                  <\/span><\/span><\/p>\n
                                  \u5728\u4f20\u7edf\u7684 DQN \u4e2d\uff0cQ \u503c\u51fd\u6570 Q(s,a) \u76f4\u63a5\u8f93\u51fa\u6bcf\u4e2a\u52a8\u4f5c\u7684\u4ef7\u503c\u3002\u7136\u800c\uff0c\u5728\u8bb8\u591a\u60c5\u51b5\u4e0b\uff0c\u67d0\u4e9b\u72b6\u6001\u4e0b\u7684\u6240\u6709\u52a8\u4f5c\u7684\u4ef7\u503c\u5dee\u5f02\u8f83\u5c0f\uff0c\u5bfc\u81f4\u5b66\u4e60\u8fc7\u7a0b\u4e2d\u7684\u68af\u5ea6\u66f4\u65b0\u6548\u7387\u4f4e\u4e0b\u3002Dueling DQN \u901a\u8fc7\u5c06 Q \u503c\u51fd\u6570\u5206\u89e3\u4e3a\u72b6\u6001\u4ef7\u503c\u51fd\u6570 V(s) \u548c\u4f18\u52bf\u51fd\u6570 A(s,a)\uff0c\u4f7f\u5f97\u7f51\u7edc\u80fd\u591f\u5206\u522b\u8bc4\u4f30\u72b6\u6001\u7684\u603b\u4f53\u4ef7\u503c\u548c\u5404\u4e2a\u52a8\u4f5c\u7684\u76f8\u5bf9\u4f18\u52bf\uff0c\u4ece\u800c\u63d0\u9ad8\u5b66\u4e60\u6548\u7387<\/span><\/span>\u3002\u5177\u4f53\u5730\uff0cDueling DQN \u7684 Q \u503c\u8ba1\u7b97\u516c\u5f0f\u4e3a\uff1a<\/span><\/span><\/span><\/span><\/p>\n

                                  <\/span><\/span><\/span><\/span><\/p>\n
                                   <\/span><\/span><\/span>Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span>=V(s)+A<\/span><\/span><\/span>(<\/span><\/span>s<\/span><\/span>,<\/span><\/span>a<\/span><\/span>)<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \u5176\u4e2d\uff0cV(s) \u662f\u72b6\u6001s\u7684\u4ef7\u503c\uff0c\u8868\u793a\u5728\u8be5\u72b6\u6001\u4e0b\u667a\u80fd\u4f53\u7684\u671f\u671b\u56de\u62a5\uff1bA(s,a) \u662f\u52a8\u4f5ca\u76f8\u5bf9\u4e8e\u5f53\u524d\u72b6\u6001\u7684\u4f18\u52bf\uff0c\u8868\u793a\u5728\u8be5\u72b6\u6001\u4e0b\u91c7\u53d6\u52a8\u4f5ca\u76f8\u5bf9\u4e8e\u5176\u4ed6\u52a8\u4f5c\u7684\u4f18\u52bf\u7a0b\u5ea6\u3002<\/span><\/p>\n

                                  <\/span><\/p>\n
                                  Dueling DQN \u7684\u7f51\u7edc\u7ed3\u6784\u5728\u4f20\u7edf DQN \u7684\u57fa\u7840\u4e0a\u8fdb\u884c\u4e86\u4fee\u6539\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \u5177\u4f53\u800c\u8a00\uff0c\u7f51\u7edc\u7684\u524d\u51e0\u5c42\u5171\u4eab\uff0c\u7528\u4e8e\u63d0\u53d6\u8f93\u5165\u72b6\u6001\u7684\u7279\u5f81\uff1b\u7136\u540e\uff0c\u7f51\u7edc\u5206\u4e3a\u4e24\u4e2a\u5206\u652f\uff1a\u4e00\u4e2a\u7528\u4e8e\u8ba1\u7b97\u72b6\u6001\u4ef7\u503c\u51fd\u6570V(s)\uff0c\u53e6\u4e00\u4e2a\u7528\u4e8e\u8ba1\u7b97\u4f18\u52bf\u51fd\u6570 A(s,a)\u3002\u6700\u540e\uff0c\u4e24\u4e2a\u5206\u652f\u7684\u8f93\u51fa\u88ab\u5408\u5e76\uff0c\u5f97\u5230\u6bcf\u4e2a\u52a8\u4f5c\u7684 Q \u503c\u4f30\u8ba1\u3002\u8fd9\u79cd\u7ed3\u6784\u4f7f\u5f97\u7f51\u7edc\u80fd\u591f\u66f4\u6709\u6548\u5730\u5b66\u4e60\u5230\u72b6\u6001\u7684\u4ef7\u503c\u548c\u52a8\u4f5c\u7684\u4f18\u52bf\uff0c\u4ece\u800c\u63d0\u9ad8\u5b66\u4e60\u6027\u80fd\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  5.1.2 Dueling DQN\u7684\u4f18\u52bf<\/span><\/span><\/span><\/h4>\n

                                  <\/span><\/span><\/h4>\n
                                  \uff081\uff09\u63d0\u9ad8\u5b66\u4e60\u6548\u7387\uff1a\u901a\u8fc7\u5206\u79bb\u72b6\u6001\u4ef7\u503c\u548c\u52a8\u4f5c\u4f18\u52bf\uff0cDueling DQN \u80fd\u591f\u66f4\u597d\u5730\u805a\u7126\u4e8e\u91cd\u8981\u4fe1\u606f\uff0c\u51cf\u5c11\u65e0\u5173\u56e0\u7d20\u5e72\u6270\uff0c\u63d0\u9ad8\u5b66\u4e60\u6548\u7387\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \uff082\uff09\u589e\u5f3a\u9c81\u68d2\u6027\uff1a\u5bf9\u72b6\u6001\u4ef7\u503c\u548c\u4f18\u52bf\u7684\u72ec\u7acb\u5efa\u6a21\u6709\u52a9\u4e8e\u7f13\u89e3\u73af\u5883\u566a\u58f0\u548c\u7a00\u758f\u5956\u52b1\u5e26\u6765\u7684\u5f71\u54cd\uff0c\u63d0\u9ad8\u7b97\u6cd5\u7684\u9c81\u68d2\u6027\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  \uff083\uff09\u63d0\u5347\u6cdb\u5316\u80fd\u529b\uff1aDueling \u67b6\u6784\u5141\u8bb8\u7f51\u7edc\u5728\u672a\u89c1\u8fc7\u7684\u73af\u5883\u4e2d\u66f4\u5feb\u5730\u63a8\u65ad\u51fa\u5408\u7406\u7684 Q \u503c\uff0c\u589e\u5f3a\u4e86\u7b97\u6cd5\u7684\u6cdb\u5316\u80fd\u529b\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  5.3 DRQN<\/span><\/span><\/span><\/h3>\n

                                  <\/span><\/span><\/h3>\n
                                  5.3.1 DRQN\u7684\u601d\u60f3<\/span><\/span><\/span><\/h4>\n

                                  <\/span><\/span><\/h4>\n
                                  <\/section>\n
                                  DRQN\u67b6\u6784<\/span><\/span><\/p>\n

                                  <\/span><\/span><\/p>\n
                                  Deep Recurrent Q-Network\uff08DRQN\uff09\u662f\u5bf9\u7ecf\u5178 DQN \u7684\u4e00\u79cd\u6269\u5c55\uff0c\u65e8\u5728\u89e3\u51b3\u90e8\u5206\u53ef\u89c2\u5bdf\u9a6c\u5c14\u53ef\u592b\u51b3\u7b56\u8fc7\u7a0b\uff08POMDP\uff09\u4e2d\u7684\u6311\u6218<\/span><\/span><\/span><\/span>[6]<\/span><\/span>\u3002\u5728 POMDP \u4e2d\uff0c\u667a\u80fd\u4f53\u65e0\u6cd5\u83b7\u5f97\u5b8c\u6574\u7684\u73af\u5883\u72b6\u6001\u4fe1\u606f\uff0c\u4f20\u7edf\u7684 DQN \u4f9d\u8d56\u4e8e\u5b8c\u6574\u7684\u89c2\u5bdf\u5e8f\u5217\u8fdb\u884c\u51b3\u7b56\uff0c\u53ef\u80fd\u5bfc\u81f4\u6027\u80fd\u4e0b\u964d\u3002DRQN \u901a\u8fc7\u5f15\u5165\u9012\u5f52\u795e\u7ecf\u7f51\u7edc\uff08RNN\uff09\uff0c\u7279\u522b\u662f\u957f\u77ed\u671f\u8bb0\u5fc6\uff08LSTM\uff09\u7f51\u7edc\uff0c\u4f7f\u667a\u80fd\u4f53\u80fd\u591f\u5904\u7406\u90e8\u5206\u89c2\u5bdf\u4fe1\u606f\uff0c\u4fdd\u7559\u5386\u53f2\u4fe1\u606f\uff0c\u4ece\u800c\u5728\u90e8\u5206\u53ef\u89c2\u5bdf\u7684\u73af\u5883\u4e2d\u8868\u73b0\u51fa\u66f4\u597d\u7684\u6027\u80fd\u3002DRQN \u7684\u6838\u5fc3\u601d\u60f3\u662f\u5c06\u4f20\u7edf DQN \u7684\u5168\u8fde\u63a5\u5c42\u66ff\u6362\u4e3a LSTM \u5c42\uff0c\u4ee5\u4fbf\u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65\u90fd\u80fd\u4fdd\u7559\u5386\u53f2\u4fe1\u606f\u3002\u5177\u4f53\u800c\u8a00\uff0cDRQN \u7684 Q \u503c\u8ba1\u7b97\u516c\u5f0f\u4e3a\uff1a<\/span><\/span><\/span><\/p>\n

                                  <\/span><\/span><\/p>\n
                                  Q<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>)<\/span><\/span>=<\/span><\/span>LSTM<\/span><\/span>(<\/span><\/span>h<\/span><\/span>t<\/span><\/sub><\/span>\u2212<\/span><\/span><\/sub>1<\/span><\/sub><\/span>,<\/span><\/span>[<\/span><\/span>\u03d5<\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>)<\/span><\/span>,<\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>]<\/span><\/span>)<\/span><\/span><\/span><\/p>\n

                                  <\/span><\/span><\/p>\n
                                  \u5176\u4e2d\uff0ch<\/span><\/span>t<\/span><\/sub><\/span>\u2212<\/span><\/span><\/sub>1<\/span><\/sub>\u662f\u4e0a\u4e00\u65f6\u95f4\u6b65\u7684\u9690\u85cf\u72b6\u6001\uff0c\u03d5<\/span><\/span><\/span>(<\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>)<\/span>\u662f\u5f53\u524d\u72b6\u6001\u7684\u7279\u5f81\u8868\u793a\uff0ca<\/span><\/span>t<\/span><\/sub>\u662f\u5f53\u524d\u52a8\u4f5c\u3002\u901a\u8fc7\u8fd9\u79cd\u65b9\u5f0f\uff0cDRQN \u80fd\u591f\u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65\u90fd\u8003\u8651\u5230\u5386\u53f2\u4fe1\u606f\uff0c\u4ece\u800c\u66f4\u597d\u5730\u5e94\u5bf9\u90e8\u5206\u53ef\u89c2\u5bdf\u7684\u73af\u5883\u3002<\/span><\/span><\/span><\/span>
                                  <\/span><\/p>\n

                                  <\/span><\/p>\n
                                  DRQN \u7684\u7f51\u7edc\u7ed3\u6784\u901a\u5e38\u5305\u62ec\u4ee5\u4e0b\u51e0\u4e2a\u90e8\u5206\uff1a<\/span><\/span><\/p>\n
                                  \uff081\uff09\u5377\u79ef\u5c42\uff08Convolutional Layers\uff09\uff1a\u7528\u4e8e\u4ece\u8f93\u5165\u56fe\u50cf\u4e2d\u63d0\u53d6\u7279\u5f81\u3002<\/span><\/span><\/p>\n
                                  \uff082\uff09LSTM \u5c42\uff1a\u7528\u4e8e\u5904\u7406\u65f6\u95f4\u5e8f\u5217\u6570\u636e\uff0c<\/span>\u4fdd\u7559\u5386\u53f2\u4fe1\u606f\u3002<\/span><\/p>\n
                                  \uff083\uff09\u5168\u8fde\u63a5\u5c42\uff08Fully Connected Layer\uff09\uff1a\u7528\u4e8e\u8f93\u51fa\u6bcf\u4e2a\u52a8\u4f5c\u7684 Q \u503c\u3002 \u8fd9\u79cd\u7ed3\u6784\u4f7f\u5f97 DRQN \u80fd\u591f\u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65\u90fd\u8003\u8651\u5230\u5386\u53f2\u4fe1\u606f\uff0c\u4ece\u800c\u66f4\u597d\u5730\u5e94\u5bf9\u90e8\u5206\u53ef\u89c2\u5bdf\u7684\u73af\u5883\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/p>\n
                                  5.3.2 DRQN\u7684\u4f18\u52bf\u4e0e\u5c40\u9650<\/span><\/span><\/span><\/h4>\n

                                  <\/span><\/span><\/h4>\n
                                  1.\u4f18\u52bf<\/span><\/span><\/p>\n
                                  \uff081\uff09\u5904\u7406\u90e8\u5206\u53ef\u89c2\u5bdf\u73af\u5883\uff1aDRQN \u80fd\u591f\u5728\u6bcf\u4e2a\u65f6\u95f4\u6b65\u90fd\u4fdd\u7559\u5386\u53f2\u4fe1\u606f\uff0c\u4ece\u800c\u5728\u90e8\u5206\u53ef\u89c2\u5bdf\u7684\u73af\u5883\u4e2d\u8868\u73b0\u51fa\u66f4\u597d\u7684\u6027\u80fd\u3002<\/span><\/span><\/p>\n
                                  \uff082\uff09\u63d0\u9ad8\u51b3\u7b56\u8d28\u91cf\uff1a\u901a\u8fc7\u5f15\u5165 LSTM \u5c42\uff0cDRQN \u80fd\u591f\u66f4\u597d\u5730\u6355\u6349\u65f6\u95f4\u5e8f\u5217\u4e2d\u7684\u957f\u671f\u4f9d\u8d56\u5173\u7cfb\uff0c\u4ece\u800c\u63d0\u9ad8\u51b3\u7b56\u8d28\u91cf\u3002<\/span><\/span><\/p>\n
                                  \uff083\uff09\u9002\u5e94\u6027\u5f3a\uff1aDRQN \u53ef\u4ee5\u9002\u5e94\u4e0d\u540c\u7684\u73af\u5883\u548c\u4efb\u52a1\uff0c\u5177\u6709\u8f83\u5f3a\u7684\u6cdb\u5316\u80fd\u529b\u3002<\/span><\/span><\/p>\n

                                  <\/span>2.\u5c40\u9650<\/span><\/span><\/p>\n
                                  \uff081\uff09\u8bad\u7ec3\u96be\u5ea6\uff1a\u7531\u4e8e\u5f15\u5165\u4e86 LSTM \u5c42\uff0cDRQN \u7684\u8bad\u7ec3\u8fc7\u7a0b\u53ef\u80fd\u6bd4\u4f20\u7edf DQN \u66f4\u52a0\u590d\u6742\u548c\u56f0\u96be\u3002<\/span><\/span><\/p>\n
                                  \uff082\uff09\u8ba1\u7b97\u8d44\u6e90\u9700\u6c42\u9ad8\uff1aLSTM \u5c42\u7684\u5f15\u5165\u589e\u52a0\u4e86\u8ba1\u7b97\u91cf\uff0c\u5bf9\u786c\u4ef6\u8d44\u6e90\u7684\u8981\u6c42\u66f4\u9ad8\u3002<\/span><\/span><\/p>\n
                                  \uff083\uff09\u8d85\u53c2\u6570\u8c03\u4f18\u590d\u6742\uff1aLSTM \u5c42\u5f15\u5165\u4e86\u66f4\u591a\u7684\u8d85\u53c2\u6570\uff0c\u9700\u8981\u8fdb\u884c\u7cbe\u7ec6\u7684\u8c03\u4f18\u3002<\/span><\/span><\/p>\n

                                  <\/span><\/span><\/h2>\n

                                  <\/span><\/span><\/h2>\n
                                  \n
                                  \n
                                  \n
                                  \n
                                  \n

                                  <\/span><\/section>\n

                                  <\/span><\/section>\n
                                  \n
                                  <\/section>\n<\/section>\n<\/section>\n
                                  \n
                                  6. DQN \u7684\u5e94\u7528\u9886\u57df<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n
                                  \n

                                  <\/span><\/section>\n

                                  <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n

                                  <\/span><\/p>\n
                                  6.1 Atari \u6e38\u620f\u4e2d\u7684\u5e94\u7528\u4e0e\u8868\u73b0<\/span><\/span><\/span><\/h3>\n

                                  <\/span><\/span><\/h3>\n
                                  [1]<\/span><\/span>\u7684\u4f5c\u8005\u5728 49 \u6b3e Atria \u6e38\u620f\u4e0a\u8fdb\u884c\u6d4b\u8bd5\uff0c\u5927\u591a\u6570\u8fbe\u5230\u4e86\u4eba\u7c7b\u6c34\u5e73\uff0c\u5e76\u4e14\u6709 43 \u6b3e\u8fbe\u5230\u4e86 SOTA\uff08\u6700\u5148\u8fdb\u6c34\u5e73State-Of-The-Art\uff09\u3002\u56fe\u4e2d 100% \u548c 0% \u5206\u522b\u4ee3\u8868\u4eba\u7c7b\u6c34\u5e73\u548c\u968f\u673a\u7b56\u7565\u6c34\u5e73\u3002<\/span><\/span><\/span><\/h3>\n

                                  <\/span><\/span><\/h3>\n
                                  <\/section>\n
                                  \u6027\u80fd\u8868\u73b0\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n

                                  <\/span>\u6027\u80fd\u8868\u73b0<\/span><\/span><\/span><\/p>\n

                                  <\/span><\/span><\/p>\n
                                  DQN\u572849\u6b3eAtari 2600\u6e38\u620f\u4e0a\u7684\u6d4b\u8bd5\u7ed3\u679c\uff0c\u53c2\u89c1\uff0c\u5f15\u81ea<\/span><\/span>[1]<\/span><\/span>\u3002\u5176\u5c55\u73b0\u4e86\u7a81\u7834\u6027\u6027\u80fd\u3002\u5728\u7edf<\/span>\u4e00\u7f51\u7edc\u67b6\u6784\u3001\u7b97\u6cd5\u548c\u8d85\u53c2\u6570\u7684\u524d\u63d0\u4e0b\uff0c\u4ec5\u4ee5\u539f\u59cb\u50cf\u7d20\u548c\u6e38\u620f\u5206\u6570\u4e3a\u8f93\u5165\uff0cDQN\u572843\u6b3e\u6e38\u620f\u4e2d\u8d85\u8d8a\u4e86\u6b64\u524d\u6240\u6709\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\uff08\u5982\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u548cSARSA\uff09\u3002\u66f4\u663e\u8457\u7684\u662f\uff0c\u5176\u8868\u73b0\u8fbe\u5230\u4eba\u7c7b\u4e13\u4e1a\u73a9\u5bb6\u6c34\u5e73\u768475%\u4ee5\u4e0a\u7684\u6e38\u620f\u4e2d\u536029\u6b3e\uff08\u8d85\u8fc7\u603b\u6570\u534a\u6570\uff09\u3002\u5178\u578b\u6848\u4f8b\u5982\uff1a<\/span><\/span><\/p>\n
                                    \n
                                  • Breakout\u4e2d\uff0cDQN\u5f97\u5206\u8fbe401.2\uff08\u968f\u673a\u7b56\u7565\u4e3a1.7\uff0c\u4eba\u7c7b\u4e3a31.8\uff09\uff0c\u76f8\u5f53\u4e8e\u4eba\u7c7b\u6c34\u5e73\u76841327%\u3002\u5176\u6210\u529f\u6e90\u4e8e\u81ea\u4e3b\u5b66\u4f1a”\u6316\u96a7\u9053”\u7b56\u7565\uff1a\u4f18\u5148\u51fb\u788e\u4fa7\u8fb9\u7816\u5757\uff0c\u5f15\u5bfc\u7403\u4f53\u4ece\u540e\u65b9\u7834\u574f\u6574\u5217\u7816\u5757\u3002<\/span><\/span><\/span><\/li>\n
                                  • Pong\u4ee518.9\u5206\uff08\u4eba\u7c7b9.3\u5206\uff09\u5b9e\u73b0132%\u7684\u4eba\u7c7b\u6c34\u5e73\uff0c\u52a8\u4f5c\u4ef7\u503c\u9884\u6d4b\u7cbe\u51c6\uff1b<\/span><\/span><\/span><\/li>\n
                                  • Video Pinball\u66f4\u4ee542,684\u5206\uff08\u4eba\u7c7b17,298\u5206\uff09\u8fbe\u62102539%\u7684\u4eba\u7c7b\u6c34\u5e73\uff0c\u8bc1\u660e\u5bf9\u590d\u6742\u7269\u7406\u673a\u5236\u7684\u9ad8\u6548\u5229\u7528\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n

                                    <\/span><\/p>\n
                                    \u8bad\u7ec3\u8fc7\u7a0b\u7a33\u5b9a\u6027\u901a\u8fc7\u5e73\u5747\u6bcf\u5c40\u5f97\u5206\u548c\u9884\u6d4bQ\u503c\u66f2\u7ebf\u5f97\u5230\u9a8c\u8bc1\uff0c\u89c1\u56fe,<\/span><\/span><\/p>\n

                                    <\/span><\/p>\n
                                    <\/section>\n
                                    \u9884\u6d4bQ\u503c\u66f2\u7ebf\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n

                                    <\/span>\u4e8c\u8005\u5747\u968f\u8bad\u7a33\u6b65\u4e0a\u5347\uff0c\u672a\u51fa\u73b0\u4f20Q\u5b66\u4e60\u7684\u53d1\u6563\u73b0\u8c61\u3002\u5956\u52b1\u88c1\u526a\u7b56\u7565\uff08\u6b63\u5956\u52b1+1\uff0c\u8d1f\u5956\u52b1-1\uff09\u5219\u786e\u4fdd\u4e86\u4e0d\u540c\u6e38\u620f\u95f4\u5b66\u4e60\u7387\u7684\u4e00\u81f4\u6027\u3002<\/span><\/span><\/p>\n

                                    <\/span><\/p>\n
                                    \u6cdb\u5316\u4e0e\u8868\u5f81\u80fd\u529b<\/span><\/span><\/span><\/p>\n

                                    <\/span><\/span><\/p>\n
                                    DQN\u7684\u6838\u5fc3\u7a81\u7834\u5728\u4e8e\u5176\u4ece\u9ad8\u7ef4\u8f93\u5165\u4e2d\u5b66\u4e60\u901a\u7528\u8868\u5f81\u7684\u80fd\u529b\u3002<\/span><\/span><\/p>\n

                                    <\/span><\/p>\n
                                    \u901a\u8fc7t-SNE\u964d\u7ef4\u53ef\u89c6\u5316\uff0cDQN\u7684\u6700\u7ec8\u9690\u85cf\u5c42\u8868\u5f81\u663e\u793a\uff1a<\/span><\/span><\/p>\n

                                    <\/span><\/p>\n
                                    \u611f<\/span>\u77e5\u76f8\u4f3c\u72b6\u6001\uff08\u5982\u654c\u673a\u5bc6\u96c6\u6392\u5217\u7684\u5c4f\u5e55\uff09\u88ab\u6620\u5c04\u5230\u5d4c\u5165\u7a7a\u95f4\u7684\u76f8\u90bb\u4f4d\u7f6e\uff0c\u89c1\u56fe;<\/span><\/p>\n

                                    <\/span><\/p>\n
                                    <\/section>\n
                                    \u4eba\u7c7b\u8868\u5f81\u5bf9\u6bd4\uff0c\u5f15\u81ea<\/span><\/span><\/span>[1]<\/span><\/span><\/span><\/span><\/p>\n

                                    <\/span>\u5956\u52b1\u7b49\u4ef7\u4f46\u77e5\u4e0d\u540c\u72b6\u6001\uff08\u5982\u6e38\u620f\u5c3e\u9636\u6bb5\u7684\u63a9\u4f53\u573a\u666f\uff09\u540c\u6837\u88ab\u805a\u7c7b\u2014\u2014\u8fd9\u4e9b\u72b6\u6001\u56e0\u5373\u65f6\u5956\u52b1\u6f5c\u529b\u4f4e\u88ab\u8d4b\u4e88\u76f8\u4f3c\u4ef7\u503c\uff08V\u503c\uff09\uff0c\u8bc1\u660e\u7f51\u7edc\u8d85\u8d8a\u4e86\u8868\u5c42\u50cf\u7d20\u7279\u5f81\uff0c\u5b66\u4f1a\u4e86\u4e0e\u51b3\u7b56\u76f8\u5173\u7684\u62bd\u8c61\u7279\u5f81\u3002 \u8fdb\u4e00\u6b65\u5b9e\u9a8c\u8868\u660e\uff0c\u8be5\u8868\u5f81\u53ef\u6cdb\u5316\u81f3\u4eba\u7c7b\u73a9\u5bb6\u4ea7\u751f\u7684\u72b6\u6001\uff1a\u5c06\u4eba\u7c7b30\u5206\u949f\u6e38\u620f\u6570\u636e\u8f93\u5165DQN\u7f51\u7edc\uff0c\u5176t-SNE\u5d4c\u5165\u4e0e\u667a\u80fd\u4f53\u81ea\u8eab2\u5c0f\u65f6\u6e38\u620f\u6570\u636e\u7684\u5d4c\u5165\u7ed3\u6784\u9ad8\u5ea6\u91cd\u53e0\uff0c\u8bc1\u660e\u8868\u5f81\u4e0d\u53d7\u7b56\u7565\u4f9d\u8d56\u6027\u7684\u9650\u5236\u3002<\/span><\/span><\/p>\n

                                    <\/span>\u4ef7\u503c\u51fd\u6570\u7684\u9884\u6d4b\u80fd\u529b\u901a\u8fc7\u5177\u4f53\u6848\u4f8b\u5f97\u5230\u9a8c\u8bc1\uff1a<\/span><\/span><\/span><\/p>\n
                                      \n
                                    • Breakout\u4e2d\uff0c\u5f53\u667a\u80fd\u4f53\u5373\u5c06\u201c\u6316\u901a\u201d\u7816\u5899\u65f6\uff0cQ\u503c\u8dc3\u5347\u81f323\uff0c\u9884\u793a\u540e\u7eed\u81ea\u52a8\u6e05\u5c4f\u7684\u9ad8\u6536\u76ca\uff1b<\/span><\/span><\/span><\/li>\n
                                    • Pong\u4e2d\uff0c\u82e5\u7403\u4f53\u63a5\u8fd1\u7403\u62cd\u65f6\u9009\u62e9\u201c\u5411\u4e0b\u201d\u52a8\u4f5c\uff0cQ\u503c\u5b9e\u65f6\u964d\u81f3-0.9\uff0c\u51c6\u786e\u53cd\u6620\u4e22\u5931\u7403\u4f53\u7684\u98ce\u9669\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n

                                      <\/span><\/p>\n
                                      \u8fd9\u4e9b\u7ed3\u679c\u8bc1\u660e\uff0cDQN\u901a\u8fc7\u7aef\u5230\u7aef\u8bad\u7ec3\u6784\u5efa\u7684\u8868\u5f81\u80fd\u540c\u65f6\u652f\u6301\u611f\u77e5\u6cdb\u5316\u4e0e\u957f\u671f\u4ef7\u503c\u63a8\u7406\u3002<\/span><\/span><\/p>\n

                                      <\/span><\/span><\/h3>\n
                                      6.2 \u673a\u5668\u4eba\u63a7\u5236\u4e0e\u8def\u5f84\u89c4\u5212<\/span><\/span><\/span><\/h3>\n

                                      <\/span><\/span><\/p>\n
                                      \u53c2\u8003\u7efc\u8ff0<\/span><\/span>[7]<\/span><\/span>\u8fd9\u4e9b\u65b9\u6cd5\u90fd\u662f\u4f7f\u7528DQN\u76f4\u63a5\u4ece\u539f\u59cb\u4f20\u611f\u5668\u6570\u636e\u6620\u5c04\u5230\u52a8\u4f5c\uff0c\u65e0\u9700\u4f9d\u8d56\u9884\u5148\u6784\u5efa\u7684\u73af\u5883\u6a21\u578b\u6216\u5148\u9a8c\u77e5\u8bc6\u3002<\/span><\/span><\/span><\/p>\n
                                        \n
                                      • \n
                                        \u8f93\u5165\u7c7b\u578b\uff1a\u5305\u62ecRGB\u56fe\u50cf\u3001\u6df1\u5ea6\u56fe\u50cf\u3001\u6fc0\u5149\u96f7\u8fbe\u70b9\u4e91\u7b49\uff0c\u7528\u4e8e\u611f\u77e5\u73af\u5883\u72b6\u6001\u3002<\/span><\/span><\/section>\n<\/li>\n
                                      • \n
                                        \u8f93\u51fa\u52a8\u4f5c\uff1a\u79bb\u6563\u52a8\u4f5c\u7a7a\u95f4\uff08\u5982\u524d\u8fdb\u3001\u5de6\u8f6c\u3001\u53f3\u8f6c\uff09\uff0c\u9002\u7528\u4e8e\u5bfc\u822a\u548c\u907f\u969c\u4efb\u52a1\u3002<\/span><\/span><\/section>\n<\/li>\n<\/ul>\n

                                        <\/span><\/section>\n
                                        <\/section>\n

                                        <\/span>\u5178\u578b\u5e94\u7528\u6848\u4f8b<\/span><\/span><\/span><\/p>\n
                                          \n
                                        • \u5ba4\u5185\u5bfc\u822a\u4e0e\u907f\u969c\uff1aTai\u7b49\u4eba\uff082017\uff09<\/span>[8]\uff1a\u4f7f\u7528\u6df1\u5ea6\u56fe\u50cf\u4f5c\u4e3a\u8f93\u5165\uff0c\u8bad\u7ec3\u8f6e\u5f0f\u673a\u5668\u4eba\u5728\u8d70\u5eca\u73af\u5883\u4e2d\u5b9e\u73b0\u65e0\u5730\u56fe\u5bfc\u822a\uff0c\u8f93\u51fa\u8fde\u7eed\u8f6c\u5411\u6307\u4ee4\u3002Zhang\u7b49\u4eba\uff082020\uff09<\/span>[9]\uff1a\u901a\u8fc7\u7279\u5f81\u8fc1\u79fb\u7b56\u7565\uff0c\u5c06\u4eff\u771f\u73af\u5883\u8bad\u7ec3\u7684DQN\u6a21\u578b\u8fc1\u79fb\u81f3\u771f\u5b9e\u673a\u5668\u4eba\uff0c\u89e3\u51b3\u4eff\u771f\u5230\u73b0\u5b9e\uff08Sim-to-Real\uff09\u7684\u6cdb\u5316\u95ee\u9898\u3002Barron\u7b49\u4eba\uff082019\uff09<\/span>[10]\uff1a\u57283D\u865a\u62df\u73af\u5883\u4e2d\uff0c\u91c7\u7528RGB\u56fe\u50cf\u8f93\u5165\u8bad\u7ec3\u673a\u5668\u4eba\uff0c\u5728\u590d\u6742\u81ea\u7136\u573a\u666f\u4e2d\u5b9e\u73b0\u4f18\u4e8e\u6df1\u5ea6\u8f93\u5165\u7684\u5bfc\u822a\u6027\u80fd\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n

                                          <\/span><\/p>\n
                                            \n
                                          • \u672a\u77e5\u73af\u5883\u63a2\u7d22\uff1aHaora\u7b49\u4eba\uff082020\uff09[11]\uff1a\u63d0\u51fa\u81ea\u9002\u5e94\u5168\u5377\u79efQ\u7f51\u7edc\uff08AFCQN\uff09\uff0c\u7ed3\u5408\u5c40\u90e8\u5730\u56fe\u4fe1\u606f\uff0c\u663e\u8457\u964d\u4f4e\u672a\u77e5\u73af\u5883\u4e2d\u7684\u5bfc\u822a\u5931\u8d25\u7387\u3002<\/span><\/span><\/span><\/li>\n<\/ul>\n

                                            <\/span><\/p>\n
                                              \n
                                            • \u65e0\u4eba\u673a\u63a7\u5236\uff1aKang\u7b49\u4eba\uff082019\uff09[12]\uff1a\u4ec5\u7528\u5355\u76ee\u89c6\u89c9\u8f93\u5165\uff0c\u8bad\u7ec3\u56db\u65cb\u7ffc\u65e0\u4eba\u673a\u5728\u771f\u5b9e\u73af\u5883\u4e2d\u5b9e\u73b0\u52a8\u6001\u907f\u969c\u3002<\/span><\/span><\/span>
                                              <\/span><\/span><\/span><\/li>\n<\/ul>\n

                                              <\/span><\/span><\/section>\n
                                              \u5f53\u7136\u4e5f\u5b58\u5728\u4e00\u4e9b\u5c40\u9650\u6027\uff0c\u6bd4\u5982DQN \u4ec5\u652f\u6301\u79bb\u6563\u52a8\u4f5c\uff0c\u65e0\u6cd5\u76f4\u63a5\u5e94\u7528\u4e8e\u8fde\u7eed\u63a7\u5236\uff08\u5982\u65e0\u4eba\u673a\u7cbe\u7ec6\u59ff\u6001\u8c03\u6574\uff09\uff0c\u9700\u4f9d\u8d56DDPG\u7b49\u7b97\u6cd5\u8865\u5145\u3002\u8bad\u7ec3\u9700\u5927\u91cf\u4ea4\u4e92\u6570\u636e\uff0c\u73b0\u5b9e\u573a\u666f\u4e2d\u6570\u636e\u91c7\u96c6\u6210\u672c\u9ad8\u3002\u7a00\u758f\u5956\u52b1\u73af\u5883\u4e0b\uff08\u5982\u76ee\u6807\u8ddd\u79bb\u8fdc\uff09\uff0c\u5b66\u4e60\u6548\u7387\u4e0b\u964d\u3002\u5c31\u662fDQN \u901a\u8fc7\u7aef\u5230\u7aef\u5b66\u4e60\u4e0e\u7ecf\u9a8c\u56de\u653e\u673a\u5236\uff0c\u5728\u79fb\u52a8\u673a\u5668\u4eba\u89c6\u89c9\u5bfc\u822a\u3001\u5ba4\u5185\u907f\u969c\u53ca\u65e0\u4eba\u673a\u63a7\u5236\u4e2d\u53d6\u5f97\u663e\u8457\u6210\u679c\u3002\u5176\u6838\u5fc3\u4ef7\u503c\u5728\u4e8e\u5904\u7406\u9ad8\u7ef4\u611f\u77e5\u6570\u636e\u7684\u80fd\u529b\u548c\u7b97\u6cd5\u7a33\u5b9a\u6027\uff0c\u4f46\u9700\u7ed3\u5408\u6539\u8fdb\u6280\u672f\uff08\u5982Double DQN\u3001Rainbow\uff09\u4ee5\u63d0\u5347\u6027\u80fd\u3002\u672a\u6765\u7814\u7a76\u9700\u89e3\u51b3\u8fde\u7eed\u52a8\u4f5c\u652f\u6301\u4e0e\u6837\u672c\u6548\u7387\u95ee\u9898\uff0c\u4ee5\u6269\u5927\u5e94\u7528\u573a\u666f\u3002<\/span><\/span><\/span><\/section>\n
                                              6.3 \u63a8\u8350\u7cfb\u7edf\u4e2d\u7684\u7b56\u7565\u4f18\u5316<\/span><\/span><\/span><\/h3>\n

                                              <\/span><\/span><\/h3>\n
                                              \u6587\u7ae0<\/span><\/span>[13]<\/span><\/span>\u63d0\u4f9b\u4e86\u5f3a\u5316\u5b66\u4e60\u5728\u63a8\u8350\u7cfb\u7edf\u4e2d\u7684\u5e94\u7528\uff0c\u5177\u4f53\u601d\u8def\u6d41\u7a0b\u89c1\u53f3\u56fe\u6240\u793a\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              <\/span><\/span><\/p>\n
                                              \u57fa\u4e8e\u5f3a\u5316\u5b66\u4e60\u7684\u63a8\u8350\u7cfb\u7edf\uff0c\u5f15\u81ea<\/span><\/span><\/span>[13]<\/span><\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u5176\u4e2d\uff0c\u6df1\u5ea6Q\u7f51\u7edc\uff08DQN\uff09\u901a\u8fc7\u5efa\u6a21\u7528\u6237\u52a8\u6001\u504f\u597d\u548c\u957f\u671f\u4ef7\u503c\uff0c\u89e3\u51b3\u4e86\u4f20\u7edf\u63a8\u8350\u7cfb\u7edf\u9762\u4e34\u7684\u63a2\u7d22-\u5229\u7528\u56f0\u5883\u548c\u5ef6\u8fdf\u5956\u52b1\u4f18\u5316\u95ee\u9898\u3002\u5176\u6838\u5fc3\u601d\u60f3\u662f\u5c06\u63a8\u8350\u89c6\u4e3a\u5e8f\u5217\u51b3\u7b56\u8fc7\u7a0b\uff1a\u7528\u6237\u72b6\u6001 <\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\u4e3a\u5386\u53f2\u4ea4\u4e92\u7279\u5f81\uff0c\u52a8\u4f5c <\/span><\/span>a<\/span><\/span>t<\/span><\/span><\/sub>\u4e3a\u5f85\u63a8\u8350\u7269\u54c1\uff0c\u5956\u52b1 <\/span><\/span>r<\/span><\/span>t<\/span><\/span><\/sub>\u4e3a\u70b9\u51fb\/\u8f6c\u5316\u7b49\u53cd\u9988\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span>\u95ee\u9898\u5efa\u6a21\u4e0e\u4f18\u52bf<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u52a8\u6001\u504f\u597d\u6355\u6349\uff08vs. \u9759\u6001\u534f\u540c\u8fc7\u6ee4\uff09\uff1a DQN\u901a\u8fc7\u65f6\u5e8f\u72b6\u6001\u8f6c\u79fb\uff08 <\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\u2192<\/span><\/span>s<\/span><\/span>t+1<\/span><\/sub>\uff09\u5b9e\u65f6\u66f4\u65b0\u7528\u6237\u5174\u8da3\u8868\u5f81\uff0c\u514b\u670d\u534f\u540c\u8fc7\u6ee4\u7684\u9759\u6001\u5efa\u6a21\u5c40\u9650\u3002<\/span><\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/span><\/p>\n
                                              \u957f\u671f\u4ef7\u503c\u4f18\u5316\uff08vs. \u5373\u65f6\u6536\u76ca\u6a21\u578b\uff09\uff1a\u6700\u5927\u5316\u7d2f\u8ba1\u5956\u52b1<\/span><\/span>\"image.png\"<\/span><\/sub>\uff0c\u907f\u514d\u77ed\u89c6\u7b56\u7565\uff08\u5982\u4ec5\u63a8\u8350\u70ed\u95e8\u5546\u54c1\uff09\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u63a2\u7d22\u673a\u5236\uff1a\u03b5-greedy\u7b56\u7565\u4e3b\u52a8\u63a8\u8350\u4f4e\u66dd\u5149\u5546\u54c1\uff0c\u7f13\u89e3\u51b7\u542f\u52a8\u95ee\u9898\u3002<\/span><\/span><\/p>\n

                                              <\/span>\u6280\u672f\u5b9e\u73b0\u67b6\u6784<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u72b6\u6001\u8868\u5f81\uff1a\u7528\u6237\u753b\u50cf\uff08\u6027\u522b\/\u5e74\u9f84\uff09 + \u884c\u4e3a\u5e8f\u5217\uff08\u70b9\u51fb\/\u8d2d\u4e70\u5386\u53f2\uff09 \u2192 \u5d4c\u5165\u5c42 \u2192 LSTM\u7f16\u7801 \u2192 \u72b6\u6001\u5411\u91cf <\/span><\/span>s<\/span><\/span>t<\/span><\/span><\/sub>\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u52a8\u4f5c\u7a7a\u95f4\uff1a\u79bb\u6563\u5019\u9009\u7269\u54c1\u96c6 <\/span><\/span>a<\/span><\/span>1<\/span><\/span><\/sub>,<\/span><\/span>a<\/span><\/span>2<\/span><\/span><\/sub>,<\/span><\/span>.<\/span><\/span>.<\/span><\/span>.<\/span><\/span>,<\/span><\/span>a<\/span><\/span>N<\/span><\/span><\/sub>\uff08N=\u5343\u81f3\u767e\u4e07\u7ea7\uff09\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \u7f51\u7edc\u7ed3\u6784\u5982\u53f3\u56fe\u6240\u793a\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              <\/span><\/p>\n
                                              \u57fa\u4e8edqn\u7684\u63a8\u8350\u7cfb\u7edf\u7b97\u6cd5\uff0c\u5f15\u81ea<\/span><\/span><\/span>[13]<\/span><\/span><\/span><\/span><\/p>\n

                                              <\/span><\/p>\n

                                              <\/span><\/span><\/h2>\n
                                              \n
                                              \n
                                              \n
                                              \n
                                              \n

                                              <\/span><\/section>\n

                                              <\/span><\/section>\n
                                              \n
                                              <\/section>\n<\/section>\n<\/section>\n
                                              \n
                                              7. DQN \u7684\u5c40\u9650\u6027\u4e0e\u6311\u6218<\/span><\/strong><\/span><\/strong><\/strong><\/p>\n<\/section>\n
                                              \n

                                              <\/span><\/section>\n

                                              <\/span><\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n<\/h3>\n

                                              <\/span><\/p>\n
                                              7.1 \u5bf9\u8fde\u7eed\u52a8\u4f5c\u7a7a\u95f4\u7684\u5904\u7406\u56f0\u96be<\/span><\/span><\/span><\/h3>\n

                                              <\/span><\/span><\/h3>\n
                                              DQN \u4f5c\u4e3a\u4e00\u4e2a\u57fa\u4e8e\u79bb\u6563\u52a8\u4f5c\u7a7a\u95f4\u7684\u7b97\u6cd5\uff0c\u5728\u9762\u5bf9\u8fde\u7eed\u52a8\u4f5c\u7a7a\u95f4\u65f6\u8868\u73b0\u4e0d\u4f73\u3002\u4f20\u7edf\u7684 DQN \u5c06\u52a8\u4f5c\u7a7a\u95f4\u79bb\u6563\u5316\u4e3a\u6709\u9650\u7684\u51e0\u79cd\u9009\u62e9\uff0c\u5e76\u5bf9\u6bcf\u4e00\u4e2a\u52a8\u4f5c\u5206\u522b\u8fdb\u884c Q \u503c\u66f4\u65b0\u3002\u7136\u800c\uff0c\u5728\u5b9e\u9645\u5e94\u7528\u4e2d\uff0c\u8bb8\u591a\u4efb\u52a1\u8981\u6c42\u5728\u8fde\u7eed\u7684\u52a8\u4f5c\u7a7a\u95f4\u4e2d\u8fdb\u884c\u9009\u62e9\uff08\u5982\u673a\u5668\u4eba\u63a7\u5236\u3001\u81ea\u52a8\u9a7e\u9a76\u7b49\uff09\uff0c\u8fd9\u4f7f\u5f97 DQN \u96be\u4ee5\u76f4\u63a5\u5e94\u7528\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\uff0c\u7814\u7a76\u8005\u4eec\u63d0\u51fa\u4e86\u5982\u6df1\u5ea6\u786e\u5b9a\u6027\u7b56\u7565\u68af\u5ea6\uff08DDPG\uff09<\/span><\/span>[14]<\/span><\/span>\u3001\u8f6f\u884c\u4e3a\u8005-\u8bc4\u8bba\u5bb6\uff08SAC\uff09\u7b49\u7b97\u6cd5\uff0c\u8fd9\u4e9b\u7b97\u6cd5\u4f7f\u7528\u8fde\u7eed\u7684\u7b56\u7565\u8f93\u51fa\uff0c\u5e76\u7ed3\u5408\u7b56\u7565\u68af\u5ea6\u65b9\u6cd5\u6765\u4f18\u5316\u8fde\u7eed\u52a8\u4f5c\u7a7a\u95f4\u4e2d\u7684\u51b3\u7b56\u8fc7\u7a0b\u3002\u7136\u800c\uff0cDQN \u5728\u5904\u7406\u8fde\u7eed\u52a8\u4f5c\u7a7a\u95f4\u65f6\u7684\u9002\u5e94\u6027\u4ecd\u7136\u662f\u4e00\u4e2a\u4e9f\u5f85\u89e3\u51b3\u7684\u96be\u9898\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              7.2 \u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u6837\u672c\u6548\u7387\u95ee\u9898<\/span><\/span><\/span><\/h3>\n

                                              <\/span><\/span><\/h3>\n
                                              DQN \u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u9700\u8981\u5927\u91cf\u7684\u4e0e\u73af\u5883\u7684\u4ea4\u4e92\u6570\u636e\uff0c\u624d\u80fd\u6709\u6548\u5730\u5b66\u4e60\u5230\u6700\u4f18\u7b56\u7565\uff0c\u8fd9\u4f7f\u5f97\u6837\u672c\u6548\u7387\u4f4e\u4e0b\u3002\u4e3a\u4e86\u51cf\u5c11\u8bad\u7ec3\u65f6\u95f4\uff0cDQN \u5f15\u5165\u4e86\u7ecf\u9a8c\u56de\u653e\uff08Experience Replay\uff09\u673a\u5236\uff0c\u5c06\u5386\u53f2\u7ecf\u9a8c\u5b58\u50a8\u5728\u56de\u653e\u7f13\u51b2\u533a\uff0c\u5e76\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u968f\u673a\u62bd\u53d6\u6837\u672c\u8fdb\u884c\u5b66\u4e60\u3002\u867d\u7136\u8fd9\u79cd\u65b9\u6cd5\u80fd\u6709\u6548\u7f13\u89e3\u76f8\u5173\u6027\u5e26\u6765\u7684\u95ee\u9898\uff0c\u4f46\u5b83\u5e76\u4e0d\u80fd\u4ece\u6839\u672c\u4e0a\u89e3\u51b3\u6837\u672c\u6548\u7387\u95ee\u9898\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u4f4e\u6837\u672c\u6548\u7387\u4f7f\u5f97 DQN \u5728\u9700\u8981\u5feb\u901f\u53cd\u5e94\u6216\u4ea4\u4e92\u53d7\u9650\u7684\u73af\u5883\u4e2d\uff08\u5982\u73b0\u5b9e\u4e16\u754c\u4e2d\u7684\u673a\u5668\u4eba\u63a7\u5236\uff09\u8868\u73b0\u4e0d\u7406\u60f3\u3002\u6b64\u5916\uff0c\u8bad\u7ec3\u6570\u636e\u7684\u9ad8\u9700\u6c42\u589e\u52a0\u4e86\u8ba1\u7b97\u5f00\u9500\uff0c\u5e76\u4e14\u53ef\u80fd\u5bfc\u81f4\u7b97\u6cd5\u5728\u5b9e\u9645\u4efb\u52a1\u4e2d\u7684\u5e94\u7528\u4e0d\u591f\u7075\u6d3b\u3002\u6587\u7ae0<\/span><\/span>[15]<\/span><\/span>\u63d0\u51fa\u4e86Rainbow DQN\uff0c\u5c06\u591a\u79cd DQN \u7684\u6539\u8fdb\u6280\u672f\u878d\u5408\uff0c\u4ee5\u63d0\u9ad8 DQN \u7684\u7a33\u5b9a\u6027\u548c\u6837\u672c\u6548\u7387\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              7.3 \u7b56\u7565\u6536\u655b\u901f\u5ea6\u6162\u4e0e\u7a33\u5b9a\u6027\u95ee\u9898<\/span><\/span><\/span><\/h3>\n

                                              <\/span><\/span><\/h3>\n
                                              DQN \u7684\u8bad\u7ec3\u8fc7\u7a0b\u901a\u5e38\u5b58\u5728\u6536\u655b\u901f\u5ea6\u6162\u548c\u8bad\u7ec3\u4e0d\u7a33\u5b9a\u7684\u95ee\u9898\u3002\u7531\u4e8e Q \u503c\u51fd\u6570\u662f\u901a\u8fc7\u795e\u7ecf\u7f51\u7edc\u903c\u8fd1\u7684\uff0c\u800c\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u672c\u8eab\u5bb9\u6613\u51fa\u73b0\u68af\u5ea6\u7206\u70b8\u3001\u68af\u5ea6\u6d88\u5931\u7b49\u95ee\u9898\uff0c\u8fd9\u4f1a\u5bfc\u81f4\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u4e0d\u7a33\u5b9a\u3002\u6b64\u5916\uff0cQ \u503c\u66f4\u65b0\u8fc7\u7a0b\u4e2d\u7684\u76ee\u6807\u503c\uff08\u5373\u6700\u5927\u5316\u7684 Q \u503c\uff09\u901a\u5e38\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u5bfc\u81f4\u76ee\u6807\u7f51\u7edc\u548c\u884c\u4e3a\u7f51\u7edc\u4e4b\u95f4\u7684\u4e0d\u4e00\u81f4\uff0c\u8fd9\u4f1a\u8fdb\u4e00\u6b65\u52a0\u5267\u8bad\u7ec3\u7684\u9707\u8361\u548c\u4e0d\u7a33\u5b9a\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u4e3a\u4e86\u63d0\u9ad8\u7a33\u5b9a\u6027\uff0cDQN \u5f15\u5165\u4e86\u76ee\u6807\u7f51\u7edc\uff08Target Network\uff09\u548c\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u3002\u76ee\u6807\u7f51\u7edc\u7684\u5f15\u5165\u4f7f\u5f97 Q \u503c\u7684\u66f4\u65b0\u76ee\u6807\u5728\u4e00\u5b9a\u65f6\u95f4\u5185\u4fdd\u6301\u4e0d\u53d8\uff0c\u4ece\u800c\u907f\u514d\u4e86\u76ee\u6807\u7f51\u7edc\u7684\u4e0d\u65ad\u53d8\u5316\u5bf9\u8bad\u7ec3\u8fc7\u7a0b\u7684\u5f71\u54cd\u3002\u6587\u7ae0<\/span><\/span>[16]<\/span><\/span>\u4ecb\u7ecd\u4e86 PPO \u7b97\u6cd5\uff0c\u89e3\u51b3\u4e86\u7b56\u7565\u4f18\u5316\u4e2d\u7a33\u5b9a\u6027\u5dee\u548c\u6536\u655b\u901f\u5ea6\u6162\u7684\u95ee\u9898\u3002\u7136\u800c\uff0c\u5c3d\u7ba1\u8fd9\u4e9b\u6280\u672f\u6539\u5584\u4e86\u7a33\u5b9a\u6027\uff0cDQN \u7684\u6536\u655b\u901f\u5ea6\u548c\u8bad\u7ec3\u8fc7\u7a0b\u7684\u7a33\u5b9a\u6027\u95ee\u9898\u4f9d\u7136\u5b58\u5728\uff0c\u7279\u522b\u662f\u5728\u590d\u6742\u7684\u73af\u5883\u4e2d\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              7.4 \u6cdb\u5316\u80fd\u529b\u4e0e\u8fc7\u62df\u5408\u98ce\u9669<\/span><\/span><\/span><\/h3>\n

                                              <\/span><\/span><\/h3>\n
                                              DQN \u5728\u9762\u5bf9\u9ad8\u7ef4\u72b6\u6001\u7a7a\u95f4\u548c\u590d\u6742\u73af\u5883\u65f6\uff0c\u5bb9\u6613\u51fa\u73b0\u8fc7\u62df\u5408\u95ee\u9898\u3002\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\uff0cDQN \u901a\u8fc7\u5bf9\u5927\u91cf\u7684\u73af\u5883\u4ea4\u4e92\u8fdb\u884c\u5b66\u4e60\u6765\u66f4\u65b0 Q \u503c\u51fd\u6570\u3002\u7136\u800c\uff0c\u5728\u6709\u9650\u7684\u8bad\u7ec3\u6570\u636e\u4e0a\uff0cDQN \u53ef\u80fd\u4f1a\u8fc7\u5ea6\u62df\u5408\u67d0\u4e9b\u7279\u5b9a\u7684\u72b6\u6001\u548c\u52a8\u4f5c\uff0c\u4ece\u800c\u5931\u53bb\u5bf9\u672a\u77e5\u73af\u5883\u6216\u672a\u89c1\u8fc7\u72b6\u6001\u7684\u9002\u5e94\u80fd\u529b\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u8fd9\u79cd\u8fc7\u62df\u5408\u7684\u73b0\u8c61\u7279\u522b\u660e\u663e\u5728\u9ad8\u7ef4\u7a7a\u95f4\u6216\u591a\u53d8\u73af\u5883\u4e2d\uff0c\u667a\u80fd\u4f53\u7684\u6cdb\u5316\u80fd\u529b\u53d7\u5230\u9650\u5236\u3002\u5f53\u667a\u80fd\u4f53\u5728\u4e00\u4e2a\u6709\u9650\u7684\u8bad\u7ec3\u73af\u5883\u4e2d\u83b7\u5f97\u8bad\u7ec3\u540e\uff0c\u5176\u5b66\u5230\u7684\u7b56\u7565\u5728\u65b0\u7684\u6216\u7565\u5fae\u4e0d\u540c\u7684\u73af\u5883\u4e2d\u8868\u73b0\u8f83\u5dee\u3002\u4e3a\u4e86\u89e3\u51b3\u8fd9\u4e00\u95ee\u9898\uff0c\u63d0\u51fa\u4e86\u66f4\u5148\u8fdb\u7684\u6280\u672f\uff0c\u5982\u6b63\u5219\u5316\u3001\u81ea\u52a8\u8c03\u6574\u7f51\u7edc\u7ed3\u6784\u7b49\u65b9\u6cd5\uff0c\u4ee5\u589e\u5f3a\u6a21\u578b\u7684\u6cdb\u5316\u80fd\u529b\uff0c\u4f46\u8fd9\u4e9b\u65b9\u6cd5\u4ecd\u672a\u80fd\u5b8c\u5168\u6d88\u9664 DQN \u5728\u590d\u6742\u73af\u5883\u4e2d\u9762\u4e34\u7684\u8fc7\u62df\u5408\u98ce\u9669\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              <\/span><\/p>\n

                                              <\/span><\/p>\n

                                              <\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              \n
                                              \n
                                              \n
                                              \u53c2\u8003\u6587\u732e<\/span><\/strong><\/p>\n<\/section>\n<\/section>\n<\/section>\n
                                              \n
                                              \n
                                              \n
                                              \n

                                              <\/span><\/p>\n
                                              1. Mnih, V.; Kavukcuoglu, K. (2015). “Human-level control through deep reinforcement learning”. Nature. 518 (7540): 529\u2013533. doi:10.1038\/nature14236. PMID 25719670.<\/span><\/span><\/p>\n
                                              2. Overmars M G. Analyzing the convergence of Q-learning through Markov decision theory[D]. University of Twente, 2021.<\/span><\/span><\/p>\n
                                              3. Even-Dar E, Mansour Y. Convergence of optimistic and incremental Q-learning[J]. Advances in neural information processing systems, 2001, 14.<\/span><\/span><\/p>\n
                                              4. Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning[C]\/\/Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).<\/span><\/span><\/p>\n
                                              5. Wang Z, Schaul T, Hessel M, et al. Dueling network architectures for deep reinforcement learning[C]\/\/International conference on machine learning. PMLR, 2016: 1995-2003.<\/span><\/span><\/p>\n
                                              6. Hausknecht M J, Stone P. Deep Recurrent Q-Learning for Partially Observable MDPs[C]\/\/AAAI fall symposia. 2015, 45: 141.<\/span><\/span><\/p>\n
                                              7. Sun, Huihui; Zhang, Weijie; Yu, Runxiang; Zhang, Yujie (2021). “Motion planning for mobile robots\u2014Focusing on deep reinforcement learning: A systematic review”. IEEE Access. IEEE. 9: 69061\u201369081. doi:10.1109\/ACCESS.2021.3076530.<\/span><\/span><\/p>\n
                                              8. Tai, Lei; Paolo, Giuseppe; Liu, Ming (2017). “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation”. IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE: 31\u201336. doi:10.1109\/IROS.2017.8202134.<\/span><\/span><\/p>\n
                                              9. Wang, Y.; Sun, J.; He, H.; Sun, C. (2020). “Deterministic policy gradient with integral compensator for robust quadrotor control”. IEEE Transactions on Systems, Man, and Cybernetics: Systems. IEEE. 50 (10): 3713\u20133725. doi:10.1109\/TSMC.2020.3002218.<\/span><\/span><\/p>\n
                                              10. Barron, T.; Whitehead, M.; Yeung, A. (2016). Deep reinforcement learning in a 3-D blockworld environment. IJCAI\/AAAI Press. pp. 1\u20136. {{cite conference}}: Unknown parameter |booktitle= ignored (help)<\/span><\/span><\/p>\n
                                              11. Li, H.; Zhang, Q.; Zhao, D. (2020). “Deep reinforcement learning-based automatic exploration for navigation in unknown environment”. IEEE Transactions on Neural Networks and Learning Systems. IEEE. 31 (6): 2064\u20132076. doi:10.1109\/TNNLS.2019.2930059.<\/span><\/span><\/p>\n
                                              12. Kang, Kyunghoon; Belkhale, Suneel; Kahn, Gregory; Abbeel, Pieter; Levine, Sergey (2019). “Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight”. IEEE International Conference on Robotics and Automation (ICRA). IEEE: 6008\u20136014. doi:10.1109\/ICRA.2019.8793701.<\/span><\/span><\/p>\n
                                              13. {Chen, Xiaocong and Yao, Lina and McAuley, Julian and Zhou, Guanglin and Wang, Xianzhi} (2021). “A survey of deep reinforcement learning in recommender systems: A systematic review and future directions”. arXiv preprint: arXiv:2109.03540.<\/span><\/span><\/p>\n
                                              14. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., … & Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.<\/span><\/span><\/p>\n
                                              15. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., … & Silver, D. (2018, April). Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).<\/span><\/span><\/p>\n
                                              16. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.<\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n
                                              \n
                                              \n
                                              \u53c2\u8003\u6587\u732e\u53ef<\/span>\u4e0a\u4e0b\u6ed1\u52a8<\/span><\/strong>\u67e5\u770b<\/span><\/p>\n<\/section>\n<\/section>\n

                                              <\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u4f5c\u8005\u7b80\u4ecb<\/span><\/span><\/p>\n
                                              <\/section>\n
                                              <\/section>\n

                                              <\/span><\/section>\n
                                              \n
                                              \u672c\u8bcd\u6761\u7531\u96c6\u667a\u4ff1\u4e50\u90e8\u4f17\u5305\u751f\u4ea7\uff0c\u96be\u514d\u5b58\u5728\u7eb0\u6f0f\u548c\u95ee\u9898\uff0c\u6b22\u8fce\u5927\u5bb6\u7559\u8a00\u53cd\u9988\uff0c\u4e00\u7ecf\u91c7\u7eb3\uff0c\u53ef\u4ee5\u83b7\u5f97\u5bf9\u5e94\u7684\u79ef\u5206\u5956\u52b1\u5662\uff01<\/span><\/p>\n<\/blockquote>\n

                                              <\/span><\/span><\/section>\n

                                              <\/span><\/section>\n
                                              \n
                                              \n
                                              \n
                                              \n
                                              \u52a0\u5165\u6211\u4eec<\/span><\/strong><\/p>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n

                                              <\/span><\/section>\n
                                              \u4eb2\u7231\u7684\u793e\u533a\u4f19\u4f34\u4e0e\u77e5\u8bc6\u63a2\u7d22\u8005\uff1a<\/span><\/span><\/strong><\/p>\n
                                              \u6211\u4eec\u8bda\u631a\u9080\u8bf7\u70ed\u7231\u77e5\u8bc6\u5206\u4eab\u7684\u60a8\uff0c\u52a0\u5165<\/span><\/span>\u96c6\u667a\u767e\u79d1\u8bcd\u6761\u7f16\u5199\u5fd7\u613f\u56e2\u961f<\/span><\/span><\/strong>\uff01\u65e0\u8bba\u60a8\u662f\u9886\u57df\u4e13\u5bb6\uff0c\u8fd8\u662f\u5bf9\u7279\u5b9a\u4e3b\u9898\u5145\u6ee1\u70ed\u5ff1\u7684\u5b66\u4e60\u8005\uff0c\u8fd9\u91cc\u90fd\u6709\u60a8\u7684\u821e\u53f0\u3002\u901a\u8fc7\u7f16\u5199\u767e\u79d1\u8bcd\u6761\uff0c\u60a8\u5c06\u4e3a\u5168\u7403\u8bfb\u8005\u4f20\u9012\u6743\u5a01\u77e5\u8bc6\uff0c\u540c\u65f6\u83b7\u5f97<\/span><\/span>\u4e13\u5bb6\u6307\u5bfc<\/span><\/span><\/strong>\u4e0e<\/span><\/span>\u4e2a\u4eba\u80fd\u529b<\/span><\/span><\/strong>\u8dc3\u5347\u7684\u53cc\u91cd\u6210\u957f\u3002<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \ud83d\udcdd <\/span><\/span>\u5fd7\u613f\u8005\u804c\u8d23<\/span><\/span><\/strong><\/p>\n
                                              \u521b\u4f5c\u65b0\u8bcd\u6761\uff1a<\/span><\/span><\/strong>\u8986\u76d6\u590d\u6742\u7cfb\u7edf\u3001\u4eba\u5de5\u667a\u80fd\u7b49\u524d\u6cbf\u9886\u57df<\/span><\/span><\/p>\n
                                              \u8fed\u4ee3\u7ecf\u5178\u5185\u5bb9<\/span><\/span><\/strong>\uff1a\u66f4\u65b0\u73b0\u6709\u8bcd\u6761\uff0c\u5b88\u62a4\u77e5\u8bc6\u7684\u51c6\u786e\u6027\u4e0e\u65f6\u6548\u6027<\/span><\/span><\/p>\n
                                              \u8d28\u91cf\u5b88\u62a4\u8005<\/span><\/span><\/strong>\uff1a\u53c2\u4e0e\u5185\u5bb9\u6821\u5bf9\u5ba1\u6838\uff0c\u5171\u5efa\u7cbe\u54c1\u77e5\u8bc6\u5e93<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \ud83c\udf1f <\/span><\/span>\u6211\u4eec\u671f\u5f85\u60a8<\/span><\/span><\/strong><\/p>\n
                                              \u96c6\u667a\u8bfb\u4e66\u4f1a\u6210\u5458\uff08\u9700\u5b8c\u6210\u5171\u521b\u4efb\u52a1\u5e76\u83b7\u5f97\u9000\u8d39\u8d44\u683c\uff09<\/span><\/span><\/p>\n
                                              \u62e5\u6709\u6e05\u6670\u8868\u8fbe\u590d\u6742\u6982\u5ff5\u7684\u5199\u4f5c\u80fd\u529b<\/span><\/span><\/p>\n
                                              \u5bf9\u7279\u5b9a\u9886\u57df\u6709\u6df1\u5ea6\u7814\u7a76\u6216\u5f3a\u70c8\u5174\u8da3<\/span><\/span><\/p>\n
                                              \u5177\u5907\u4fe1\u606f\u68c0\u7d22\u4e0e\u6574\u5408\u7d20\u517b<\/span><\/span><\/p>\n
                                              \u6000\u63e3\u8d23\u4efb\u611f\u4e0e\u534f\u4f5c\u7cbe\u795e\uff0c\u613f\u4e3a\u77e5\u8bc6\u5171\u4eab\u8d4b\u80fd<\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \ud83c\udf81 <\/span><\/span>\u60a8\u5c06\u6536\u83b7<\/span><\/span><\/strong><\/p>\n
                                              \u767e\u79d1\u79ef\u5206\uff08\u652f\u6301\u5151\u6362\u96c6\u667a\u4ff1\u4e50\u90e8\u5468\u8fb9\uff1a\u6587\u5316\u886b\u3001\u590d\u6742\u79d1\u5b66\u77e5\u8bc6\u5361\u7b49\uff09<\/span><\/span><\/p>\n
                                              \u96c6\u667a\u4ff1\u4e50\u90e8\u521b\u59cb\u4eba\u5f20\u6c5f\u6559\u6388\u4eb2\u81ea\u6307\u5bfc\u5199\u4f5c<\/span><\/span><\/p>\n
                                              \u79d1\u7814\u52a9\u7406\u664b\u5347\u901a\u9053\uff1a\u8868\u73b0\u4f18\u5f02\u8005\u53ef\u52a0\u5165\u5f20\u6c5f\u6559\u6388\u79d1\u7814\u56e2\u961f<\/span><\/span><\/p>\n
                                              \"\u56fe\u7247\"<\/section>\n
                                              <\/section>\n
                                              \u52a0\u5165\u5f20\u6c5f\u7ec4\u53d1\u8d77\u7684\u4efb\u4f55\u4e00\u671f\u8bfb\u4e66\u4f1a\u5e76\u5b8c\u6210\u5171\u521b\u4efb\u52a1\u5373\u53ef \u8ba9\u6211\u4eec\u4e00\u8d77\uff0c\u7528\u77e5\u8bc6\u8fde\u63a5\u4e16\u754c\uff01<\/span><\/span><\/p>\n
                                              \n

                                              <\/span><\/span><\/strong><\/span><\/h1>\n

                                              <\/span><\/span><\/strong><\/span><\/p>\n
                                              \n
                                              \n
                                              \n
                                              \n
                                              \u81ea\u7531\u80fd\u539f\u7406\u4e0e\u5f3a\u5316\u5b66\u4e60\u8bfb\u4e66\u4f1a<\/span><\/strong><\/p>\n<\/section>\n<\/section>\n<\/section>\n<\/section>\n

                                              <\/span><\/p>\n
                                              \u81ea\u7531\u80fd\u539f\u7406\u88ab\u8ba4\u4e3a\u662f\u201c\u81ea\u8fbe\u5c14\u6587\u81ea\u7136\u9009\u62e9\u7406\u8bba\u540e\u6700\u5305\u7f57\u4e07\u8c61\u7684\u601d\u60f3\u201d\uff0c\u5b83\u8bd5\u56fe\u4ece\u7269\u7406\u3001\u751f\u7269\u548c\u5fc3\u667a\u7684\u89d2\u5ea6\u63d0\u4f9b\u667a\u80fd\u4f53\u611f\u77e5\u548c\u884c\u52a8\u7684\u7edf\u4e00\u6027\u89c4\u5f8b\uff0c\u4ece\u7b2c\u4e00\u6027\u539f\u7406\u51fa\u53d1\u89e3\u91ca\u667a\u80fd\u4f53\u66f4\u65b0\u8ba4\u77e5\u3001\u63a2\u7d22\u548c\u6539\u53d8\u4e16\u754c\u7684\u673a\u5236\uff0c\u4ece\u800c\u5bf9\u4eba\u5de5\u667a\u80fd\uff0c\u7279\u522b\u662f\u5f3a\u5316\u5b66\u4e60\u4e16\u754c\u6a21\u578b\u3001\u901a\u7528\u4eba\u5de5\u667a\u80fd\u7814\u7a76\u5177\u6709\u91cd\u8981\u542f\u53d1\u610f\u4e49\u3002<\/span><\/span><\/section>\n

                                              <\/span><\/p>\n
                                              \u96c6\u667a\u4ff1\u4e50\u90e8\u8054\u5408\u5317\u4eac\u5e08\u8303\u5927\u5b66\u7cfb\u7edf\u79d1\u5b66\u5b66\u9662\u535a\u58eb\u751f\u725f\u7267\u4e91\uff0c\u5357\u4eac\u822a\u7a7a\u822a\u5929\u5927\u5b66\u526f\u6559\u6388\u4f55\u771f\uff0c\u4ee5\u53ca\u9aa5\u667a\u667a\u80fd\u79d1\u6280\u7b97\u6cd5\u5de5\u7a0b\u5e08\u3001\u516c\u4f17\u53f7 CreateAMind \u4e3b\u7f16\u5f20\u5fb7\u7965\uff0c\u5171\u540c\u53d1\u8d77\u300c\u81ea\u7531\u80fd\u539f\u7406\u4e0e\u5f3a\u5316\u5b66\u4e60\u8bfb\u4e66\u4f1a<\/a>\u300d\uff0c\u5e0c\u671b\u63a2\u8ba8\u81ea\u7531\u80fd\u539f\u7406\u3001\u5f3a\u5316\u5b66\u4e60\u4e16\u754c\u6a21\u578b\uff0c\u4ee5\u53ca\u8111\u4e0e\u610f\u8bc6\u95ee\u9898\u4e2d\u7684\u9884\u6d4b\u52a0\u5de5\u7406\u8bba\u7b49\u524d\u6cbf\u4ea4\u53c9\u95ee\u9898\uff0c\u63a2\u7d22\u8fd9\u4e9b\u4e0d\u540c\u9886\u57df\u80cc\u540e\u8574\u542b\u7684\u611f\u77e5\u548c\u884c\u52a8\u7684\u7edf\u4e00\u539f\u7406\u3002<\/span>\u8bfb\u4e66\u4f1a\u5df2\u5b8c\u7ed3\uff0c\u73b0\u5728\u62a5\u540d\u53ef\u52a0\u5165\u793e\u7fa4\u5e76\u89e3\u9501\u56de\u653e\u89c6\u9891\u6743\u9650\u3002<\/span><\/span><\/span><\/p>\n

                                              <\/span><\/span><\/p>\n
                                              <\/span><\/span><\/span><\/p>\n

                                              <\/span><\/p>\n
                                              \u8be6\u60c5\u8bf7\u89c1\uff1a<\/span><\/span><\/section>\n
                                              \u81ea\u7531\u80fd\u539f\u7406\u4e0e\u5f3a\u5316\u5b66\u4e60\u8bfb\u4e66\u4f1a\u542f\u52a8\uff1a\u63a2\u7d22\u611f\u77e5\u548c\u884c\u52a8\u7684\u7edf\u4e00\u539f\u7406<\/a><\/span><\/span><\/section>\n

                                              <\/span><\/section>\n<\/section>\n

                                              <\/span><\/p>\n
                                              \u63a8\u8350\u9605\u8bfb<\/span><\/span><\/strong><\/strong><\/strong><\/p>\n
                                              1. \u57fa\u4e8e\u4e16\u754c\u6a21\u578b\u7684\u5f3a\u5316\u5b66\u4e60\u6846\u67b6: \u505a\u68a6\u8005\uff5c\u96c6\u667a\u767e\u79d1<\/span><\/a><\/span><\/strong><\/section>\n
                                              2. \u5f3a\u5316\u5b66\u4e60\uff1a\u5173\u4e8e\u884c\u52a8\u7684\u667a\u80fd | \u4fde\u626c<\/span><\/a><\/span><\/strong>
                                              <\/span><\/strong><\/span><\/section>\n
                                              3. AI\u6253\u6e38\u620f\u80cc\u540e\u7684\u6280\u672f\u539f\u7406 | \u5f20\u6c5f<\/span><\/a><\/span><\/strong>
                                              <\/span><\/strong><\/span><\/section>\n
                                              4. <\/span>\u6d8c\u73b0\u52a8\u529b\u5b66\u5982\u4f55\u7528\u6765\u5206\u6790\u590d\u6742\u7cfb\u7edf\uff1f | \u65b0\u8bfe\u4e0a\u7ebf<\/span><\/a><\/span><\/strong><\/span>
                                              <\/span><\/strong><\/strong><\/span><\/section>\n
                                              <\/strong>5. <\/span><\/strong>\u96c6\u667a\u5b66\u56ed\u7cbe\u54c1\u8bfe\u7a0b\u514d\u8d39\u5f00\u653e\uff0c\u89e3\u9501\u7cfb\u7edf\u79d1\u5b66\u4e0e AI \u65b0\u4e16\u754c<\/span><\/a><\/span><\/strong><\/section>\n
                                              6. \u9ad8\u8003\u5206\u6570\u53ea\u662f\u5f20\u5165\u573a\u5238\uff0c\u4f60\u7684\u79d1\u7814\u5192\u9669\u5728\u8fd9\u91cc\u542f\u822a\uff01<\/span><\/a><\/span><\/strong><\/p>\n
                                              7.<\/span><\/span><\/strong><\/strong><\/span> <\/span><\/span><\/strong><\/strong><\/span>\u7cfb\u7edf\u79d1\u5b66\u524d\u6cbf\u5341\u8bb2\uff1a\u7406\u89e3\u81ea\u7136\u3001\u751f\u547d\u4e0e\u793e\u4f1a\u7684\u590d\u6742\u6027<\/a><\/span><\/strong><\/strong><\/span><\/p>\n

                                              <\/span><\/strong><\/span><\/section>\n

                                              <\/span><\/p>\n
                                              \u70b9\u51fb\u201c\u9605\u8bfb\u539f\u6587\u201d\uff0c<\/span>\u62a5\u540d\u8bfb\u4e66\u4f1a<\/span><\/strong><\/span><\/strong><\/span><\/strong><\/p>\n<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
                                              \u5bfc\u8bed \u6df1\u5ea6Q\u7f51\u7edc(Deep Q-Network, DQN)\u662f\u4e00\u79cd\u57fa\u4e8e\u6df1\u5ea6\u5b66\u4e60\u7684\u5f3a\u5316\u5b66\u4e60\u7b97\u6cd5\uff0c\u5b83\u901a\u8fc7\u6df1\u5ea6\u795e\u7ecf\u7f51\u7edc\u6765\u8fd1\u4f3c\u6700\u4f18\u52a8\u4f5c\u4ef7\u503c\u51fd\u6570\uff08Q\u51fd\u6570\uff09\uff0c\u80fd\u591f\u76f4\u63a5\u4ece\u9ad8\u7ef4\u611f\u77e5\u8f93\u5165\uff08\u5982\u6e38\u620f\u753b\u9762\u50cf\u7d20\uff09\u4e2d\u5b66\u4e60\u6700\u4f18\u51b3\u7b56\u7b56\u7565\u3002\u8be5\u7b97\u6cd5\u7684\u6838\u5fc3\u521b\u65b0\u5728\u4e8e\u7ed3\u5408\u4e86\u7ecf\u9a8c\u56de\u653e\u673a\u5236\u548c\u76ee\u6807\u7f51\u7edc\u6280\u672f\uff0c\u6709\u6548\u89e3\u51b3\u4e86\u4f20\u7edfQ\u5b66\u4e60\u5728\u975e\u7ebf\u6027\u51fd\u6570\u903c\u8fd1\u4e2d\u7684\u7a33…<\/p>\n","protected":false},"author":0,"featured_media":63790,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"special":[],"_links":{"self":[{"href":"https:\/\/swarma.org\/index.php?rest_route=\/wp\/v2\/posts\/63835"}],"collection":[{"href":"https:\/\/swarma.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/swarma.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/swarma.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=63835"}],"version-history":[{"count":0,"href":"https:\/\/swarma.org\/index.php?rest_route=\/wp\/v2\/posts\/63835\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/swarma.org\/index.php?rest_route=\/wp\/v2\/media\/63790"}],"wp:attachment":[{"href":"https:\/\/swarma.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=63835"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/swarma.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=63835"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/swarma.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=63835"},{"taxonomy":"special","embeddable":true,"href":"https:\/\/swarma.org\/index.php?rest_route=%2Fwp%2Fv2%2Fspecial&post=63835"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}