Autonomous driving (AD) has a huge market and IS receiving enormous attention in both academia and industry. To deal with complex scenarios, autonomous vehicles (AVs) will use reinforcement learning (RL) to design high-level planners in the functional layer but always suffer from safety issues during sim-to-real transfer. One of the main challenges is that the current practice of functional-layer design does not sufficiently consider the uncertainty in the architecture layer, e.g., the software layer and hardware layer. This open challenge will be tackled in this project by a comprehensive study of the interaction between RL and architecture-layer uncertainty. Specifically, we will build virtual AD scenarios on the simulation platform with formal modeling of architecture-layer uncertainty based on real-world data (WP1). The impact of uncertainties on RL will be discussed via the design of cross-layer uncertainty-aware RL (WP2). Inversely, we will also study the robustness of an RL with respect to cross-layer uncertainty by computing the Pareto front of the largest software/hardware uncertainty patterns that a given RL is robust to (WP3). Extensive analysis including verification (WP2, WP3), simulation (WP2, WP3), and real-world experiments (WP4) will be carried out. The success of this project will greatly improve the practicability of RL in AD with a broader impact on other robotics applications.