Ensuring the safety of autonomous vehicles (AVs) is critical for their mass deployment and public adoption. However, security attacks that violate safety constraints and cause accidents are a major deterrent to achieving public trust in AVs which hinders a vendors' ability to deploy the AVs. Creating a security hazard that results in a serious safety compromise (for example, an accident) is compelling from an attacker’s perspective. In this paper, we introduce an attack model, a method to deploy the attack in the form of a smart malware, and experimental evaluation of its impact on production grade autonomous driving software. We find that determining the time interval during which to launch the attack is critically important for causing safety hazards (such as collision) with a high degree of success. For example, the smart malware caused 32× more forced emergency braking compared to random attacks, and accidents in 52.6% of the driving simulations.