This work presents a new approach that allows the use of cases in a case base as heuristics to speed up Reinforcement Learning algorithms, combining Case Based Reasoning (CBR) and Reinforcement Learning (RL) techniques. This approach, called Case Based Heuristically Accelerated Reinforcement Learning (CB-HARL), builds upon an emerging technique, the Heuristic Accelerated Reinforcement Learning (HARL), in which RL methods are accelerated by making use of heuristic information. CB-HARL is a subset of RL that makes use of a heuristic function derived from a case base, in a Case Based Reasoning manner. An algorithm that incorporates CBR techniques into Q-learning and Minimax-Q algorithms is also proposed and a set of empirical evaluations were conducted in  simulators for the robot soccer domain, comparing the three solutions for this problem: MRL, HAMRL and CB-HARL. Experimental results show that using CB-HARL, the agents learn faster than using RL or HARL methods.