Nothing is known of this castle's origins, one of the oldest in Europe. Roman, Visigothic and Arabic traces give way to the present-day architecture, which dates from the 15th century.

The history of the castle, which was a residence of Abderraman the Great, and also of the Kings of Castile during the Middle Ages, is steeped in legends. The best-known of these is the tragic love of the beautiful Doña Elvira, lady of the castle, and the monk Roberto, who after loving each other as adolescents, meet again after many years when the monk is named chaplain to Count Ridoura, Elvira's husband, and lord of the castle.
On returning from the battle of Las Navas de Tolosa, the Count discovers the Count discovers the infamy, kills the monk by goring him with an iron crown, and sets fire to the tower where the Countess has fled.

After being occupied by Henry IV, it later became the property of the Constables of Castle, and entered the history of Europe when, after the Battle of Pavía, the sons of Francois I of France were held hostage there until the Treaty of Cambray was signed.

It was 1921 when, on one of continuous visits to Segovia from Paris, the painter Ignacio Zuloaga took an interest in the castle, then abandoned and in ruins, bought it and partially restored it. From then on, it was one of his favourite places for resting and working. It was here he painted his famous works Eufemio, Paisaje de Pedraza ands some others.

Notable elements of the austere construction are its turreted ramparts with the two square towers, the original 15th-century doorway made from black poplar, the large keep tower, the courtyard with a Romanesque water tank, and the arches, which stand out strikingly from the hollowed walls of its interior structure. Pedraza Castle is nowadays a construction with little more than walls and moat, with the stumps of its towers, the impressive bulk of the building standing on the northern side of the rocky hill where Pedraza de la Sierra is located. The site is truly spectacular, overlooking the River Cega valley, with the town of La Velilla beneath, and the views from its privileged vantage point, with its imposing silhouette, look out over one of the most beautiful horizons in all Castile.

Every year, since 1983, between the months of June and July, in the splendid setting of its courtyard, major concerts are held, aiming to progressively encourage Pedraza's cultural atmosphere.