Day trip to Kutna Hora for the Sedlec Ossuary if you're a fan of chapels built from the bones of thousands of Black Death victims.
Otherwise there's a museum of medieval torture devices, and a museum of sex machines. Take turns guessing which one you're in as you wander around.
Be sure to visit U Flecku, a bar where they brew their own delicious beer. When they start offering the walnut liqueur, refuse as it's expensive and will get you trousered in a less pleasant way than the beer.
The castle is pretty good, set aside a day for a visit there and the park on that side of the river.