This article presents the results of petrophysical analyses of limestones and sandstones used for the construction of the wall structures of a Roman rural settlement located in Podšilo Bay on Rab Island (Croatia). An on-site analysis of the walls indicated the use of different lithotypes, which is an uncommon case in the area. So far, no petrophysical properties of the applied materials have been tested, and their provenance has not been specified. The aim of this research was to determine their usability as construction materials in an attempt to determine the possible reasons behind the usage of multiple lithotypes and their suitability as building materials. The following procedure was used to address these issues: (1) determination of the petrographic characteristics of the rocks, (2) performance of tests to characterise the mechanical properties in a complex stress state of uniaxial tension followed by uniaxial and triaxial compression, and, finally, (3) determination of the internal structure of the rocks using methods based on X-ray imaging. Multi-proxy characteristics of the materials including numerous observations and methods were performed: optical microscopy used to characterise rock petrography and mineralogy, scanning electron microscopy (SEM) coupled with EDS, as well as grinding tests; furthermore, mechanical properties were determined on cylindrical samples in accordance with European standards. X-ray microtomography using the XμCT method enabled microscopic observations and determination of the orientation of discontinuities and the rock structure. The performed analyses allowed us to distinguish three lithological types of sandstone and two types of limestone among the examined stone blocks. Stone blocks of fine- and medium-grained sandstone with carbonate binders, as well as sparitic limestone and mudstone with calcite veins, were used to build the studied structures. The analysed blocks showed traces of partial edge processing. Despite the defects in the material structure identified using XμCT, all the types of rock were characterised by high or very high strength. High values of longitudinal wave velocity confirmed the good quality of the material. These results contribute to a better understanding of the construction process and the related technological choices, and they provide the first dataset which can be used for the reconstruction of the building’s original appearance in the future.